test(benchmark): calibrate again after the protocol and judge against the slower value
CI / policy (push) Successful in 8s
CI / validate (push) Successful in 30s
CI / build (push) Successful in 36s
CI / policy (pull_request) Successful in 8s
CI / benchmark (push) Successful in 1m14s
CI / validate (pull_request) Successful in 29s
CI / security (push) Successful in 1m5s
CI / test (push) Successful in 1m54s
CI / build (pull_request) Successful in 23s
CI / security (pull_request) Successful in 58s
CI / images (push) Successful in 56s
CI / required (push) Successful in 1s
CI / benchmark (pull_request) Successful in 1m46s
CI / test (pull_request) Successful in 1m49s
CI / images (pull_request) Successful in 1m16s
CI / required (pull_request) Successful in 2s

The relative gate calibrated once before the scenarios, so load that
started afterwards (the push and pull_request runs of one commit overlap
on the same runner, next to the test, build and image jobs) slowed the
scenarios but not the yardstick. A pull_request run failed `two years`
at 1653.1 ms against a 1648.7 ms budget this way.

The spec now repeats the calibration after the protocol and uses the
slower of the two values. The budgets can only loosen, never tighten,
and the log reprints the table with the budgets actually applied when
they change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
nessarandClaude Sonnet 5.5 committed 2026-10-19 10:11:26 +02:00
1 parent 7fc3457b63
commit b37101435a
3 files changed
+51 -25

No files matched your search

+4
View File
@@ -24,6 +24,10 @@ Before the protocol, in the same process, the spec times the calibration engine
a871f8f) on the `two years` scenario, best of three, and prints the result. This
bundle never changes, so ratios stay comparable with the recorded history; the
differential check's frozen bundle can be rebuilt independently.
The spec repeats the calibration after the protocol and uses the slower of the two
values, so load that starts during the protocol slows the yardstick too; the second
value can only loosen the budgets. When it does, the log prints it and the summary
table again with the budgets actually applied.
Each scenario's budget is its multiplier times that calibration; the mean
`planningMs` of its two measured repetitions must not exceed it. A scenario
without a multiplier fails too. #286 tightened the multipliers once #302 met