test(benchmark): calibrate again after the protocol and judge against the slower value
CI / policy (push) Successful in 8s
CI / validate (push) Successful in 30s
CI / build (push) Successful in 36s
CI / policy (pull_request) Successful in 8s
CI / benchmark (push) Successful in 1m14s
CI / validate (pull_request) Successful in 29s
CI / security (push) Successful in 1m5s
CI / test (push) Successful in 1m54s
CI / build (pull_request) Successful in 23s
CI / security (pull_request) Successful in 58s
CI / images (push) Successful in 56s
CI / required (push) Successful in 1s
CI / benchmark (pull_request) Successful in 1m46s
CI / test (pull_request) Successful in 1m49s
CI / images (pull_request) Successful in 1m16s
CI / required (pull_request) Successful in 2s
CI / policy (push) Successful in 8s
CI / validate (push) Successful in 30s
CI / build (push) Successful in 36s
CI / policy (pull_request) Successful in 8s
CI / benchmark (push) Successful in 1m14s
CI / validate (pull_request) Successful in 29s
CI / security (push) Successful in 1m5s
CI / test (push) Successful in 1m54s
CI / build (pull_request) Successful in 23s
CI / security (pull_request) Successful in 58s
CI / images (push) Successful in 56s
CI / required (push) Successful in 1s
CI / benchmark (pull_request) Successful in 1m46s
CI / test (pull_request) Successful in 1m49s
CI / images (pull_request) Successful in 1m16s
CI / required (pull_request) Successful in 2s
The relative gate calibrated once before the scenarios, so load that started afterwards (the push and pull_request runs of one commit overlap on the same runner, next to the test, build and image jobs) slowed the scenarios but not the yardstick. A pull_request run failed `two years` at 1653.1 ms against a 1648.7 ms budget this way. The spec now repeats the calibration after the protocol and uses the slower of the two values. The budgets can only loosen, never tighten, and the log reprints the table with the budgets actually applied when they change. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7fc3457b63
commit
b37101435a
3 files changed
+51
-25
No files matched your search
@@ -24,6 +24,10 @@ Before the protocol, in the same process, the spec times the calibration engine
|
||||
a871f8f) on the `two years` scenario, best of three, and prints the result. This
|
||||
bundle never changes, so ratios stay comparable with the recorded history; the
|
||||
differential check's frozen bundle can be rebuilt independently.
|
||||
The spec repeats the calibration after the protocol and uses the slower of the two
|
||||
values, so load that starts during the protocol slows the yardstick too; the second
|
||||
value can only loosen the budgets. When it does, the log prints it and the summary
|
||||
table again with the budgets actually applied.
|
||||
Each scenario's budget is its multiplier times that calibration; the mean
|
||||
`planningMs` of its two measured repetitions must not exceed it. A scenario
|
||||
without a multiplier fails too. #286 tightened the multipliers once #302 met
|
||||
|
||||
Reference in new issue
Block a user