ci(benchmark): gate scenario speed relative to the runner
Before the protocol, time the frozen engine on the two-year scenario, best of three, and fail any scenario whose mean planning time exceeds its multiplier times that calibration. Multipliers come from development-machine ratios with about 1.5x headroom until #286 tightens them. The 300-second timeout stays only as a hang guard. The benchmark keeps running on push and pull_request events because the push run gates image publication. This replaces the absolute 300-second acceptance protocol named in #78. Refs #285 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
68a9e33de6
commit
acdb19352a
7 files changed
+144
-14
No files matched your search
@@ -6,10 +6,45 @@
|
||||
unchanged Node acceptance protocol on the documented development laptop:
|
||||
**298.6 seconds**, including one warm-up and two measured repetitions of all
|
||||
seven scenarios, without coverage, with one worker and a 2 GiB Node heap.
|
||||
The full protocol retains its 300-second timeout. All expected outcomes,
|
||||
The full protocol keeps a 300-second timeout only as a hang guard (see
|
||||
[Relative CI gate](#relative-ci-gate)). All expected outcomes,
|
||||
placement counts, final stages and determinism checks pass. The fixture inputs
|
||||
are unchanged; the test now prints its existing report.
|
||||
|
||||
## Relative CI gate
|
||||
|
||||
The relative gate replaces the 300-second acceptance protocol named in
|
||||
[#78](https://gitea.nessar.fr/nessar/vac-optimizer/issues/78) as the benchmark
|
||||
policy. This is an explicit policy change from #279/#285: runners differ in
|
||||
speed, so a fixed wall-clock limit was flaky.
|
||||
|
||||
Before the protocol, in the same process, the spec times the frozen engine
|
||||
(`src/app/services/frozen-engine/leave-planner.core.frozen.mjs`, the engine at
|
||||
a871f8f) on the `two years` scenario, best of three, and prints
|
||||
`calibration\tfrozen two years best of 3\t<ms>`. Each scenario's budget is its
|
||||
multiplier times that calibration; the mean `planningMs` of its two measured
|
||||
repetitions must not exceed it. A scenario without a multiplier fails too.
|
||||
`#286` tightens the multipliers once the speed target is met.
|
||||
|
||||
| Scenario | Mean planning (s) | Ratio to calibration | Multiplier |
|
||||
| --------------------------- | ----------------- | -------------------- | ---------- |
|
||||
| representative year | 0.40 | 0.15 | 0.25 |
|
||||
| typed obligations | 5.02 | 1.83 | 3 |
|
||||
| chained earned entitlements | 0.025 | 0.01 | 0.05 |
|
||||
| deliberate conflict | 0.013 | 0.005 | 0.05 |
|
||||
| loaded year | 44.3 | 16.2 | 25 |
|
||||
| two years | 1.84 | 0.67 | 1 |
|
||||
| three years | 5.94 | 2.17 | 3.5 |
|
||||
|
||||
Ratios come from the development machine (calibration about 2.74 s); the
|
||||
multipliers add roughly 1.5x headroom, rounded up, with a floor of 0.05 for the
|
||||
two tiny scenarios. A failure lists, per scenario, the measured mean, the
|
||||
budget, the multiplier and the calibration in milliseconds. The 300-second
|
||||
`timeout` in CI is only a hang guard.
|
||||
|
||||
TODO: measure the ratios on at least one CI runner; no runner numbers are
|
||||
recorded yet.
|
||||
|
||||
The post-removal functional integration checks, browser QA and unchanged-input
|
||||
handoff for #78 are recorded in [the #77 verification report](../trace-free-planning-verification.md).
|
||||
|
||||
|
||||
Reference in new issue
Block a user