ci(benchmark): gate scenario speed relative to the runner

Before the protocol, time the frozen engine on the two-year scenario, best
of three, and fail any scenario whose mean planning time exceeds its
multiplier times that calibration. Multipliers come from development-machine
ratios with about 1.5x headroom until #286 tightens them. The 300-second
timeout stays only as a hang guard. The benchmark keeps running on push and
pull_request events because the push run gates image publication.

This replaces the absolute 300-second acceptance protocol named in #78.

Refs #285

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
nessarandClaude Opus 5.5 committed 2026-10-17 00:45:28 +02:00
1 parent 68a9e33de6
commit acdb19352a
7 files changed
+144 -14

No files matched your search

+36 -1
View File
@@ -6,10 +6,45 @@
unchanged Node acceptance protocol on the documented development laptop:
**298.6 seconds**, including one warm-up and two measured repetitions of all
seven scenarios, without coverage, with one worker and a 2 GiB Node heap.
The full protocol retains its 300-second timeout. All expected outcomes,
The full protocol keeps a 300-second timeout only as a hang guard (see
[Relative CI gate](#relative-ci-gate)). All expected outcomes,
placement counts, final stages and determinism checks pass. The fixture inputs
are unchanged; the test now prints its existing report.
## Relative CI gate
The relative gate replaces the 300-second acceptance protocol named in
[#78](https://gitea.nessar.fr/nessar/vac-optimizer/issues/78) as the benchmark
policy. This is an explicit policy change from #279/#285: runners differ in
speed, so a fixed wall-clock limit was flaky.
Before the protocol, in the same process, the spec times the frozen engine
(`src/app/services/frozen-engine/leave-planner.core.frozen.mjs`, the engine at
a871f8f) on the `two years` scenario, best of three, and prints
`calibration\tfrozen two years best of 3\t<ms>`. Each scenario's budget is its
multiplier times that calibration; the mean `planningMs` of its two measured
repetitions must not exceed it. A scenario without a multiplier fails too.
`#286` tightens the multipliers once the speed target is met.
| Scenario | Mean planning (s) | Ratio to calibration | Multiplier |
| --------------------------- | ----------------- | -------------------- | ---------- |
| representative year | 0.40 | 0.15 | 0.25 |
| typed obligations | 5.02 | 1.83 | 3 |
| chained earned entitlements | 0.025 | 0.01 | 0.05 |
| deliberate conflict | 0.013 | 0.005 | 0.05 |
| loaded year | 44.3 | 16.2 | 25 |
| two years | 1.84 | 0.67 | 1 |
| three years | 5.94 | 2.17 | 3.5 |
Ratios come from the development machine (calibration about 2.74 s); the
multipliers add roughly 1.5x headroom, rounded up, with a floor of 0.05 for the
two tiny scenarios. A failure lists, per scenario, the measured mean, the
budget, the multiplier and the calibration in milliseconds. The 300-second
`timeout` in CI is only a hang guard.
TODO: measure the ratios on at least one CI runner; no runner numbers are
recorded yet.
The post-removal functional integration checks, browser QA and unchanged-input
handoff for #78 are recorded in [the #77 verification report](../trace-free-planning-verification.md).