ci(benchmark): gate scenario speed relative to the runner

Before the protocol, time the frozen engine on the two-year scenario, best
of three, and fail any scenario whose mean planning time exceeds its
multiplier times that calibration. Multipliers come from development-machine
ratios with about 1.5x headroom until #286 tightens them. The 300-second
timeout stays only as a hang guard. The benchmark keeps running on push and
pull_request events because the push run gates image publication.

This replaces the absolute 300-second acceptance protocol named in #78.

Refs #285

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.antigravityrules

@@ -33,4 +33,4 @@ All AI agents (including Cline, Antigravity, Cursor, and Copilot) must adhere to
      - **Unit Tests and Coverage**: generate the language registry, then run the coverage command in rule 1, matching the Gitea functional-test job.
      - **CI policy and publication**: `node scripts/ci-policy.test.mjs` and `node scripts/ci-image.test.mjs`
      - **Production Build**: `bun run build -- --configuration production`
-   - The separate performance benchmark remains advisory while issue #78 is open. Run it outside coverage with `NODE_OPTIONS=--max-old-space-size=2048 bunx vitest run src/app/benchmark/leave-planner-benchmark.spec.ts --maxWorkers=1` when verifying performance; see `docs/benchmark/README.md`.
+   - The separate performance benchmark is a required CI job with a relative gate: each scenario's budget is a multiple of the frozen engine's two-year calibration measured in the same run. Run it outside coverage with `NODE_OPTIONS=--max-old-space-size=2048 bunx vitest run src/app/benchmark/ --maxWorkers=1` when verifying performance; see `docs/benchmark/README.md`.

.gitea/workflows/ci.yml

@@ -85,8 +85,8 @@ jobs:
             - run: npm install --global bun@1.4.2
             - run: bun install --frozen-lockfile
             - run: bun scripts/generate-langs.js
-            - name: Unchanged benchmark protocol
-              run: timeout 300s bunx vitest run src/app/benchmark/leave-planner-benchmark.spec.ts --maxWorkers=1
+            - name: Benchmark protocol with relative gate
+              run: timeout 300s bunx vitest run src/app/benchmark/ --maxWorkers=1
 
     build:
         name: build

docs/benchmark/README.md

@@ -6,10 +6,45 @@
 unchanged Node acceptance protocol on the documented development laptop:
 **298.6 seconds**, including one warm-up and two measured repetitions of all
 seven scenarios, without coverage, with one worker and a 2 GiB Node heap.
-The full protocol retains its 300-second timeout. All expected outcomes,
+The full protocol keeps a 300-second timeout only as a hang guard (see
+[Relative CI gate](#relative-ci-gate)). All expected outcomes,
 placement counts, final stages and determinism checks pass. The fixture inputs
 are unchanged; the test now prints its existing report.
 
+## Relative CI gate
+
+The relative gate replaces the 300-second acceptance protocol named in
+[#78](https://gitea.nessar.fr/nessar/vac-optimizer/issues/78) as the benchmark
+policy. This is an explicit policy change from #279/#285: runners differ in
+speed, so a fixed wall-clock limit was flaky.
+
+Before the protocol, in the same process, the spec times the frozen engine
+(`src/app/services/frozen-engine/leave-planner.core.frozen.mjs`, the engine at
+a871f8f) on the `two years` scenario, best of three, and prints
+`calibration\tfrozen two years best of 3\t<ms>`. Each scenario's budget is its
+multiplier times that calibration; the mean `planningMs` of its two measured
+repetitions must not exceed it. A scenario without a multiplier fails too.
+`#286` tightens the multipliers once the speed target is met.
+
+| Scenario                    | Mean planning (s) | Ratio to calibration | Multiplier |
+| --------------------------- | ----------------- | -------------------- | ---------- |
+| representative year         | 0.40              | 0.15                 | 0.25       |
+| typed obligations           | 5.02              | 1.83                 | 3          |
+| chained earned entitlements | 0.025             | 0.01                 | 0.05       |
+| deliberate conflict         | 0.013             | 0.005                | 0.05       |
+| loaded year                 | 44.3              | 16.2                 | 25         |
+| two years                   | 1.84              | 0.67                 | 1          |
+| three years                 | 5.94              | 2.17                 | 3.5        |
+
+Ratios come from the development machine (calibration about 2.74 s); the
+multipliers add roughly 1.5x headroom, rounded up, with a floor of 0.05 for the
+two tiny scenarios. A failure lists, per scenario, the measured mean, the
+budget, the multiplier and the calibration in milliseconds. The 300-second
+`timeout` in CI is only a hang guard.
+
+TODO: measure the ratios on at least one CI runner; no runner numbers are
+recorded yet.
+
 The post-removal functional integration checks, browser QA and unchanged-input
 handoff for #78 are recorded in [the #77 verification report](../trace-free-planning-verification.md).
 

docs/gitea-actions.md

@@ -16,16 +16,24 @@ synchronization, and title/body edits. Manual dispatch runs validation without
 publishing images. There is no schedule or deployment job in the preserved
 configuration; the Compose files remain manual deployment options.
 
-| Job         | Behavior                                                                                                                                                                                                  |
-| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `policy`    | Tests the policy and publisher, then validates the checked-out commit, GitFlow, PR title, commits, and stable release version.                                                                            |
-| `validate`  | Runs lint, formatting, and application/test type checks.                                                                                                                                                  |
-| `test`      | Runs functional coverage with one worker and a 2 GiB Node heap; uploads coverage for seven days even on failure.                                                                                          |
-| `benchmark` | Required seven-scenario protocol, with one warm-up and two measured repetitions, outside coverage with one worker, a 2 GiB Node heap and the unchanged 300-second timeout. |
-| `build`     | Builds browser/server production bundles and retains `dist` for seven days.                                                                                                                               |
-| `security`  | Runs Semgrep JavaScript/TypeScript SAST and Trivy dependency/secret scans; retains SARIF reports for seven days. Scanner execution must succeed; findings remain reports rather than a new severity gate. |
-| `images`    | Publishes branch/SHA images after required validation, including the benchmark, or promotes an existing SHA image for a stable tag. Runs only for pushes and same-repository PRs. |
-| `required`  | Fails on failed, cancelled, or skipped required validation, including the benchmark. Requires image publication when expected; accepts its intentional skip for fork PRs and manual dispatch. |
+| Job         | Behavior                                                                                                                                                                                                                                                                                                                                     |
+| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `policy`    | Tests the policy and publisher, then validates the checked-out commit, GitFlow, PR title, commits, and stable release version.                                                                                                                                                                                                               |
+| `validate`  | Runs lint, formatting, and application/test type checks.                                                                                                                                                                                                                                                                                     |
+| `test`      | Runs functional coverage with one worker and a 2 GiB Node heap; uploads coverage for seven days even on failure.                                                                                                                                                                                                                             |
+| `benchmark` | Required seven-scenario protocol (one warm-up, two measured repetitions) outside coverage with one worker and a 2 GiB Node heap. It first times the frozen engine on `two years` (best of three) and fails any scenario whose mean planning time exceeds its multiplier times that calibration; the 300-second timeout is only a hang guard. |
+| `build`     | Builds browser/server production bundles and retains `dist` for seven days.                                                                                                                                                                                                                                                                  |
+| `security`  | Runs Semgrep JavaScript/TypeScript SAST and Trivy dependency/secret scans; retains SARIF reports for seven days. Scanner execution must succeed; findings remain reports rather than a new severity gate.                                                                                                                                    |
+| `images`    | Publishes branch/SHA images after required validation, including the benchmark, or promotes an existing SHA image for a stable tag. Runs only for pushes and same-repository PRs.                                                                                                                                                            |
+| `required`  | Fails on failed, cancelled, or skipped required validation, including the benchmark. Requires image publication when expected; accepts its intentional skip for fork PRs and manual dispatch.                                                                                                                                                |
+
+The benchmark runs on both push and pull_request events: the push run gates image
+publication (`images.needs` includes `benchmark`) and the `required` aggregate.
+The relative gate absorbs a slower runner and load that lasts through the whole
+job, because calibration runs in the same job. It does not absorb load that
+starts or stops during the protocol, such as an overlapping push and
+pull_request run of the same commit; the 300-second hang guard still bounds
+those runs.
 
 Push and PR runs are separate so PR title and rebase checks run on the PR event.
 New runs cancel older runs for the same event/ref. Image jobs serialize per

src/app/benchmark/benchmark-budget.spec.ts

@@ -0,0 +1,30 @@
+import { BenchmarkMeasurement, budgetViolations } from './leave-planner-benchmark';
+
+const measure = (scenario: string, planningMs: number) =>
+	({ scenario, planningMs }) as BenchmarkMeasurement;
+
+describe('benchmark budget gate', () => {
+	const multipliers = { a: 2 };
+	it('accepts a mean within multiplier x calibration', () => {
+		expect(budgetViolations([measure('a', 150), measure('a', 250)], 100, multipliers)).toEqual(
+			[],
+		);
+	});
+	it('reports measured time, budget, multiplier and calibration when over budget', () => {
+		const [message] = budgetViolations(
+			[measure('a', 300), measure('a', 320)],
+			100,
+			multipliers,
+		);
+		expect(message).toContain('a:');
+		expect(message).toContain('measured 310.0 ms');
+		expect(message).toContain('budget 200.0 ms');
+		expect(message).toContain('multiplier 2');
+		expect(message).toContain('calibration 100.0 ms');
+	});
+	it('fails a scenario without a multiplier', () => {
+		expect(budgetViolations([measure('new', 1)], 100, multipliers)).toEqual([
+			'new: no budget multiplier defined',
+		]);
+	});
+});

src/app/benchmark/leave-planner-benchmark.spec.ts

@@ -1,10 +1,13 @@
 import { vi } from 'vitest';
+import { calculatePlanningAsyncInternal } from '../services/frozen-engine/leave-planner.core.frozen.mjs';
 import { createPlanningCalculation } from '../services/leave-planner.service';
 import {
+	BENCHMARK_BUDGET_MULTIPLIERS,
 	BENCHMARK_REPORT_HEADER,
 	benchmarkReport,
 	benchmarkReportRows,
 	benchmarkScenarios,
+	budgetViolations,
 	runLeavePlannerBenchmark,
 } from './leave-planner-benchmark';
 
@@ -66,6 +69,16 @@ describe('leave planner benchmark protocol', () => {
 					),
 				).toHaveLength(2);
 			}
+			// Frozen engine (a871f8f) on 'two years', best of three: the runner-speed yardstick.
+			const calibrationRequest = scenarios.find((item) => item.name === 'two years')!.request;
+			let calibrationMs = Infinity;
+			for (let run = 0; run < 3; run++) {
+				const request = structuredClone(calibrationRequest);
+				const start = performance.now();
+				await calculatePlanningAsyncInternal({ ...request, calculationDate: '2024-01-01' });
+				calibrationMs = Math.min(calibrationMs, performance.now() - start);
+			}
+			console.info(`calibration\tfrozen two years best of 3\t${calibrationMs.toFixed(1)}`);
 			console.info(BENCHMARK_REPORT_HEADER);
 			const measurements = await runLeavePlannerBenchmark(2, (rows) =>
 				console.info(benchmarkReportRows(rows)),
@@ -106,6 +119,13 @@ describe('leave planner benchmark protocol', () => {
 			expect(report).toContain('planning_ms\twall_ms');
 			expect(report).toContain('planning_work_ms\tpeak_main_thread_heap_bytes');
 			expect(report).toContain('\tbaseline\t');
+			const violations = budgetViolations(
+				measurements,
+				calibrationMs,
+				BENCHMARK_BUDGET_MULTIPLIERS,
+			);
+			if (violations.length)
+				throw new Error(`Benchmark over budget:\n${violations.join('\n')}`);
 		} finally {
 			vi.stubGlobal('window', browserWindow);
 		}

src/app/benchmark/leave-planner-benchmark.ts

@@ -307,3 +307,40 @@ export function benchmarkReport(measurements: BenchmarkMeasurement[]): string {
 		? [BENCHMARK_REPORT_HEADER, benchmarkReportRows(measurements)].join('\n')
 		: BENCHMARK_REPORT_HEADER;
 }
+
+// Measured on the development machine (ratio to the frozen two-years run), tightened by #286.
+export const BENCHMARK_BUDGET_MULTIPLIERS: Record<string, number> = {
+	'representative year': 0.25,
+	'typed obligations': 3,
+	'chained earned entitlements': 0.05,
+	'deliberate conflict': 0.05,
+	'loaded year': 25,
+	'two years': 1,
+	'three years': 3.5,
+};
+
+/** One message per scenario whose mean planning time exceeds multiplier x calibration, or that has no multiplier. */
+export function budgetViolations(
+	measurements: BenchmarkMeasurement[],
+	calibrationMs: number,
+	multipliers: Record<string, number> = BENCHMARK_BUDGET_MULTIPLIERS,
+): string[] {
+	const byScenario = new Map<string, number[]>();
+	for (const { scenario, planningMs } of measurements)
+		byScenario.set(scenario, [...(byScenario.get(scenario) ?? []), planningMs]);
+	const violations: string[] = [];
+	for (const [scenario, times] of byScenario) {
+		const mean = times.reduce((sum, value) => sum + value, 0) / times.length;
+		const multiplier = multipliers[scenario];
+		if (multiplier === undefined) {
+			violations.push(`${scenario}: no budget multiplier defined`);
+			continue;
+		}
+		const budget = multiplier * calibrationMs;
+		if (mean > budget)
+			violations.push(
+				`${scenario}: measured ${mean.toFixed(1)} ms > budget ${budget.toFixed(1)} ms (multiplier ${multiplier} x calibration ${calibrationMs.toFixed(1)} ms)`,
+			);
+	}
+	return violations;
+}