test(benchmark): keep the a871f8f engine as a fixed calibration engine

The differential check's frozen bundle will be rebuilt for #302; the
relative gate keeps timing the unchanged a871f8f bundle so its ratios
stay comparable with the recorded history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
nessarandClaude Opus 5.5 committed 2026-10-18 19:14:28 +02:00
1 parent ba05fe88b3
commit b356117263
4 files changed
+3251 -5

No files matched your search

+5 -3
View File
@@ -18,9 +18,11 @@ The relative gate replaces the 300-second acceptance protocol named in
policy. This is an explicit policy change from #279/#285: runners differ in policy. This is an explicit policy change from #279/#285: runners differ in
speed, so a fixed wall-clock limit was flaky. speed, so a fixed wall-clock limit was flaky.
Before the protocol, in the same process, the spec times the frozen engine Before the protocol, in the same process, the spec times the calibration engine
(`src/app/services/frozen-engine/leave-planner.core.frozen.mjs`, the engine at (`src/app/services/frozen-engine/leave-planner.core.calibration.mjs`, the engine at
a871f8f) on the `two years` scenario, best of three, and prints the result. a871f8f) on the `two years` scenario, best of three, and prints the result. This
bundle never changes, so ratios stay comparable with the recorded history; the
differential check's frozen bundle can be rebuilt independently.
Each scenario's budget is its multiplier times that calibration; the mean Each scenario's budget is its multiplier times that calibration; the mean
`planningMs` of its two measured repetitions must not exceed it. A scenario `planningMs` of its two measured repetitions must not exceed it. A scenario
without a multiplier fails too. `#286` tightens the multipliers once the speed without a multiplier fails too. `#286` tightens the multipliers once the speed
@@ -1,5 +1,5 @@
import { vi } from '../../testing/vitest-compat'; import { vi } from '../../testing/vitest-compat';
import { calculatePlanningAsyncInternal } from '../services/frozen-engine/leave-planner.core.frozen.mjs'; import { calculatePlanningAsyncInternal } from '../services/frozen-engine/leave-planner.core.calibration.mjs';
import { createPlanningCalculation } from '../services/leave-planner.service'; import { createPlanningCalculation } from '../services/leave-planner.service';
import { import {
BENCHMARK_BUDGET_MULTIPLIERS, BENCHMARK_BUDGET_MULTIPLIERS,
@@ -69,7 +69,7 @@ describe('leave planner benchmark protocol', () => {
), ),
).toHaveLength(2); ).toHaveLength(2);
} }
// Frozen engine (a871f8f) on 'two years', best of three: the runner-speed yardstick. // Calibration engine (frozen at a871f8f) on 'two years', best of three: the runner-speed yardstick.
const calibrationRequest = scenarios.find((item) => item.name === 'two years')!.request; const calibrationRequest = scenarios.find((item) => item.name === 'two years')!.request;
let calibrationMs = Infinity; let calibrationMs = Infinity;
for (let run = 0; run < 3; run++) { for (let run = 0; run < 3; run++) {
@@ -0,0 +1,22 @@
import type {
PlanningArtifactsCallback,
PlanningCheckpoint,
PlanningExplanationCallback,
PlanningOutcome,
PlanningOutcomeCallback,
} from '../leave-planner.core';
export function calculatePlanningInternal(
request: unknown,
checkpointCallback?: undefined,
artifactsCallback?: PlanningArtifactsCallback,
): PlanningOutcome;
export function calculatePlanningAsyncInternal(
request: unknown,
checkpointCallback?: PlanningCheckpoint,
artifactsCallback?: PlanningArtifactsCallback,
outcomeCallback?: PlanningOutcomeCallback,
explanationCallback?: PlanningExplanationCallback,
isCancelled?: () => boolean,
): Promise<PlanningOutcome>;
File diff suppressed because it is too large. Load diff