docs(benchmark): compare engines by hand before each frozen-engine rebuild

A standing "no worse than the previous engine" check is not added to
CI: the differential check already fails on unintended changes, and a
fixed acceptance threshold would need renegotiating with every
deliberate change. Instead, the rebuild procedure now runs the
comparison while the frozen engine is still the previous one, and
records it with the change. The runner compares against the frozen
engine by default and takes `calibration` to reproduce the #302 record.

Part of #302.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/benchmark/README.md

@@ -150,19 +150,30 @@ with horizon length and balances; the slowest case takes about one second.
 
 The generator is `src/app/services/frozen-engine/random-planning-requests.ts`.
 `compare-engines.ts` beside it is an opt-in runner, outside the functional suite
-and CI, that classifies how the current engine's plans differ from the
-calibration engine's over the same seeds; [the #302 record](issue-302.md#comparison-record)
-shows its use.
+and CI, that classifies how the current engine's plans differ from the frozen
+engine's, or with `calibration` from the calibration engine's, over the same
+seeds; [the #302 record](issue-302.md#comparison-record) shows its use.
 
 The fixture is `src/app/services/frozen-engine/leave-planner.core.frozen.mjs`,
 a generated bundle with `date-holidays` left external, and its hand-written
 `leave-planner.core.frozen.d.mts`. Rebuild it only for an intended behavior
-change, in its own reviewed commit, from the commit that introduces the change:
+change, in its own reviewed commit, from the commit that introduces the change.
+Before rebuilding, while the frozen engine is still the previous one, run the
+comparison and record its counts and every request that became worse on a real
+objective with the change, as [the #302 record](issue-302.md#comparison-record)
+does:
 
 ```sh
+bun src/app/services/frozen-engine/compare-engines.ts small 280000 5000
 bun build src/app/services/leave-planner.core.ts --target=node --external date-holidays --outfile src/app/services/frozen-engine/leave-planner.core.frozen.mjs
 ```
 
+The comparison is deliberately not a CI check. The differential check already
+fails on any unintended change; the comparison only describes an intended one,
+and a fixed acceptance threshold would have to be renegotiated with every
+rebuild. The loaded-year-sized sample (`loaded-year 302000 50`) takes about an
+hour and is worth running when a change targets large plans.
+
 A failure prints the seed and the full request as JSON. To run only that seed:
 
 ```sh

docs/benchmark/issue-302.md

@@ -182,7 +182,7 @@ The opt-in runner compares the current engine with the calibration engine
 requests. It runs outside the functional suite and CI:
 
 ```sh
-bun src/app/services/frozen-engine/compare-engines.ts small 280000 5000
+bun src/app/services/frozen-engine/compare-engines.ts small 280000 5000 calibration
 ```
 
 | Result                            | Requests |
@@ -211,7 +211,9 @@ objective:
 
 Seed 282254 is better: its `tw1` preference goes from 1 to 2. The runner prints
 every non-identical seed with its first differing objective; a first seed and
-a count of 1, such as `small 284696 1`, reproduce a single request.
+a count of 1, such as `small 284696 1 calibration`, reproduce a single request.
+Without `calibration`, the runner compares against the frozen engine, which is
+rebuilt from this change and therefore matches it.
 
 None of these 5,000 requests reaches the earned-entitlement fallback. A
 separate search of seeds 200000 to 259999 with the same generator found 28
@@ -230,7 +232,7 @@ solution limit, so the calibration engine previews eight solutions. Fifty of
 them were compared, in five parallel runs of ten seeds:
 
 ```sh
-bun src/app/services/frozen-engine/compare-engines.ts loaded-year 302000 50
+bun src/app/services/frozen-engine/compare-engines.ts loaded-year 302000 50 calibration
 ```
 
 | Result                          | Requests |

src/app/services/frozen-engine/compare-engines.ts

@@ -1,6 +1,7 @@
 // Opt-in before/after record, outside the functional suite and CI: compares the
-// current engine with the calibration engine (a871f8f) over seeded requests.
-//   bun src/app/services/frozen-engine/compare-engines.ts [small|loaded-year] [first seed] [count]
+// current engine with the frozen engine (before its rebuild) or the calibration
+// engine (a871f8f) over seeded requests.
+//   bun src/app/services/frozen-engine/compare-engines.ts [small|loaded-year] [first seed] [count] [frozen|calibration]
 import { isDeepStrictEqual } from 'node:util';
 import type { PlanningArtifacts, PlanningObjective, PlanningOutcome } from '../leave-planner.core';
 import { compareObjectiveValues } from '../planning-objectives';
@@ -9,9 +10,15 @@ import { compareObjectiveValues } from '../planning-objectives';
 await import('@angular/compiler');
 const { generateLoadedYearRequest, generateRequest } = await import('./random-planning-requests');
 const current = await import('../leave-planner.core');
-const calibration = await import('./leave-planner.core.calibration.mjs');
+const baselines = {
+	frozen: await import('./leave-planner.core.frozen.mjs'),
+	calibration: await import('./leave-planner.core.calibration.mjs'),
+};
 
-const [kind = 'small', first = '280000', count = '5000'] = process.argv.slice(2);
+const [kind = 'small', first = '280000', count = '5000', baselineName = 'frozen'] =
+	process.argv.slice(2);
+const baseline = baselines[baselineName as keyof typeof baselines];
+if (!baseline) throw new Error(`Unknown baseline engine: ${baselineName}`);
 const generate = kind === 'loaded-year' ? generateLoadedYearRequest : generateRequest;
 type Engine = (
 	request: unknown,
@@ -45,7 +52,7 @@ const counts = new Map<string, number>();
 const time = { before: 0, after: 0 };
 for (let seed = Number(first); seed < Number(first) + Number(count); seed++) {
 	const request = generate(seed);
-	const before = run(calibration.calculatePlanningInternal, request);
+	const before = run(baseline.calculatePlanningInternal, request);
 	const after = run(current.calculatePlanningInternal, request);
 	time.before += before.ms;
 	time.after += after.ms;
@@ -73,8 +80,10 @@ for (let seed = Number(first); seed < Number(first) + Number(count); seed++) {
 	if (category !== 'identical') console.log(`${seed}\t${category}${detail}`);
 	counts.set(category, (counts.get(category) ?? 0) + 1);
 }
-console.log(`${kind} requests, seeds ${first}..${Number(first) + Number(count) - 1}`);
+console.log(
+	`${kind} requests, seeds ${first}..${Number(first) + Number(count) - 1}, against the ${baselineName} engine`,
+);
 for (const [category, total] of [...counts].sort()) console.log(`${category}\t${total}`);
 console.log(
-	`calibration engine ${(time.before / 1000).toFixed(1)} s, current engine ${(time.after / 1000).toFixed(1)} s`,
+	`${baselineName} engine ${(time.before / 1000).toFixed(1)} s, current engine ${(time.after / 1000).toFixed(1)} s`,
 );