test(planning): compare the engine against a frozen engine on random requests

Bundle the planning core at a871f8f (develop after PR #278) as a frozen
fixture and compare it with the current engine on 5,000 seeded random
requests. Outcomes, stable explanations, explanation errors and final-state
snapshots must be strictly deep-equal. Requests cover obligations,
preferences, IF_THEN and TIME_WINDOW rules, earned entitlements, composed
optimization rules, declared days, year boundaries and public holidays.

Refs #280

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
nessarandClaude Opus 5.5 committed 2026-10-16 23:30:30 +02:00
1 parent a871f8f28c
commit 4d26e65672
5 files changed
+3531

No files matched your search

+43
View File
@@ -44,6 +44,49 @@ The measurements below describe earlier implementations and protocols. Their
trace counters and timings are historical; current evidence is in
[the #78 report](issue-78.md).
## Frozen-engine differential check
`src/app/services/frozen-engine.spec.ts` runs 5,000 seeded random planning
requests (seeds 280000 to 284999, one test per 200 seeds) through the current
engine and through a frozen copy of the engine at `a871f8f`, develop after
PR #278. For each request, the planning outcome, stable plan explanation,
explanation error and final-state snapshot must be strictly deep-equal,
including `PlanningConflict` and `InvalidInput` outcomes. Timings are not
compared.
Start dates span 2025 to 2027. About 15% of requests start in late December and
cross a year boundary, and about 15% start shortly before a weekday French
public holiday. Most horizons last 7 to 28 days; about 10% last 29 to 70 days
with optimization presets and at most one calendar rule. Requests cover
obligation and preference rules, `IF_THEN` rules with earned leave
entitlements, `TIME_WINDOW` rules, optimization presets in random order and
subsets, random composed optimization configurations, disabled rules, declared
non-working days and a few deliberately invalid requests. The last test fails
if an outcome kind, rule type, year-crossing horizon or public-holiday horizon
stops appearing. Requests stay small because candidate search grows quickly
with horizon length and balances; the slowest case takes about one second.
The fixture is `src/app/services/frozen-engine/leave-planner.core.frozen.mjs`,
a generated bundle with `date-holidays` left external, and its hand-written
`leave-planner.core.frozen.d.mts`. Rebuild it only for an intended behavior
change, in its own reviewed commit, from the commit that introduces the change:
```sh
bun build src/app/services/leave-planner.core.ts --target=node --external date-holidays --outfile src/app/services/frozen-engine/leave-planner.core.frozen.mjs
```
A failure prints the seed and the full request as JSON. To run only that seed:
```sh
FROZEN_ENGINE_SEED=<seed> bunx vitest run src/app/services/frozen-engine.spec.ts
```
The check runs in the functional suite. On the development laptop it takes
about 32 seconds without coverage and 38 seconds with coverage
(`NODE_OPTIONS=--max-old-space-size=2048 bunx vitest run src/app/services/frozen-engine.spec.ts --maxWorkers=1`).
A flipped objective direction in `compareObjectiveValues` made 26 of its tests
fail, starting at seed 280002; the change was reverted.
## Reference-phone protocol
Open the deployed application at `/benchmark` on the reference phone. Tap