test(planning): compare the engine against a frozen engine on random requests
Bundle the planning core at a871f8f (develop after PR #278) as a frozen
fixture and compare it with the current engine on 5,000 seeded random
requests. Outcomes, stable explanations, explanation errors and final-state
snapshots must be strictly deep-equal. Requests cover obligations,
preferences, IF_THEN and TIME_WINDOW rules, earned entitlements, composed
optimization rules, declared days, year boundaries and public holidays.
Refs #280
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a871f8f28c
commit
4d26e65672
5 files changed
+3531
No files matched your search
@@ -44,6 +44,49 @@ The measurements below describe earlier implementations and protocols. Their
|
||||
trace counters and timings are historical; current evidence is in
|
||||
[the #78 report](issue-78.md).
|
||||
|
||||
## Frozen-engine differential check
|
||||
|
||||
`src/app/services/frozen-engine.spec.ts` runs 5,000 seeded random planning
|
||||
requests (seeds 280000 to 284999, one test per 200 seeds) through the current
|
||||
engine and through a frozen copy of the engine at `a871f8f`, develop after
|
||||
PR #278. For each request, the planning outcome, stable plan explanation,
|
||||
explanation error and final-state snapshot must be strictly deep-equal,
|
||||
including `PlanningConflict` and `InvalidInput` outcomes. Timings are not
|
||||
compared.
|
||||
|
||||
Start dates span 2025 to 2027. About 15% of requests start in late December and
|
||||
cross a year boundary, and about 15% start shortly before a weekday French
|
||||
public holiday. Most horizons last 7 to 28 days; about 10% last 29 to 70 days
|
||||
with optimization presets and at most one calendar rule. Requests cover
|
||||
obligation and preference rules, `IF_THEN` rules with earned leave
|
||||
entitlements, `TIME_WINDOW` rules, optimization presets in random order and
|
||||
subsets, random composed optimization configurations, disabled rules, declared
|
||||
non-working days and a few deliberately invalid requests. The last test fails
|
||||
if an outcome kind, rule type, year-crossing horizon or public-holiday horizon
|
||||
stops appearing. Requests stay small because candidate search grows quickly
|
||||
with horizon length and balances; the slowest case takes about one second.
|
||||
|
||||
The fixture is `src/app/services/frozen-engine/leave-planner.core.frozen.mjs`,
|
||||
a generated bundle with `date-holidays` left external, and its hand-written
|
||||
`leave-planner.core.frozen.d.mts`. Rebuild it only for an intended behavior
|
||||
change, in its own reviewed commit, from the commit that introduces the change:
|
||||
|
||||
```sh
|
||||
bun build src/app/services/leave-planner.core.ts --target=node --external date-holidays --outfile src/app/services/frozen-engine/leave-planner.core.frozen.mjs
|
||||
```
|
||||
|
||||
A failure prints the seed and the full request as JSON. To run only that seed:
|
||||
|
||||
```sh
|
||||
FROZEN_ENGINE_SEED=<seed> bunx vitest run src/app/services/frozen-engine.spec.ts
|
||||
```
|
||||
|
||||
The check runs in the functional suite. On the development laptop it takes
|
||||
about 32 seconds without coverage and 38 seconds with coverage
|
||||
(`NODE_OPTIONS=--max-old-space-size=2048 bunx vitest run src/app/services/frozen-engine.spec.ts --maxWorkers=1`).
|
||||
A flipped objective direction in `compareObjectiveValues` made 26 of its tests
|
||||
fail, starting at seed 280002; the change was reverted.
|
||||
|
||||
## Reference-phone protocol
|
||||
|
||||
Open the deployed application at `/benchmark` on the reference phone. Tap
|
||||
|
||||
Reference in new issue
Block a user