test(planning): compare the engine against a frozen engine on random requests #280

New Issue

2026-10-16T22:39:45+02:00

nessar commented

2026-10-16 22:39:45 +02:00

Parent

What to build

A permanent CI check that compares the planning engine against a frozen copy of the engine on develop right after PR #278 merges. A seeded generator produces small random planning requests: planning horizons, leave balances with availability and expiry dates, declared non-working days, and rule mixes including obligation rules, IF_THEN and TIME_WINDOW rules, and composed optimization rules. Every request runs through both engines, and the planning outcome, stable plan explanation and final-state snapshot must be deep-equal. Every later optimization ticket is verified by this check.

Acceptance criteria

The frozen engine is produced from develop after #278 and stored as a test fixture. How to rebuild it deliberately for an intended behavior change is documented.
The generator is seeded and deterministic. A failure reports the seed and the full request.
Planning outcomes, stable plan explanations and final-state snapshots are compared in full, including planning conflicts and invalid planning input.
Requests cover obligation rules, preference rules, IF_THEN and TIME_WINDOW rules, composed optimization rules, earned leave entitlements and declared non-working days.
Measure the case count's CI time and choose a count that fits it, starting near 2,000. The check runs in the functional CI suite, and its duration is recorded.
A deliberately injected scoring change (for example a flipped tie-break) makes the check fail. Revert it before merging.
The existing functional coverage suite, type checks, lint and formatting pass.

Blocked by

perf(planning): restore the integrated benchmark gate (#278)

## Parent #279 ## What to build A permanent CI check that compares the planning engine against a frozen copy of the engine on `develop` right after PR #278 merges. A seeded generator produces small random planning requests: planning horizons, leave balances with availability and expiry dates, declared non-working days, and rule mixes including obligation rules, IF_THEN and TIME_WINDOW rules, and composed optimization rules. Every request runs through both engines, and the planning outcome, stable plan explanation and final-state snapshot must be deep-equal. Every later optimization ticket is verified by this check. ## Acceptance criteria - [ ] The frozen engine is produced from `develop` after #278 and stored as a test fixture. How to rebuild it deliberately for an intended behavior change is documented. - [ ] The generator is seeded and deterministic. A failure reports the seed and the full request. - [ ] Planning outcomes, stable plan explanations and final-state snapshots are compared in full, including planning conflicts and invalid planning input. - [ ] Requests cover obligation rules, preference rules, IF_THEN and TIME_WINDOW rules, composed optimization rules, earned leave entitlements and declared non-working days. - [ ] Measure the case count's CI time and choose a count that fits it, starting near 2,000. The check runs in the functional CI suite, and its duration is recorded. - [ ] A deliberately injected scoring change (for example a flipped tie-break) makes the check fail. Revert it before merging. - [ ] The existing functional coverage suite, type checks, lint and formatting pass. ## Blocked by - #278

nessar added the ready-for-agent label 2026-10-16 22:39:45 +02:00

nessar added a new dependency 2026-10-16 22:39:45 +02:00

#278 perf(planning): restore the integrated benchmark gate

nessar referenced this issue

2026-10-16 22:39:45 +02:00

refactor(planning): consolidate plan observations in one module #282

nessar added a new dependency 2026-10-16 22:39:46 +02:00

#282 refactor(planning): consolidate plan observations in one module

nessar referenced this issue

2026-10-16 22:39:46 +02:00

ci(benchmark): gate scenario speed relative to the runner #285

nessar added a new dependency 2026-10-16 22:39:46 +02:00