vac-optimizer/docs/benchmark/issue-279.md

Issue #279: incremental exact scoring

Step-by-step record for issue #279. Each step keeps planning outcomes, stable explanations and final states identical to develop after PR #278 (a871f8f). The frozen-engine differential check guards every step.

All measurements use Node 24.18.0, Bun 1.3.14 and Vitest 5.0.3 on the development laptop, with no other test, build or profiling workload running alongside the protocol. Only same-session before/after pairs are comparable.

NODE_OPTIONS=--max-old-space-size=2048 timeout 300s bunx vitest run src/app/benchmark/leave-planner-benchmark.spec.ts --maxWorkers=1

The per-scenario figures below are the means of the two measured calculations.

Baseline profile

A CPU profile of the bundled core over all seven scenarios, with calculationDate: '2024-01-01', put 50.5 of 67.0 loaded-year seconds in candidate evaluation (evaluateRuleCandidate). Of those, 41.3 seconds went to active-objective evaluation: 15.6 seconds in time-off-period observations, 11.5 seconds in building observations and 7.6 seconds in sorting placement dates for the placement tie key and the observation cache key. The asynchronous adapter's 1 ms timer yields cost about 7% on typed obligations.

#282: consolidate plan observations

Time-off periods, work sequences, public-holiday weeks, balance observations, the placement list and both tie keys now live in src/app/services/plan-observations.ts. The code moved without behavior changes. All seven scenarios produce byte-identical outcomes, explanations and final states compared with a871f8f.

Measurement	Before	After
Vitest duration	262.000 s	262.030 s
Loaded year	66.010 s	66.497 s
Typed obligations	4.887 s	4.964 s
Two years	2.914 s	2.977 s
Three years	10.203 s	10.049 s
Representative year	0.557 s	0.569 s

The differences are within run-to-run noise. The functional suite passes 666 tests across 34 files, including the frozen-engine check.

#283: incremental plan observations

Each placements map now carries an immutable occupancy: its sorted placement dates, time-off period groups, public-holiday-week completeness and lazily computed calendar observations. A candidate derived from its parent by one added or removed date, or by a copy with the same dates, updates only the affected groups and holiday weeks. Unchanged groups keep their cached periods and observation objects. Maps built from snapshots, maps observed against another calendar and dates that are not in-horizon working days fall back to the from-scratch computation.

The from-scratch computation remains the oracle. A fast-check property test applies random add, remove and retype sequences across year boundaries, horizon edges, declared days, weekends and public holidays, and compares every step with it. An opt-in shadow mode (VAC_OBSERVATION_SHADOW=1 or setObservationShadowMode(true)) compares both on every observation and throws on any mismatch. It passed across the frozen-engine check and all seven benchmark scenarios, whose outcomes, explanations and final states stay byte-identical.

Measurement	Before	After
Vitest duration	262.030 s	193.680 s
Loaded year	66.497 s	48.489 s
Typed obligations	4.964 s	4.947 s
Two years	2.977 s	1.906 s
Three years	10.049 s	6.152 s
Representative year	0.569 s	0.441 s

A fresh loaded-year CPU profile of the bundled core took 46.8 seconds. Of the 1.68 million observed placement maps, one was built from scratch; the rest were derived. Observations now take about 4 seconds. Candidate evaluation still dominates: 24.3 seconds in evaluateRuleCandidate, including 15.2 seconds of active objectives and 6.9 seconds of candidate finalization, plus 6.2 seconds assigning dates and about 5 seconds each of garbage collection and timer idle time.

The functional suite passes 669 tests across 35 files.

#284: prune candidates that cannot beat the protected prefix

Before #284, evaluateRuleCandidate finalized every candidate before comparing its objectives. In the loaded year, 650,369 of 1,233,535 evaluations (53%) then ended at a worse protected prefix. For typed obligations it was 74,336 of 273,470, and for three years 38,048 of 193,621.

Finalizing never moves or retypes a placement. It edits only rule provenance, supporting status, earned activations and earned balances. Before finalizing, the evaluation therefore walks the candidate's protected prefix in objective order, using only objectives whose pre-finalize value is already their exact final value, so the bound is tight:

• time-window and if-then preferences read only placement dates and types, plus declared days;
• public-holiday-week, time-off-period and work-sequence optimizations read only the placement dates;
• leave-balance optimizations are exact only when no enabled if-then rule earns days and the candidate has no earned activation;
• leave-placement optimizations have no safe bound, because finalizing rewrites supporting status.

The walk prunes at the first differing objective when it is worse than the baseline. It stops without pruning at a better value or at an objective without a safe bound. A pruned candidate would have failed finalization or returned at the same comparison, and incumbents are retained only after the full objective vector is built. Pruning therefore never changes which candidate wins or which incumbent is retained.

The frozen-engine check now also runs every tenth seed through both engines' asynchronous adapter, cancelled after a seed-derived number of steps. It compares outcomes, artifacts, every checkpoint, the explanation and outcome callbacks, and the step count. Of those 500 cases, 422 end incomplete and 217 of those keep an incumbent. Pruning on any differing objective breaks both variants. Wrongly treating leave-placement or earning leave-balance objectives as exact went undetected, because the generated requests rarely put those objectives in a protected prefix, so their exclusion rests on the argument above. The check now takes about 60 seconds with coverage.

Measurement	Before	After
Vitest duration	193.680 s	180.250 s
Loaded year	48.489 s	44.287 s
Typed obligations	4.947 s	5.023 s
Two years	1.906 s	1.837 s
Three years	6.152 s	5.935 s
Representative year	0.441 s	0.400 s

All seven scenarios keep byte-identical outcomes, explanations and final states. The functional suite passes 684 tests across 35 files.