vac-optimizer/docs/benchmark/issue-279.md

Issue #279: incremental exact scoring

Step-by-step record for issue #279. Each step keeps planning outcomes, stable explanations and final states identical to develop after PR #278 (a871f8f). The frozen-engine differential check guards every step.

All measurements use Node 24.18.0, Bun 1.3.14 and Vitest 5.0.3 on the development laptop, with no other test, build or profiling workload running alongside the protocol. Only same-session before/after pairs are comparable.

NODE_OPTIONS=--max-old-space-size=2048 timeout 300s bunx vitest run src/app/benchmark/leave-planner-benchmark.spec.ts --maxWorkers=1

The per-scenario figures below are the means of the two measured calculations.

Baseline profile

A CPU profile of the bundled core over all seven scenarios, with calculationDate: '2024-01-01', put 50.5 of 67.0 loaded-year seconds in candidate evaluation (evaluateRuleCandidate). Of those, 41.3 seconds went to active-objective evaluation: 15.6 seconds in time-off-period observations, 11.5 seconds in building observations and 7.6 seconds in sorting placement dates for the placement tie key and the observation cache key. The asynchronous adapter's 1 ms timer yields cost about 7% on typed obligations.

#282: consolidate plan observations

Time-off periods, work sequences, public-holiday weeks, balance observations, the placement list and both tie keys now live in src/app/services/plan-observations.ts. The code moved without behavior changes. All seven scenarios produce byte-identical outcomes, explanations and final states compared with a871f8f.

Measurement	Before	After
Vitest duration	262.000 s	262.030 s
Loaded year	66.010 s	66.497 s
Typed obligations	4.887 s	4.964 s
Two years	2.914 s	2.977 s
Three years	10.203 s	10.049 s
Representative year	0.557 s	0.569 s

The differences are within run-to-run noise. The functional suite passes 666 tests across 34 files, including the frozen-engine check.

#283: incremental plan observations

Each placements map now carries an immutable occupancy: its sorted placement dates, time-off period groups, public-holiday-week completeness and lazily computed calendar observations. A candidate derived from its parent by one added or removed date, or by a copy with the same dates, updates only the affected groups and holiday weeks. Unchanged groups keep their cached periods and observation objects. Maps built from snapshots, maps observed against another calendar and dates that are not in-horizon working days fall back to the from-scratch computation.

The from-scratch computation remains the oracle. A fast-check property test applies random add, remove and retype sequences across year boundaries, horizon edges, declared days, weekends and public holidays, and compares every step with it. An opt-in shadow mode (VAC_OBSERVATION_SHADOW=1 or setObservationShadowMode(true)) compares both on every observation and throws on any mismatch. It passed across the frozen-engine check and all seven benchmark scenarios, whose outcomes, explanations and final states stay byte-identical.

Measurement	Before	After
Vitest duration	262.030 s	193.680 s
Loaded year	66.497 s	48.489 s
Typed obligations	4.964 s	4.947 s
Two years	2.977 s	1.906 s
Three years	10.049 s	6.152 s
Representative year	0.569 s	0.441 s

A fresh loaded-year CPU profile of the bundled core took 46.8 seconds. Of the 1.68 million observed placement maps, one was built from scratch; the rest were derived. Observations now take about 4 seconds. Candidate evaluation still dominates: 24.3 seconds in evaluateRuleCandidate, including 15.2 seconds of active objectives and 6.9 seconds of candidate finalization, plus 6.2 seconds assigning dates and about 5 seconds each of garbage collection and timer idle time.

The functional suite passes 669 tests across 35 files.

#284: prune candidates that cannot beat the protected prefix

Before #284, evaluateRuleCandidate finalized every candidate before comparing its objectives. In the loaded year, 650,369 of 1,233,535 evaluations (53%) then ended at a worse protected prefix. For typed obligations it was 74,336 of 273,470, and for three years 38,048 of 193,621.

Finalizing never moves or retypes a placement. It edits only rule provenance, supporting status, earned activations and earned balances. Before finalizing, the evaluation therefore walks the candidate's protected prefix in objective order, using only objectives whose pre-finalize value is already their exact final value, so the bound is tight:

• time-window and if-then preferences read only placement dates and types, plus declared days;
• public-holiday-week, time-off-period and work-sequence optimizations read only the placement dates;
• leave-balance optimizations are exact only when no enabled if-then rule earns days and the candidate has no earned activation;
• leave-placement optimizations have no safe bound, because finalizing rewrites supporting status.

The walk prunes at the first differing objective when it is worse than the baseline. It stops without pruning at a better value or at an objective without a safe bound. A pruned candidate would have failed finalization or returned at the same comparison, and incumbents are retained only after the full objective vector is built. Pruning therefore never changes which candidate wins or which incumbent is retained.

The frozen-engine check now also runs every tenth seed through both engines' asynchronous adapter, cancelled after a seed-derived number of steps. It compares outcomes, artifacts, every checkpoint, the explanation and outcome callbacks, and the step count. Of those 500 cases, 422 end incomplete and 217 of those keep an incumbent. Pruning on any differing objective breaks both variants. Wrongly treating leave-placement or earning leave-balance objectives as exact went undetected, because the generated requests rarely put those objectives in a protected prefix, so their exclusion rests on the argument above. The check now takes about 60 seconds with coverage.

Measurement	Before	After
Vitest duration	193.680 s	180.250 s
Loaded year	48.489 s	44.287 s
Typed obligations	4.947 s	5.023 s
Two years	1.906 s	1.837 s
Three years	6.152 s	5.935 s
Representative year	0.441 s	0.400 s

All seven scenarios keep byte-identical outcomes, explanations and final states. The functional suite passes 684 tests across 35 files.

#286: speed target not reached

The target is a loaded year under 10 seconds and every other scenario under 3 seconds, using the existing protocol in Node on the development laptop. The exact optimizations of #282 to #284 do not reach it. Means from the latest protocol run, with the #285 calibration of 2.854 seconds:

Scenario	Target	Latest	Gap factor
Loaded year	< 10 s	44.094 s	4.4x
Typed obligations	< 3 s	5.071 s	1.7x
Three years	< 3 s	5.887 s	2.0x
Two years	< 3 s	1.865 s	met
Representative year	< 3 s	0.402 s	met
Chained earned entitlements	< 3 s	0.025 s	met
Deliberate conflict	< 3 s	0.013 s	met

From develop after PR #278 to now, the loaded year went from 66.0 to 44.1 seconds (−33%), three years from 10.2 to 5.9 seconds and two years from 2.9 to 1.9 seconds. Typed obligations is unchanged at about 5 seconds. The protocol went from 262.0 to 180.3 seconds.

Because the target is not met, the #285 multipliers stay at current speed and the hang-guard timeout is not lowered. #286 remains open for a separate decision on search changes.

Remaining loaded-year profile

A fresh CPU profile of the bundled core at the #285 commit took 43.5 seconds, with outcomes still byte-identical:

Inclusive cost	Seconds
Candidate evaluation (evaluateRuleCandidate)	21.4
— active objectives, including the pruning walk	16.7
— of which objective evaluation (evaluateOptimization)	7.7
— of which plan observations	4.2
— candidate finalization	2.7
Date assignment and release (assignDateToBalance, releasePlacement)	8.1
Garbage collection	5.1
Idle time between asynchronous slices	4.6

The remaining cost is spread over about 1.2 million candidate evaluations rather than concentrated in one computation. Further exact work could:

• yield with something cheaper than a 1 ms timer between 8 ms slices, saving up to the 4.6 seconds of idle time;
• validate optimization configurations once instead of on every evaluation, about 1 second;
• copy fewer balances and earned activations when assigning a date.

Together these are worth perhaps 8 to 10 seconds, still well above 10 seconds for the loaded year. Reaching the target needs fewer candidate evaluations, which is a search change outside #279's exactness constraint.

ADR 0003

ADR 0003 targets five seconds for the dependency-free planner. Under composable optimization rules, the loaded year takes about 44 seconds in Node on the development laptop even after these exact optimizations. That target should be revisited for the loaded year, together with the decision on search changes. This record does not revise the ADR.