nessarandClaude Opus 5.5
68a9e33de6
Finalizing a candidate never moves or retypes a placement, so preference results and calendar optimizations already have their final values before it, and leave-balance optimizations do too when nothing can be earned. Reject a candidate before finalizing when the first differing such objective of the protected prefix is worse than the baseline. Leave-placement objectives and earning contexts stop the walk. The frozen-engine check now also compares cancelled asynchronous runs, including checkpoints and retained incumbents. The loaded year drops from 48.5 s to 44.3 s; outcomes, explanations and final states are unchanged. Refs #284 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
138 lines
6.8 KiB
Markdown
138 lines
6.8 KiB
Markdown
# Issue #279: incremental exact scoring
|
|
|
|
Step-by-step record for [issue #279](https://gitea.nessar.fr/nessar/vac-optimizer/issues/279).
|
|
Each step keeps planning outcomes, stable explanations and final states identical
|
|
to develop after PR #278 (`a871f8f`). The
|
|
[frozen-engine differential check](README.md#frozen-engine-differential-check)
|
|
guards every step.
|
|
|
|
All measurements use Node 24.18.0, Bun 1.3.14 and Vitest 5.0.3 on the development
|
|
laptop, with no other test, build or profiling workload running alongside the
|
|
protocol. Only same-session before/after pairs are comparable.
|
|
|
|
```sh
|
|
NODE_OPTIONS=--max-old-space-size=2048 timeout 300s bunx vitest run src/app/benchmark/leave-planner-benchmark.spec.ts --maxWorkers=1
|
|
```
|
|
|
|
The per-scenario figures below are the means of the two measured calculations.
|
|
|
|
## Baseline profile
|
|
|
|
A CPU profile of the bundled core over all seven scenarios, with
|
|
`calculationDate: '2024-01-01'`, put 50.5 of 67.0 loaded-year seconds in
|
|
candidate evaluation (`evaluateRuleCandidate`). Of those, 41.3 seconds went to
|
|
active-objective evaluation: 15.6 seconds in time-off-period observations,
|
|
11.5 seconds in building observations and 7.6 seconds in sorting placement dates
|
|
for the placement tie key and the observation cache key. The asynchronous
|
|
adapter's 1 ms timer yields cost about 7% on typed obligations.
|
|
|
|
## #282: consolidate plan observations
|
|
|
|
Time-off periods, work sequences, public-holiday weeks, balance observations,
|
|
the placement list and both tie keys now live in
|
|
`src/app/services/plan-observations.ts`. The code moved without behavior
|
|
changes. All seven scenarios produce byte-identical outcomes, explanations and
|
|
final states compared with `a871f8f`.
|
|
|
|
| Measurement | Before | After |
|
|
| ------------------- | --------: | --------: |
|
|
| Vitest duration | 262.000 s | 262.030 s |
|
|
| Loaded year | 66.010 s | 66.497 s |
|
|
| Typed obligations | 4.887 s | 4.964 s |
|
|
| Two years | 2.914 s | 2.977 s |
|
|
| Three years | 10.203 s | 10.049 s |
|
|
| Representative year | 0.557 s | 0.569 s |
|
|
|
|
The differences are within run-to-run noise. The functional suite passes
|
|
666 tests across 34 files, including the frozen-engine check.
|
|
|
|
## #283: incremental plan observations
|
|
|
|
Each placements map now carries an immutable occupancy: its sorted placement
|
|
dates, time-off period groups, public-holiday-week completeness and lazily
|
|
computed calendar observations. A candidate derived from its parent by one added
|
|
or removed date, or by a copy with the same dates, updates only the affected
|
|
groups and holiday weeks. Unchanged groups keep their cached periods and
|
|
observation objects. Maps built from snapshots, maps observed against another
|
|
calendar and dates that are not in-horizon working days fall back to the
|
|
from-scratch computation.
|
|
|
|
The from-scratch computation remains the oracle. A fast-check property test
|
|
applies random add, remove and retype sequences across year boundaries, horizon
|
|
edges, declared days, weekends and public holidays, and compares every step with
|
|
it. An opt-in shadow mode (`VAC_OBSERVATION_SHADOW=1` or
|
|
`setObservationShadowMode(true)`) compares both on every observation and throws
|
|
on any mismatch. It passed across the frozen-engine check and all seven
|
|
benchmark scenarios, whose outcomes, explanations and final states stay
|
|
byte-identical.
|
|
|
|
| Measurement | Before | After |
|
|
| ------------------- | --------: | --------: |
|
|
| Vitest duration | 262.030 s | 193.680 s |
|
|
| Loaded year | 66.497 s | 48.489 s |
|
|
| Typed obligations | 4.964 s | 4.947 s |
|
|
| Two years | 2.977 s | 1.906 s |
|
|
| Three years | 10.049 s | 6.152 s |
|
|
| Representative year | 0.569 s | 0.441 s |
|
|
|
|
A fresh loaded-year CPU profile of the bundled core took 46.8 seconds. Of the
|
|
1.68 million observed placement maps, one was built from scratch; the rest were
|
|
derived. Observations now take about 4 seconds. Candidate evaluation still
|
|
dominates: 24.3 seconds in `evaluateRuleCandidate`, including 15.2 seconds of
|
|
active objectives and 6.9 seconds of candidate finalization, plus 6.2 seconds
|
|
assigning dates and about 5 seconds each of garbage collection and timer idle
|
|
time.
|
|
|
|
The functional suite passes 669 tests across 35 files.
|
|
|
|
## #284: prune candidates that cannot beat the protected prefix
|
|
|
|
Before #284, `evaluateRuleCandidate` finalized every candidate before comparing
|
|
its objectives. In the loaded year, 650,369 of 1,233,535 evaluations (53%) then
|
|
ended at a worse protected prefix. For typed obligations it was 74,336 of
|
|
273,470, and for three years 38,048 of 193,621.
|
|
|
|
Finalizing never moves or retypes a placement. It edits only rule
|
|
provenance, supporting status, earned activations and earned balances. Before
|
|
finalizing, the evaluation therefore walks the candidate's protected prefix in
|
|
objective order, using only objectives whose pre-finalize value is already their
|
|
exact final value, so the bound is tight:
|
|
|
|
- time-window and if-then preferences read only placement dates and types,
|
|
plus declared days;
|
|
- public-holiday-week, time-off-period and work-sequence optimizations read
|
|
only the placement dates;
|
|
- leave-balance optimizations are exact only when no enabled if-then rule earns
|
|
days and the candidate has no earned activation;
|
|
- leave-placement optimizations have no safe bound, because finalizing rewrites
|
|
supporting status.
|
|
|
|
The walk prunes at the first differing objective when it is worse than the
|
|
baseline. It stops without pruning at a better value or at an objective without
|
|
a safe bound. A pruned candidate would have failed finalization or returned at
|
|
the same comparison, and incumbents are retained only after the full objective
|
|
vector is built. Pruning therefore never changes which candidate wins or which
|
|
incumbent is retained.
|
|
|
|
The frozen-engine check now also runs every tenth seed through both engines'
|
|
asynchronous adapter, cancelled after a seed-derived number of steps. It
|
|
compares outcomes, artifacts, every checkpoint, the explanation and outcome
|
|
callbacks, and the step count. Of those 500 cases, 422 end incomplete and 217
|
|
of those keep an incumbent. Pruning on any differing objective breaks both
|
|
variants. Wrongly treating leave-placement or earning leave-balance objectives
|
|
as exact went undetected, because the generated requests rarely put those
|
|
objectives in a protected prefix, so their exclusion rests on the argument
|
|
above. The check now takes about 60 seconds with coverage.
|
|
|
|
| Measurement | Before | After |
|
|
| ------------------- | --------: | --------: |
|
|
| Vitest duration | 193.680 s | 180.250 s |
|
|
| Loaded year | 48.489 s | 44.287 s |
|
|
| Typed obligations | 4.947 s | 5.023 s |
|
|
| Two years | 1.906 s | 1.837 s |
|
|
| Three years | 6.152 s | 5.935 s |
|
|
| Representative year | 0.441 s | 0.400 s |
|
|
|
|
All seven scenarios keep byte-identical outcomes, explanations and final states.
|
|
The functional suite passes 684 tests across 35 files.
|