nessarandClaude Opus 5.5
cfdb28c90a
CI / policy (push) Successful in 6s
CI / validate (push) Successful in 53s
CI / build (push) Successful in 1m18s
CI / policy (pull_request) Successful in 9s
CI / security (push) Successful in 1m12s
CI / validate (pull_request) Successful in 1m18s
CI / benchmark (push) Successful in 3m56s
CI / test (push) Successful in 4m6s
CI / build (pull_request) Successful in 36s
CI / security (pull_request) Successful in 1m5s
CI / test (pull_request) Successful in 3m2s
CI / images (push) Successful in 1m15s
CI / required (push) Successful in 1s
CI / benchmark (pull_request) Successful in 7m18s
CI / images (pull_request) Successful in 1m50s
CI / required (pull_request) Successful in 1s
The first PR run landed on srvhome, about 1.9 times slower than nessar-desktop. The fixed 300-second timeout killed its protocol during the loaded year, although every completed scenario was within its relative budget. Raise the CI timeout to 600 seconds and the test timeout to 570 seconds so they only guard against hangs, and record the first CI-runner ratios. Refs #285 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
201 lines
10 KiB
Markdown
201 lines
10 KiB
Markdown
# Issue #279: incremental exact scoring
|
||
|
||
Step-by-step record for [issue #279](https://gitea.nessar.fr/nessar/vac-optimizer/issues/279).
|
||
Each step keeps planning outcomes, stable explanations and final states identical
|
||
to develop after PR #278 (`a871f8f`). The
|
||
[frozen-engine differential check](README.md#frozen-engine-differential-check)
|
||
guards every step.
|
||
|
||
All measurements use Node 24.18.0, Bun 1.3.14 and Vitest 5.0.3 on the development
|
||
laptop, with no other test, build or profiling workload running alongside the
|
||
protocol. Only same-session before/after pairs are comparable.
|
||
|
||
```sh
|
||
NODE_OPTIONS=--max-old-space-size=2048 timeout 300s bunx vitest run src/app/benchmark/leave-planner-benchmark.spec.ts --maxWorkers=1
|
||
```
|
||
|
||
The per-scenario figures below are the means of the two measured calculations.
|
||
|
||
## Baseline profile
|
||
|
||
A CPU profile of the bundled core over all seven scenarios, with
|
||
`calculationDate: '2024-01-01'`, put 50.5 of 67.0 loaded-year seconds in
|
||
candidate evaluation (`evaluateRuleCandidate`). Of those, 41.3 seconds went to
|
||
active-objective evaluation: 15.6 seconds in time-off-period observations,
|
||
11.5 seconds in building observations and 7.6 seconds in sorting placement dates
|
||
for the placement tie key and the observation cache key. The asynchronous
|
||
adapter's 1 ms timer yields cost about 7% on typed obligations.
|
||
|
||
## #282: consolidate plan observations
|
||
|
||
Time-off periods, work sequences, public-holiday weeks, balance observations,
|
||
the placement list and both tie keys now live in
|
||
`src/app/services/plan-observations.ts`. The code moved without behavior
|
||
changes. All seven scenarios produce byte-identical outcomes, explanations and
|
||
final states compared with `a871f8f`.
|
||
|
||
| Measurement | Before | After |
|
||
| ------------------- | --------: | --------: |
|
||
| Vitest duration | 262.000 s | 262.030 s |
|
||
| Loaded year | 66.010 s | 66.497 s |
|
||
| Typed obligations | 4.887 s | 4.964 s |
|
||
| Two years | 2.914 s | 2.977 s |
|
||
| Three years | 10.203 s | 10.049 s |
|
||
| Representative year | 0.557 s | 0.569 s |
|
||
|
||
The differences are within run-to-run noise. The functional suite passes
|
||
666 tests across 34 files, including the frozen-engine check.
|
||
|
||
## #283: incremental plan observations
|
||
|
||
Each placements map now carries an immutable occupancy: its sorted placement
|
||
dates, time-off period groups, public-holiday-week completeness and lazily
|
||
computed calendar observations. A candidate derived from its parent by one added
|
||
or removed date, or by a copy with the same dates, updates only the affected
|
||
groups and holiday weeks. Unchanged groups keep their cached periods and
|
||
observation objects. Maps built from snapshots, maps observed against another
|
||
calendar and dates that are not in-horizon working days fall back to the
|
||
from-scratch computation.
|
||
|
||
The from-scratch computation remains the oracle. A fast-check property test
|
||
applies random add, remove and retype sequences across year boundaries, horizon
|
||
edges, declared days, weekends and public holidays, and compares every step with
|
||
it. An opt-in shadow mode (`VAC_OBSERVATION_SHADOW=1` or
|
||
`setObservationShadowMode(true)`) compares both on every observation and throws
|
||
on any mismatch. It passed across the frozen-engine check and all seven
|
||
benchmark scenarios, whose outcomes, explanations and final states stay
|
||
byte-identical.
|
||
|
||
| Measurement | Before | After |
|
||
| ------------------- | --------: | --------: |
|
||
| Vitest duration | 262.030 s | 193.680 s |
|
||
| Loaded year | 66.497 s | 48.489 s |
|
||
| Typed obligations | 4.964 s | 4.947 s |
|
||
| Two years | 2.977 s | 1.906 s |
|
||
| Three years | 10.049 s | 6.152 s |
|
||
| Representative year | 0.569 s | 0.441 s |
|
||
|
||
A fresh loaded-year CPU profile of the bundled core took 46.8 seconds. Of the
|
||
1.68 million observed placement maps, one was built from scratch; the rest were
|
||
derived. Observations now take about 4 seconds. Candidate evaluation still
|
||
dominates: 24.3 seconds in `evaluateRuleCandidate`, including 15.2 seconds of
|
||
active objectives and 6.9 seconds of candidate finalization, plus 6.2 seconds
|
||
assigning dates and about 5 seconds each of garbage collection and timer idle
|
||
time.
|
||
|
||
The functional suite passes 669 tests across 35 files.
|
||
|
||
## #284: prune candidates that cannot beat the protected prefix
|
||
|
||
Before #284, `evaluateRuleCandidate` finalized every candidate before comparing
|
||
its objectives. In the loaded year, 650,369 of 1,233,535 evaluations (53%) then
|
||
ended at a worse protected prefix. For typed obligations it was 74,336 of
|
||
273,470, and for three years 38,048 of 193,621.
|
||
|
||
Finalizing never moves or retypes a placement. It edits only rule
|
||
provenance, supporting status, earned activations and earned balances. Before
|
||
finalizing, the evaluation therefore walks the candidate's protected prefix in
|
||
objective order, using only objectives whose pre-finalize value is already their
|
||
exact final value, so the bound is tight:
|
||
|
||
- time-window and if-then preferences read only placement dates and types,
|
||
plus declared days;
|
||
- public-holiday-week, time-off-period and work-sequence optimizations read
|
||
only the placement dates;
|
||
- leave-balance optimizations are exact only when no enabled if-then rule earns
|
||
days and the candidate has no earned activation;
|
||
- leave-placement optimizations have no safe bound, because finalizing rewrites
|
||
supporting status.
|
||
|
||
The walk prunes at the first differing objective when it is worse than the
|
||
baseline. It stops without pruning at a better value or at an objective without
|
||
a safe bound. A pruned candidate would have failed finalization or returned at
|
||
the same comparison, and incumbents are retained only after the full objective
|
||
vector is built. Pruning therefore never changes which candidate wins or which
|
||
incumbent is retained.
|
||
|
||
The frozen-engine check now also runs every tenth seed through both engines'
|
||
asynchronous adapter, cancelled after a seed-derived number of steps. It
|
||
compares outcomes, artifacts, every checkpoint, the explanation and outcome
|
||
callbacks, and the step count. Of those 500 cases, 422 end incomplete and 217
|
||
of those keep an incumbent. Pruning on any differing objective breaks both
|
||
variants. Wrongly treating leave-placement or earning leave-balance objectives
|
||
as exact went undetected, because the generated requests rarely put those
|
||
objectives in a protected prefix, so their exclusion rests on the argument
|
||
above. The check now takes about 60 seconds with coverage.
|
||
|
||
| Measurement | Before | After |
|
||
| ------------------- | --------: | --------: |
|
||
| Vitest duration | 193.680 s | 180.250 s |
|
||
| Loaded year | 48.489 s | 44.287 s |
|
||
| Typed obligations | 4.947 s | 5.023 s |
|
||
| Two years | 1.906 s | 1.837 s |
|
||
| Three years | 6.152 s | 5.935 s |
|
||
| Representative year | 0.441 s | 0.400 s |
|
||
|
||
All seven scenarios keep byte-identical outcomes, explanations and final states.
|
||
The functional suite passes 684 tests across 35 files.
|
||
|
||
## #286: speed target not reached
|
||
|
||
The target is a loaded year under 10 seconds and every other scenario under 3
|
||
seconds, using the existing protocol in Node on the development laptop. The
|
||
exact optimizations of #282 to #284 do not reach it. Means from the latest
|
||
protocol run, with the #285 calibration of 2.854 seconds:
|
||
|
||
| Scenario | Target | Latest | Gap factor |
|
||
| --------------------------- | -----: | -------: | ---------: |
|
||
| Loaded year | < 10 s | 44.094 s | 4.4x |
|
||
| Typed obligations | < 3 s | 5.071 s | 1.7x |
|
||
| Three years | < 3 s | 5.887 s | 2.0x |
|
||
| Two years | < 3 s | 1.865 s | met |
|
||
| Representative year | < 3 s | 0.402 s | met |
|
||
| Chained earned entitlements | < 3 s | 0.025 s | met |
|
||
| Deliberate conflict | < 3 s | 0.013 s | met |
|
||
|
||
From develop after PR #278 to now, the loaded year went from 66.0 to 44.1
|
||
seconds (−33%), three years from 10.2 to 5.9 seconds and two years from 2.9 to
|
||
1.9 seconds. Typed obligations is unchanged at about 5 seconds. The protocol went
|
||
from 262.0 to 180.3 seconds.
|
||
|
||
Because the target is not met, the #285 multipliers stay at current speed and
|
||
the hang-guard timeout is not lowered. #286 remains open for a
|
||
separate decision on search changes.
|
||
|
||
### Remaining loaded-year profile
|
||
|
||
A fresh CPU profile of the bundled core at the #285 commit took 43.5 seconds,
|
||
with outcomes still byte-identical:
|
||
|
||
| Inclusive cost | Seconds |
|
||
| ----------------------------------------------------------------------- | ------: |
|
||
| Candidate evaluation (`evaluateRuleCandidate`) | 21.4 |
|
||
| — active objectives, including the pruning walk | 16.7 |
|
||
| — of which objective evaluation (`evaluateOptimization`) | 7.7 |
|
||
| — of which plan observations | 4.2 |
|
||
| — candidate finalization | 2.7 |
|
||
| Date assignment and release (`assignDateToBalance`, `releasePlacement`) | 8.1 |
|
||
| Garbage collection | 5.1 |
|
||
| Idle time between asynchronous slices | 4.6 |
|
||
|
||
The remaining cost is spread over about 1.2 million candidate evaluations
|
||
rather than concentrated in one computation. Further exact work could:
|
||
|
||
- yield with something cheaper than a 1 ms timer between 8 ms slices, saving up
|
||
to the 4.6 seconds of idle time;
|
||
- validate optimization configurations once instead of on every evaluation,
|
||
about 1 second;
|
||
- copy fewer balances and earned activations when assigning a date.
|
||
|
||
Together these are worth perhaps 8 to 10 seconds, still well above 10 seconds
|
||
for the loaded year. Reaching the target needs fewer candidate evaluations,
|
||
which is a search change outside #279's exactness constraint.
|
||
|
||
### ADR 0003
|
||
|
||
ADR 0003 targets five seconds for the dependency-free planner. Under composable
|
||
optimization rules, the loaded year takes about 44 seconds in Node on the
|
||
development laptop even after these exact optimizations. That target should be
|
||
revisited for the loaded year, together with the decision on search changes.
|
||
This record does not revise the ADR.
|