nessarandClaude Opus 5.5
fe705fe618
CI / policy (push) Successful in 28s
CI / validate (push) Successful in 46s
CI / build (push) Successful in 25s
CI / policy (pull_request) Successful in 7s
CI / validate (pull_request) Successful in 26s
CI / benchmark (push) Successful in 1m28s
CI / security (push) Successful in 1m11s
CI / build (pull_request) Successful in 23s
CI / test (push) Successful in 2m27s
CI / benchmark (pull_request) Successful in 1m33s
CI / security (pull_request) Successful in 1m0s
CI / images (push) Successful in 1m0s
CI / required (push) Successful in 1s
CI / test (pull_request) Successful in 2m6s
CI / images (pull_request) Successful in 54s
CI / required (pull_request) Successful in 1s
Since assigning a date shares unchanged activations with the parent state, counterfactual analysis edited an activation it shared with the final plan, so a later alternative date could inherit an earlier one's dates and be judged eligible. It now replaces the moved activation. Every counterfactual over seeds 280000-289999 matches the engine before the sharing step again. When every one of several obligation solutions fails earned-entitlement chronology, the planner again reports the obligation conflict, as when every preview failed. No seeded request reaches that path, so the frozen engine is unchanged. Also marks the ranking's known limit as a shortcut and fixes stale documentation found in review. Part of #302. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
292 lines
15 KiB
Markdown
292 lines
15 KiB
Markdown
# Issue #302: plan from the best-ranked obligation solution
|
|
|
|
Step-by-step record for [issue #302](https://gitea.nessar.fr/nessar/vac-optimizer/issues/302).
|
|
The exact steps keep planning outcomes, stable explanations and final states
|
|
identical to the frozen engine at `a871f8f`. The
|
|
[frozen-engine differential check](README.md#frozen-engine-differential-check),
|
|
including its cancelled asynchronous comparisons, passes after every step.
|
|
|
|
That check does not compare counterfactual analyses. Code review found that the
|
|
last exact step, which shares unchanged activations between states, let
|
|
counterfactual analysis edit an activation it shared with the final plan, so a
|
|
later alternative date could inherit an earlier one's dates. A follow-up commit
|
|
replaces that activation instead of editing it. Afterwards, every
|
|
counterfactual of every plan placement for seeds 280000 to 289999, 8,750
|
|
analyses, matched the engine before that step; the same comparison found 4
|
|
differences with the step alone.
|
|
|
|
## Per-step measurements
|
|
|
|
Each step was timed against its parent in the same session, on the development
|
|
machine (AMD Ryzen 7 7800X3D, WSL2, Bun 1.4.3), with no other test, build or
|
|
profiling workload running. Both commits' cores were bundled with the
|
|
frozen-engine build command and called through `calculatePlanningAsyncInternal`
|
|
in one Bun process, with `calculationDate: '2024-01-01'`. This is the
|
|
benchmark's in-thread asynchronous path, without the planning-calculation
|
|
service. For every scenario, each engine ran one warm-up and then one measured
|
|
calculation, alternating engines. Outcomes, explanations and final states were
|
|
hashed; every pair below matched. Only same-session pairs are comparable.
|
|
|
|
### Reuse the donor ranking across public-holiday weeks
|
|
|
|
For every public-holiday week, the holiday-week stage ranked every placement as
|
|
a potential donor by releasing it and evaluating the active objectives. That
|
|
ranking depends only on the plan, so the weeks of one plan now share each
|
|
placement's released objectives. Each week still skips its own dates and walks
|
|
the placements in the same order with the same yields.
|
|
|
|
| Scenario | Before (s) | After (s) |
|
|
| --------------------------- | ---------: | --------: |
|
|
| representative year | 0.66 | 0.66 |
|
|
| typed obligations | 5.11 | 4.89 |
|
|
| chained earned entitlements | 0.02 | 0.02 |
|
|
| deliberate conflict | 0.01 | 0.01 |
|
|
| loaded year | 58.26 | 57.75 |
|
|
| two years | 2.27 | 1.82 |
|
|
| three years | 7.45 | 5.52 |
|
|
|
|
### Retain the best valid plan more cheaply
|
|
|
|
Retaining a better valid plan for cancellation reconciled its optimization
|
|
provenance. For every placement credited by an optimization rule, that
|
|
evaluated all active objectives with the placement released. Only the credited
|
|
objectives are compared, so the released plan now evaluates just those, and
|
|
the calendar observations of the other objectives are no longer computed.
|
|
Retention also no longer computes quality metrics that nothing read. An
|
|
instrumented three-year run spent 1.16 of its 1.22 retention seconds in that
|
|
reconciliation; afterwards retention took 0.26 seconds.
|
|
|
|
| Scenario | Before (s) | After (s) |
|
|
| --------------------------- | ---------: | --------: |
|
|
| representative year | 0.45 | 0.37 |
|
|
| typed obligations | 4.76 | 5.03 |
|
|
| chained earned entitlements | 0.02 | 0.02 |
|
|
| deliberate conflict | 0.01 | 0.01 |
|
|
| loaded year | 55.59 | 53.58 |
|
|
| two years | 1.68 | 1.22 |
|
|
| three years | 5.29 | 3.85 |
|
|
|
|
The typed-obligations difference is run-to-run noise; its retention cost was
|
|
22 milliseconds.
|
|
|
|
### Yield between asynchronous slices without a timer
|
|
|
|
The asynchronous adapter yields roughly every eight milliseconds. Each yield
|
|
waited for a zero-delay timer: about 1.1 ms in Bun, and at least 4 ms in
|
|
browsers once timers nest. Browsers now yield through a message-channel task.
|
|
Bun runs port messages ahead of timers and incoming worker messages, so a
|
|
message-channel yield there kept cancellation and the heartbeat from running
|
|
until the calculation ended. Bun therefore yields with `setImmediate`, which
|
|
costs about 0.01 ms and lets both run between slices.
|
|
|
|
| Scenario | Before (s) | After (s) |
|
|
| --------------------------- | ---------: | --------: |
|
|
| representative year | 0.37 | 0.31 |
|
|
| typed obligations | 4.74 | 4.02 |
|
|
| chained earned entitlements | 0.02 | 0.01 |
|
|
| deliberate conflict | 0.01 | 0.00 |
|
|
| loaded year | 53.80 | 46.90 |
|
|
| two years | 1.23 | 1.04 |
|
|
| three years | 3.86 | 3.21 |
|
|
|
|
The browser path was checked in headless Chromium 149 on the same machine, in a
|
|
dedicated worker with a 50 ms heartbeat timer, running the bundled core before
|
|
and after this step, alternating, twice each. The `three years` calculation took
|
|
3.92 and 3.89 seconds before and 2.70 and 2.66 seconds after; `representative
|
|
year` took 0.51 and 0.45 seconds before and 0.30 seconds after. The heartbeat's
|
|
largest gap stayed between 58 and 67 ms in every run. Cancelling `three years`
|
|
one second after the request returned the retained plan 3.9 to 4.0 ms after the
|
|
cancel message reached the worker, against 7.2 to 8.7 ms before. This is desktop
|
|
evidence, not a reference-phone measurement.
|
|
|
|
### Validate each optimization configuration once
|
|
|
|
Every objective evaluation revalidated its optimization configuration against
|
|
the catalog. Rule normalization already validates each configuration once per
|
|
calculation and keeps its own copy. The presets are constant, and the summed
|
|
variant of a valid configuration is valid because every catalog row accepts
|
|
the sum. The core therefore scores them without revalidating;
|
|
`evaluateOptimization` still validates for its other callers.
|
|
|
|
| Scenario | Before (s) | After (s) |
|
|
| --------------------------- | ---------: | --------: |
|
|
| representative year | 0.35 | 0.30 |
|
|
| typed obligations | 4.03 | 3.46 |
|
|
| chained earned entitlements | 0.01 | 0.01 |
|
|
| deliberate conflict | 0.00 | 0.00 |
|
|
| loaded year | 48.12 | 44.03 |
|
|
| two years | 1.04 | 0.96 |
|
|
| three years | 3.35 | 3.03 |
|
|
|
|
### Copy only the debited balance when assigning a date
|
|
|
|
Assigning a date copied every balance and every earned activation. Balances and
|
|
activations are only edited right after a state copies them, so the new state
|
|
now shares them with its parent, except the balance it debits. Placement copying
|
|
is unchanged. The benchmark scenarios have at most six balances and few earned
|
|
activations, so the gain is within run-to-run noise.
|
|
|
|
| Scenario | Before (s) | After (s) |
|
|
| --------------------------- | ---------: | --------: |
|
|
| representative year | 0.29 | 0.28 |
|
|
| typed obligations | 3.38 | 3.27 |
|
|
| chained earned entitlements | 0.01 | 0.01 |
|
|
| deliberate conflict | 0.00 | 0.00 |
|
|
| loaded year | 43.93 | 44.05 |
|
|
| two years | 0.95 | 0.92 |
|
|
| three years | 3.00 | 2.97 |
|
|
|
|
Across the five exact steps, the same harness measured three years at
|
|
7.45 seconds before the first step and 2.97 seconds after the last, two years at
|
|
2.27 and 0.92 seconds, typed obligations at 5.11 and 3.27 seconds, and the loaded
|
|
year at 58.26 and 44.05 seconds. These figures come from different sessions.
|
|
|
|
## Choosing the obligation solution
|
|
|
|
This is the one deliberate behavior change
|
|
([ADR 0010](../adr/0010-rank-obligation-solutions-before-preferences.md)).
|
|
When obligation rules can be satisfied jointly in several ways, the planner used
|
|
to run the whole preference chain on candidate solutions as previews: every
|
|
solution when their enumeration completed, or the eight best-ranked when a
|
|
window search stopped at its limit. It kept the solution whose preview scored
|
|
best and ran the preference stages again on it. The planner now orders the
|
|
solutions by the pre-preference ranking that chose those eight, the active
|
|
objectives then the tie-breaks of each obligation-only plan, with enumeration
|
|
order breaking equal ranks. It runs the preference stages once, on the first
|
|
solution. If that run ends with earned leave available only after its final
|
|
merged time-off period, the stages restart on the next solution, and a
|
|
conflict is reported only when every solution fails. With several solutions,
|
|
that is the obligation conflict the previews reported when every preview
|
|
failed.
|
|
|
|
### Timing
|
|
|
|
Measured like the exact steps, against the last exact step, with two measured
|
|
calculations per engine. All seven scenarios keep byte-identical outcomes,
|
|
explanations and final states.
|
|
|
|
| Scenario | Before (s) | After (s) |
|
|
| --------------------------- | ---------: | --------: |
|
|
| representative year | 0.26 | 0.26 |
|
|
| typed obligations | 3.26 | 0.11 |
|
|
| chained earned entitlements | 0.00 | 0.01 |
|
|
| deliberate conflict | 0.00 | 0.00 |
|
|
| loaded year | 43.09 | 5.54 |
|
|
| two years | 0.94 | 0.97 |
|
|
| three years | 3.10 | 3.06 |
|
|
|
|
### Comparison record
|
|
|
|
The opt-in runner compares the current engine with the calibration engine
|
|
(`a871f8f`, which still previews) on the frozen-engine check's 5,000 seeded
|
|
requests. It runs outside the functional suite and CI:
|
|
|
|
```sh
|
|
bun src/app/services/frozen-engine/compare-engines.ts small 280000 5000
|
|
```
|
|
|
|
| Result | Requests |
|
|
| --------------------------------- | -------: |
|
|
| Identical | 4,970 |
|
|
| Same objective values, other plan | 6 |
|
|
| Better | 1 |
|
|
| Worse | 23 |
|
|
| Changed outcome kind | 0 |
|
|
|
|
Fourteen of the worse requests differ only in a final tie-break: eight on
|
|
placement dates and six on balance expiry. The other nine are worse on a real
|
|
objective:
|
|
|
|
| Seed | Worse objective | Previews → ranking |
|
|
| ------ | --------------------------- | ------------------ |
|
|
| 280112 | longest work sequence | 2 → 3 |
|
|
| 280204 | composed optimization | 6 → 3 |
|
|
| 280686 | longest work sequence | 2 → 3 |
|
|
| 283162 | supporting placements | 0 → 1 |
|
|
| 283484 | longest work sequence | 4 → 5 |
|
|
| 283930 | `tw0` preference | 2 → 1 |
|
|
| 284628 | longest work sequence | 8 → 16 |
|
|
| 284696 | non-working-day gain | 3 → 0 |
|
|
| 284824 | short work-sequence deficit | 0 → 2 |
|
|
|
|
Seed 282254 is better: its `tw1` preference goes from 1 to 2. The runner prints
|
|
every non-identical seed with its first differing objective; a first seed and
|
|
a count of 1, such as `small 284696 1`, reproduce a single request.
|
|
|
|
None of these 5,000 requests reaches the earned-entitlement fallback. A
|
|
separate search of seeds 200000 to 259999 with the same generator found 28
|
|
requests whose best-ranked solution earns its leave too late; all 28 plan from
|
|
a later solution, and 26 of them are identical to the calibration engine's
|
|
plan. Without the fallback, each would have become a conflict.
|
|
|
|
### Loaded-year-sized sample
|
|
|
|
The 5,000 requests last 7 to 70 days. A second generator,
|
|
`generateLoadedYearRequest`, builds one-year requests shaped like the `loaded
|
|
year` scenario: four to eight CP or RTT balances of 6 to 14 days, two to four
|
|
obligation windows of two or three days in distinct months, up to two
|
|
preference windows, and the six presets. Their window search always stops at its
|
|
solution limit, so the calibration engine previews eight solutions. Fifty of
|
|
them were compared, in five parallel runs of ten seeds:
|
|
|
|
```sh
|
|
bun src/app/services/frozen-engine/compare-engines.ts loaded-year 302000 50
|
|
```
|
|
|
|
| Result | Requests |
|
|
| ------------------------------- | -------: |
|
|
| Identical | 45 |
|
|
| Worse, placement-date tie-break | 1 |
|
|
| Worse on a real objective | 4 |
|
|
| Better or changed outcome kind | 0 |
|
|
|
|
| Seed | Worse objective | Previews → ranking |
|
|
| ------ | --------------------- | ------------------ |
|
|
| 302011 | longest work sequence | 14 → 19 |
|
|
| 302033 | public-holiday weeks | 14 → 13 |
|
|
| 302040 | `window-4` preference | 2 → 1 |
|
|
| 302049 | longest work sequence | 25 → 30 |
|
|
|
|
On these larger plans, ranking loses a real objective more often than on the
|
|
small requests: 4 in 50, against 9 in 5,000. The synchronous calibration engine
|
|
took 3,924 seconds for the 50 requests, about 78 seconds each, and the current
|
|
engine 289 seconds, about 5.8 seconds each.
|
|
|
|
## Benchmark protocol
|
|
|
|
The unchanged benchmark protocol ran on the development machine before the
|
|
first exact step (`b356117`, from a separate worktree) and after the frozen
|
|
engine rebuild, one after the other in the same session, with no other
|
|
workload:
|
|
|
|
```sh
|
|
timeout 600s bun test src/app/benchmark --timeout=600000
|
|
```
|
|
|
|
| Scenario | Before (s) | After (s) | Target (s) |
|
|
| --------------------------- | ---------: | --------: | ---------: |
|
|
| representative year | 0.45 | 0.24 | 3 |
|
|
| typed obligations | 4.70 | 0.10 | 3 |
|
|
| chained earned entitlements | 0.03 | 0.01 | 3 |
|
|
| deliberate conflict | 0.01 | 0.00 | 3 |
|
|
| loaded year | 57.25 | 5.51 | 10 |
|
|
| two years | 2.19 | 0.93 | 5 |
|
|
| three years | 7.03 | 3.11 | 5 |
|
|
| calibration (`two years`) | 3.40 | 3.30 | |
|
|
| whole test run | 232.37 | 40.78 | |
|
|
|
|
Each figure is the mean of the two measured calculations. All seven scenarios
|
|
keep their outcomes, placement counts, completed stages and determinism, and
|
|
every scenario now meets its target.
|
|
|
|
### Gate
|
|
|
|
For [#286](https://gitea.nessar.fr/nessar/vac-optimizer/issues/286), the
|
|
multipliers now add about 1.5 times headroom to these ratios:
|
|
0.15 for the representative year, 0.1 for typed obligations, 0.05 for the two
|
|
tiny scenarios, 2.5 for the loaded year, 0.5 for two years and 1.5 for three years.
|
|
srvhome is about 1.9 times slower than nessar-desktop, so it should need about
|
|
80 seconds for the protocol. The CI `timeout` is now 240 seconds and the test
|
|
timeout 210 seconds, about three times that. The CI benchmark has not yet run
|
|
on the Gitea runners with these values.
|