perf(planning): plan from the best-ranked obligation solution and remove repeated work #303

Merged

nessar merged 12 commits from feature/obligation-solution-ranking into develop 2026-10-18 23:47:14 +02:00

Conversation 1 Commits 12 Files Changed 28

+4988 -756

nessar commented 2026-10-18 22:07:29 +02:00

Owner

Copy link

Copy source

Resolves #302. Part of #286 and #78.

Summary

• Exact steps, one commit each, byte-identical against the frozen engine at a871f8f (including the cancelled asynchronous comparisons): donor-ranking reuse across public-holiday weeks, cheaper incumbent retention, a yield without a clamped timer, one-time configuration validation, and sharing untouched balances when assigning a date.
  • Yield: browsers use a message-channel task; Bun uses setImmediate, because a message channel there starves cancellation and the heartbeat.
• Selection change (8168137): obligation solutions are ranked before preferences, and the preference stages run once, on the best-ranked solution. If that solution earns its leave too late, the stages restart on the next one. ADR 0010 records the decision.
• Frozen engine rebuilt from 8168137 in its own commit (6c9ff4a). The calibration engine stays at a871f8f.
• Gate (#286): multipliers tightened to about 1.5× the new ratios; hang guard lowered from 600 s to 240 s (test timeout 210 s).
• Review fix (fe705fe): the balance-sharing step had let counterfactual analysis edit an activation shared with the final plan. It now replaces it instead. A regression test is added, and all 8,750 counterfactuals for seeds 280000–289999 match the engine before that step.

Benchmark protocol (development machine, Bun 1.4.3, same session)

Scenario	Before	After	Target
loaded year	57.25 s	5.51 s	10 s
three years	7.03 s	3.11 s	5 s
typed obligations	4.70 s	0.10 s	3 s
two years	2.19 s	0.93 s	5 s
representative year	0.45 s	0.24 s	3 s
whole protocol	232 s	41 s

Per-step figures and the Chromium yield check are in docs/benchmark/issue-302.md.

Comparison record (reviewers, please read)

• 5,000 seeded requests: 4,970 identical, 6 with the same objective values but a different plan, 1 better, 23 worse. 14 of the worse ones differ only in a final tie-break; the other 9 are worse on a real objective. These are exactly #302's figures.
• 50 loaded-year-sized requests (new evidence): 45 identical, 1 tie-break only, 4 worse on a real objective (8%). For example, the longest work sequence goes from 14 to 19 days on one seed.
• Fallback: no request among the 5,000 reaches it, but 28 of 60,000 other seeds need it, and all 28 still plan.

Reproduce with bun src/app/services/frozen-engine/compare-engines.ts small 280000 5000 or loaded-year 302000 50. The runner is opt-in and runs neither in the functional suite nor in CI.

Checks

• 753 functional tests pass in the coverage run.
• Lint, format, typecheck, git diff --check and the production build pass; the build shows only the existing bundle-size warnings.
• timeout 240s bun test src/app/benchmark --timeout=240000 passes locally.

Still open

• The CI benchmark passes on at least two runners (#286). srvhome is about 1.9× slower than nessar-desktop and should need about 80 s.
• Unresolved in #302: should a standing "no worse than the previous engine" check replace the one-off record, and should the duration estimate be recalibrated?

🤖 Generated with Claude Code

Resolves #302. Part of #286 and #78. ## Summary - **Exact steps**, one commit each, byte-identical against the frozen engine at `a871f8f` (including the cancelled asynchronous comparisons): donor-ranking reuse across public-holiday weeks, cheaper incumbent retention, a yield without a clamped timer, one-time configuration validation, and sharing untouched balances when assigning a date. - **Yield:** browsers use a message-channel task; Bun uses `setImmediate`, because a message channel there starves cancellation and the heartbeat. - **Selection change** (`8168137`): obligation solutions are ranked before preferences, and the preference stages run once, on the best-ranked solution. If that solution earns its leave too late, the stages restart on the next one. [ADR 0010](docs/adr/0010-rank-obligation-solutions-before-preferences.md) records the decision. - **Frozen engine** rebuilt from `8168137` in its own commit (`6c9ff4a`). The calibration engine stays at `a871f8f`. - **Gate (#286):** multipliers tightened to about 1.5× the new ratios; hang guard lowered from 600 s to 240 s (test timeout 210 s). - **Review fix** (`fe705fe`): the balance-sharing step had let counterfactual analysis edit an activation shared with the final plan. It now replaces it instead. A regression test is added, and all 8,750 counterfactuals for seeds 280000–289999 match the engine before that step. ## Benchmark protocol (development machine, Bun 1.4.3, same session) | Scenario | Before | After | Target | |---|---:|---:|---:| | loaded year | 57.25 s | 5.51 s | 10 s | | three years | 7.03 s | 3.11 s | 5 s | | typed obligations | 4.70 s | 0.10 s | 3 s | | two years | 2.19 s | 0.93 s | 5 s | | representative year | 0.45 s | 0.24 s | 3 s | | whole protocol | 232 s | 41 s | | Per-step figures and the Chromium yield check are in [docs/benchmark/issue-302.md](docs/benchmark/issue-302.md). ## Comparison record (reviewers, please read) - **5,000 seeded requests:** 4,970 identical, 6 with the same objective values but a different plan, 1 better, 23 worse. 14 of the worse ones differ only in a final tie-break; the other 9 are worse on a real objective. These are exactly #302's figures. - **50 loaded-year-sized requests (new evidence):** 45 identical, 1 tie-break only, **4 worse on a real objective** (8%). For example, the longest work sequence goes from 14 to 19 days on one seed. - **Fallback:** no request among the 5,000 reaches it, but 28 of 60,000 other seeds need it, and all 28 still plan. Reproduce with `bun src/app/services/frozen-engine/compare-engines.ts small 280000 5000` or `loaded-year 302000 50`. The runner is opt-in and runs neither in the functional suite nor in CI. ## Checks - 753 functional tests pass in the coverage run. - Lint, format, typecheck, `git diff --check` and the production build pass; the build shows only the existing bundle-size warnings. - `timeout 240s bun test src/app/benchmark --timeout=240000` passes locally. ## Still open - [ ] The CI benchmark passes on at least two runners (#286). srvhome is about 1.9× slower than nessar-desktop and should need about 80 s. - [ ] Unresolved in #302: should a standing "no worse than the previous engine" check replace the one-off record, and should the duration estimate be recalibrated? 🤖 Generated with [Claude Code](https://claude.com/claude-code)

nessar added 10 commits 2026-10-18 22:07:29 +02:00

test(benchmark): keep the a871f8f engine as a fixed calibration engine ... b356117263

The differential check's frozen bundle will be rebuilt for #302; the
relative gate keeps timing the unchanged a871f8f bundle so its ratios
stay comparable with the recorded history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

perf(planning): reuse the donor ranking across public-holiday weeks ... 0b0bb8f9a8

The holiday-week stage ranked every placement as a donor once per
public-holiday week, although the ranking depends only on the plan. The
weeks of one plan now share each placement's released objectives; each
week still skips its own dates and walks the placements in the same
order with the same yields.

Part of #302.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

perf(planning): retain the best valid plan more cheaply ... 2fcba758c8

Retaining a better valid plan reconciled its optimization provenance,
which evaluated every active objective once per credited placement with
that placement released. Only the credited objectives are compared, so
the released plan now evaluates just those and skips observing the
others. Retention also stops computing quality metrics that nothing
read. The retained plan and what Cancel returns are unchanged.

Part of #302.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

perf(planning): yield between asynchronous slices without a timer ... b7ec411c92

Each slice of the asynchronous adapter waited for a zero-delay timer:
about 1.1 ms in Bun, and at least 4 ms in browsers once timers nest.
Browsers now yield through a message-channel task. Bun runs port
messages ahead of timers and incoming worker messages, which starved
cancellation and heartbeats, so it yields with setImmediate. Both still
let queued cancellation and heartbeat tasks run between slices.

Part of #302.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

perf(planning): validate each optimization configuration once ... 9d9404c3db

Every objective evaluation revalidated its optimization configuration.
Rule normalization already validates each configuration once per
calculation and keeps its own copy, the presets are constant, and the
summed variant of a valid configuration stays valid, so the core now
scores them without revalidating. evaluateOptimization still validates
for other callers.

Part of #302.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

perf(planning): copy only the debited balance when assigning a date ... aa6a0c6f1b

Assigning a date copied every balance and every earned activation.
Balances and activations are only edited right after a state copies
them, so the new state now shares them with its parent, except the
balance it debits. Placement copying is unchanged.

Part of #302.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

perf(planning): plan from the best-ranked obligation solution ... 8168137cd5

With several obligation solutions, the planner ran the whole preference
chain on candidate solutions as previews: every solution after a
complete enumeration, or the eight best-ranked after a truncated window
search. It now orders all solutions by the existing pre-preference
ranking (active objectives, then tie-breaks, of each obligation-only
plan; equal ranks keep enumeration order) and runs the preference
stages once, on the first. If that run ends with earned leave available
only after its final merged time-off period, the stages restart on the
next solution, so a request that planned before never becomes a
conflict.

The loaded year drops from 43 to 5.5 seconds and typed obligations from
3.3 to 0.1 seconds, with byte-identical benchmark plans. Over the 5,000
seeded requests, 23 plans become worse (9 on a real objective) and 1
better; over 50 loaded-year-sized requests, 4 become worse on a real
objective. ADR 0010 records the decision; docs/benchmark/issue-302.md
records the comparison and the opt-in runner that reproduces it.

The frozen-engine check fails until the frozen engine is rebuilt from
this commit in the next one.

Part of #302.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(planning): rebuild the frozen engine from the obligation ranking ... 6c9ff4ae34

The frozen engine of the differential check is rebuilt from 8168137,
which plans from the best-ranked obligation solution, with the
documented build command. The check passes again, including its
cancelled asynchronous comparisons and generator-coverage assertions.
The calibration engine stays at a871f8f.

Part of #302.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(benchmark): tighten the relative gate to the measured speed ... 676ea24410

After #302, the development machine's protocol takes about 41 seconds
instead of 232. The budget multipliers now add about 1.5 times headroom
to the new ratios (0.1 for typed obligations, which leaves room for
runner jitter), and the hang guard drops from 600 to 240 seconds, with
a 210-second test timeout: about three times the expected run on
srvhome, which is about 1.9 times slower than nessar-desktop.

Part of #286 and #302. The gate still needs a passing CI benchmark on
two runners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(planning): keep counterfactual moves off the shared final plan ... fe705fe618

Since assigning a date shares unchanged activations with the parent
state, counterfactual analysis edited an activation it shared with the
final plan, so a later alternative date could inherit an earlier one's
dates and be judged eligible. It now replaces the moved activation.
Every counterfactual over seeds 280000-289999 matches the engine before
the sharing step again.

When every one of several obligation solutions fails earned-entitlement
chronology, the planner again reports the obligation conflict, as when
every preview failed. No seeded request reaches that path, so the
frozen engine is unchanged.

Also marks the ranking's known limit as a shortcut and fixes stale
documentation found in review.

Part of #302.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

nessar added 2 commits 2026-10-18 23:31:30 +02:00

docs(benchmark): compare engines by hand before each frozen-engine rebuild ... 75320da685

A standing "no worse than the previous engine" check is not added to
CI: the differential check already fails on unintended changes, and a
fixed acceptance threshold would need renegotiating with every
deliberate change. Instead, the rebuild procedure now runs the
comparison while the frozen engine is still the previous one, and
records it with the change. The runner compares against the frozen
engine by default and takes `calibration` to reproduce the #302 record.

Part of #302.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(planning): recalibrate the duration estimate to the faster engine ... 31cb4a499c

Rendered waits after #302, in throttled headless Chromium, were well
below the shown ranges: two years waited 1.9 s against 3-24 s, three
years 4.5 s against 11-68 s, and the loaded year 4.0 s against 4-26 s.
The prior keeps its structure with refitted, rounded constants,
400 + 250 x (D/365.25)^0.5 x (P/30)^1.5 x R, which stays within about
1.2 times of all six measured configurations. Five years with 30 days
per year now shows 6-37 s instead of 44-265 s. Local refinement is
unchanged.

The 4x CPU throttle slows only the main thread, not the planning
worker, so phones can still wait longer than these desktop samples.

Part of #302.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

nessar commented 2026-10-18 23:31:51 +02:00

Author

Owner

Copy link

Copy source

Both open questions from #302 are settled in the two latest commits:

• Standing quality check (75320da): not added to CI. The rebuild procedure in docs/benchmark/README.md now runs compare-engines.ts by hand while the frozen engine is still the previous one, and the result is recorded with the change. The runner compares against the frozen engine by default; pass calibration to reproduce the #302 record.
• Duration estimate (31cb4a4): refitted to rendered waits measured after #302. Two years showed 3–24 s for a 1.9 s wait; it now shows 1–7 s. Three years goes from 11–68 s to 2–15 s for a 4.5 s wait. Every measured configuration is within about 1.2× of its estimate. The measurements are desktop emulation: the 4× CPU throttle does not reach the planning worker.

The CI benchmark on two runners is still pending.

Both open questions from #302 are settled in the two latest commits: - **Standing quality check** (75320da): not added to CI. The rebuild procedure in docs/benchmark/README.md now runs `compare-engines.ts` by hand while the frozen engine is still the previous one, and the result is recorded with the change. The runner compares against the frozen engine by default; pass `calibration` to reproduce the #302 record. - **Duration estimate** (31cb4a4): refitted to rendered waits measured after #302. Two years showed 3–24 s for a 1.9 s wait; it now shows 1–7 s. Three years goes from 11–68 s to 2–15 s for a 4.5 s wait. Every measured configuration is within about 1.2× of its estimate. The measurements are desktop emulation: the 4× CPU throttle does not reach the planning worker. The CI benchmark on two runners is still pending.

nessar merged commit 7fc3457b63 into develop 2026-10-18 23:47:14 +02:00

nessar referenced this pull request 2026-10-18 23:49:16 +02:00

perf(planning): decide on search changes to reach the loaded-year speed target #288

nessar referenced this pull request 2026-10-18 23:49:16 +02:00

perf(planning): accept the speed target and lower the hang-guard timeout #286

nessar referenced this pull request 2026-10-18 23:49:16 +02:00

perf(planning): incremental exact scoring to reach a 10-second loaded year #279

nessar referenced this pull request 2026-10-18 23:49:17 +02:00

Verify and optimize planning to meet the benchmarks from #75 #78

nessar referenced this pull request 2026-10-19 00:26:57 +02:00

Improve calculation progress feedback #300

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

bug

dependencies

docker

documentation

duplicate

enhancement

github_actions

good first issue

help wanted

invalid

javascript

needs-info

needs-triage

question

ready-for-agent

wontfix

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate (renovate) Bot nessar (nessar)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: nessar/vac-optimizer#303