diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index 49b0d2d..debaed2 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -150,19 +150,30 @@ with horizon length and balances; the slowest case takes about one second. The generator is `src/app/services/frozen-engine/random-planning-requests.ts`. `compare-engines.ts` beside it is an opt-in runner, outside the functional suite -and CI, that classifies how the current engine's plans differ from the -calibration engine's over the same seeds; [the #302 record](issue-302.md#comparison-record) -shows its use. +and CI, that classifies how the current engine's plans differ from the frozen +engine's, or with `calibration` from the calibration engine's, over the same +seeds; [the #302 record](issue-302.md#comparison-record) shows its use. The fixture is `src/app/services/frozen-engine/leave-planner.core.frozen.mjs`, a generated bundle with `date-holidays` left external, and its hand-written `leave-planner.core.frozen.d.mts`. Rebuild it only for an intended behavior -change, in its own reviewed commit, from the commit that introduces the change: +change, in its own reviewed commit, from the commit that introduces the change. +Before rebuilding, while the frozen engine is still the previous one, run the +comparison and record its counts and every request that became worse on a real +objective with the change, as [the #302 record](issue-302.md#comparison-record) +does: ```sh +bun src/app/services/frozen-engine/compare-engines.ts small 280000 5000 bun build src/app/services/leave-planner.core.ts --target=node --external date-holidays --outfile src/app/services/frozen-engine/leave-planner.core.frozen.mjs ``` +The comparison is deliberately not a CI check. The differential check already +fails on any unintended change; the comparison only describes an intended one, +and a fixed acceptance threshold would have to be renegotiated with every +rebuild. The loaded-year-sized sample (`loaded-year 302000 50`) takes about an +hour and is worth running when a change targets large plans. + A failure prints the seed and the full request as JSON. To run only that seed: ```sh diff --git a/docs/benchmark/issue-302.md b/docs/benchmark/issue-302.md index 7305bd6..bc8207b 100644 --- a/docs/benchmark/issue-302.md +++ b/docs/benchmark/issue-302.md @@ -182,7 +182,7 @@ The opt-in runner compares the current engine with the calibration engine requests. It runs outside the functional suite and CI: ```sh -bun src/app/services/frozen-engine/compare-engines.ts small 280000 5000 +bun src/app/services/frozen-engine/compare-engines.ts small 280000 5000 calibration ``` | Result | Requests | @@ -211,7 +211,9 @@ objective: Seed 282254 is better: its `tw1` preference goes from 1 to 2. The runner prints every non-identical seed with its first differing objective; a first seed and -a count of 1, such as `small 284696 1`, reproduce a single request. +a count of 1, such as `small 284696 1 calibration`, reproduce a single request. +Without `calibration`, the runner compares against the frozen engine, which is +rebuilt from this change and therefore matches it. None of these 5,000 requests reaches the earned-entitlement fallback. A separate search of seeds 200000 to 259999 with the same generator found 28 @@ -230,7 +232,7 @@ solution limit, so the calibration engine previews eight solutions. Fifty of them were compared, in five parallel runs of ten seeds: ```sh -bun src/app/services/frozen-engine/compare-engines.ts loaded-year 302000 50 +bun src/app/services/frozen-engine/compare-engines.ts loaded-year 302000 50 calibration ``` | Result | Requests | diff --git a/src/app/services/frozen-engine/compare-engines.ts b/src/app/services/frozen-engine/compare-engines.ts index a1dd0bd..7df2601 100644 --- a/src/app/services/frozen-engine/compare-engines.ts +++ b/src/app/services/frozen-engine/compare-engines.ts @@ -1,6 +1,7 @@ // Opt-in before/after record, outside the functional suite and CI: compares the -// current engine with the calibration engine (a871f8f) over seeded requests. -// bun src/app/services/frozen-engine/compare-engines.ts [small|loaded-year] [first seed] [count] +// current engine with the frozen engine (before its rebuild) or the calibration +// engine (a871f8f) over seeded requests. +// bun src/app/services/frozen-engine/compare-engines.ts [small|loaded-year] [first seed] [count] [frozen|calibration] import { isDeepStrictEqual } from 'node:util'; import type { PlanningArtifacts, PlanningObjective, PlanningOutcome } from '../leave-planner.core'; import { compareObjectiveValues } from '../planning-objectives'; @@ -9,9 +10,15 @@ import { compareObjectiveValues } from '../planning-objectives'; await import('@angular/compiler'); const { generateLoadedYearRequest, generateRequest } = await import('./random-planning-requests'); const current = await import('../leave-planner.core'); -const calibration = await import('./leave-planner.core.calibration.mjs'); +const baselines = { + frozen: await import('./leave-planner.core.frozen.mjs'), + calibration: await import('./leave-planner.core.calibration.mjs'), +}; -const [kind = 'small', first = '280000', count = '5000'] = process.argv.slice(2); +const [kind = 'small', first = '280000', count = '5000', baselineName = 'frozen'] = + process.argv.slice(2); +const baseline = baselines[baselineName as keyof typeof baselines]; +if (!baseline) throw new Error(`Unknown baseline engine: ${baselineName}`); const generate = kind === 'loaded-year' ? generateLoadedYearRequest : generateRequest; type Engine = ( request: unknown, @@ -45,7 +52,7 @@ const counts = new Map(); const time = { before: 0, after: 0 }; for (let seed = Number(first); seed < Number(first) + Number(count); seed++) { const request = generate(seed); - const before = run(calibration.calculatePlanningInternal, request); + const before = run(baseline.calculatePlanningInternal, request); const after = run(current.calculatePlanningInternal, request); time.before += before.ms; time.after += after.ms; @@ -73,8 +80,10 @@ for (let seed = Number(first); seed < Number(first) + Number(count); seed++) { if (category !== 'identical') console.log(`${seed}\t${category}${detail}`); counts.set(category, (counts.get(category) ?? 0) + 1); } -console.log(`${kind} requests, seeds ${first}..${Number(first) + Number(count) - 1}`); +console.log( + `${kind} requests, seeds ${first}..${Number(first) + Number(count) - 1}, against the ${baselineName} engine`, +); for (const [category, total] of [...counts].sort()) console.log(`${category}\t${total}`); console.log( - `calibration engine ${(time.before / 1000).toFixed(1)} s, current engine ${(time.after / 1000).toFixed(1)} s`, + `${baselineName} engine ${(time.before / 1000).toFixed(1)} s, current engine ${(time.after / 1000).toFixed(1)} s`, );