test(benchmark): tighten the relative gate to the measured speed
After #302, the development machine's protocol takes about 41 seconds instead of 232. The budget multipliers now add about 1.5 times headroom to the new ratios (0.1 for typed obligations, which leaves room for runner jitter), and the hang guard drops from 600 to 240 seconds, with a 210-second test timeout: about three times the expected run on srvhome, which is about 1.9 times slower than nessar-desktop. Part of #286 and #302. The gate still needs a passing CI benchmark on two runners. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
nessarandClaude Opus 5.5 committed
1 parent
6c9ff4ae34
commit
676ea24410
7 files changed
+75
-31
No files matched your search
@@ -107,7 +107,7 @@ jobs:
|
||||
- run: bun install --frozen-lockfile
|
||||
- run: bun scripts/generate-langs.js
|
||||
- name: Benchmark protocol with relative gate
|
||||
run: timeout 600s bun test src/app/benchmark --timeout=600000
|
||||
run: timeout 240s bun test src/app/benchmark --timeout=240000
|
||||
|
||||
build:
|
||||
name: build
|
||||
|
||||
@@ -216,7 +216,7 @@ Open <http://localhost:4200>. The development server reloads when source files c
|
||||
| `bun run watch` | Rebuild the development bundle when source files change. |
|
||||
| `bun run test` | Run the unit, integration and benchmark suites once. |
|
||||
| `bun test --isolate --path-ignore-patterns='**/benchmark/**'` | Run the functional suite only. |
|
||||
| `bun test src/app/benchmark --timeout=600000` | Run the planning benchmark (no coverage). |
|
||||
| `bun test src/app/benchmark --timeout=240000` | Run the planning benchmark (no coverage). |
|
||||
| `bun test --config=bunfig.coverage.toml --isolate --path-ignore-patterns='**/benchmark/**' && bun scripts/coverage-reports.mjs` | Functional coverage: text, `lcov.info`, HTML and Cobertura in `coverage/`. Line coverage only; frozen engine excluded. |
|
||||
| `bun run lint` | Run ESLint (`bun run lint:fix` applies its fixes). |
|
||||
| `bun run format:check` | Check Prettier formatting of the application, the CI scripts and the workflow (`bun run format` writes it). |
|
||||
@@ -232,7 +232,7 @@ Angular compiles with TypeScript 6. `bun check` is Bun's TypeScript 7 checker fo
|
||||
|
||||
Build or start the application, then open `/benchmark`. The isolated harness covers a representative year, typed obligations, chained earned entitlements, a deliberate conflict, a loaded year, and two- and three-year horizons with dated annual balances. It reports preparation time, planning latency, planning-work time, total wall time through explanation delivery, UI timer delay, available main-thread heap measurements, outcome, last completed stage, placement count, and determinism.
|
||||
|
||||
The release protocol, recorded desktop measurements, and hardware limitations are documented in [docs/benchmark/README.md](docs/benchmark/README.md). No reference-phone run or individual multi-year duration ceiling is established. CI runs the complete seven-scenario protocol with `bun test` and requires each scenario to stay within its budget relative to a frozen-engine calibration, with a 600-second hang guard. Bun measurements do not establish browser or reference-phone speedups.
|
||||
The release protocol, recorded desktop measurements, and hardware limitations are documented in [docs/benchmark/README.md](docs/benchmark/README.md). No reference-phone run or individual multi-year duration ceiling is established. CI runs the complete seven-scenario protocol with `bun test` and requires each scenario to stay within its budget relative to a frozen-engine calibration, with a 240-second hang guard. Bun measurements do not establish browser or reference-phone speedups.
|
||||
|
||||
## Docker
|
||||
|
||||
@@ -253,7 +253,7 @@ The workflow runs on branch/tag pushes, PR opening/reopening/synchronization/edi
|
||||
|
||||
Jobs that run project commands install Bun 1.4.3 with `scripts/install-bun.sh` (a checksum-verified download, no npm) and run their steps with a `PATH` that holds Bun but no Node. The job containers keep Node only for Gitea's checkout and artifact actions, which are JavaScript actions the runner executes itself; see the [activation guide](docs/gitea-actions.md#runners-and-permissions).
|
||||
|
||||
Functional tests run with `bun test --isolate`; coverage is line-level only (Bun reports no branch or function data). The separate required benchmark job runs without coverage in a container limited to 3 GiB (JavaScriptCore has no heap-size flag, so this replaces the former 2 GiB Node heap cap) and keeps its 600-second hang guard, one warm-up and two measured repetitions of all seven scenarios. [#78](https://gitea.nessar.fr/nessar/vac-optimizer/issues/78) restores this gate after a complete local pass in 298.6 seconds; see [the measurements and hardware limits](docs/benchmark/issue-78.md). Benchmark success is required for image publication and the aggregate `required` job. The narrow local timing margin still needs confirmation on the CI runner.
|
||||
Functional tests run with `bun test --isolate`; coverage is line-level only (Bun reports no branch or function data). The separate required benchmark job runs without coverage in a container limited to 3 GiB (JavaScriptCore has no heap-size flag, so this replaces the former 2 GiB Node heap cap) and keeps its 240-second hang guard, one warm-up and two measured repetitions of all seven scenarios. [#78](https://gitea.nessar.fr/nessar/vac-optimizer/issues/78) restores this gate after a complete local pass in 298.6 seconds; see [the measurements and hardware limits](docs/benchmark/issue-78.md). Benchmark success is required for image publication and the aggregate `required` job. The narrow local timing margin still needs confirmation on the CI runner.
|
||||
|
||||
The `required` job aggregates successful validation and expected image publication, accepting the image job's intentional skip for fork PRs and manual runs. Image jobs require successful validation and are limited to pushes and same-repository PRs. Registry credentials are confined to the publishing step.
|
||||
|
||||
|
||||
+24
-18
@@ -6,7 +6,7 @@
|
||||
unchanged Node acceptance protocol on the documented development laptop:
|
||||
**298.6 seconds**, including one warm-up and two measured repetitions of all
|
||||
seven scenarios, without coverage, with one worker and a 2 GiB Node heap.
|
||||
The full protocol keeps a 600-second timeout only as a hang guard (see
|
||||
The full protocol keeps a 240-second timeout only as a hang guard (see
|
||||
[Relative CI gate](#relative-ci-gate)). All expected outcomes,
|
||||
placement counts, final stages and determinism checks pass. The fixture inputs
|
||||
are unchanged; the test now prints its existing report.
|
||||
@@ -25,8 +25,8 @@ bundle never changes, so ratios stay comparable with the recorded history; the
|
||||
differential check's frozen bundle can be rebuilt independently.
|
||||
Each scenario's budget is its multiplier times that calibration; the mean
|
||||
`planningMs` of its two measured repetitions must not exceed it. A scenario
|
||||
without a multiplier fails too. `#286` tightens the multipliers once the speed
|
||||
target is met.
|
||||
without a multiplier fails too. #286 tightened the multipliers once #302 met
|
||||
the speed targets.
|
||||
|
||||
The job log shows a Markdown table with one row per scenario, printed as soon
|
||||
as that scenario finishes. Each row gives both measured planning times, their
|
||||
@@ -36,18 +36,22 @@ tab-separated report follows the table.
|
||||
|
||||
| Scenario | Mean planning (s) | Ratio to calibration | Multiplier |
|
||||
| --------------------------- | ----------------- | -------------------- | ---------- |
|
||||
| representative year | 0.40 | 0.15 | 0.25 |
|
||||
| typed obligations | 5.02 | 1.83 | 3 |
|
||||
| chained earned entitlements | 0.025 | 0.01 | 0.05 |
|
||||
| deliberate conflict | 0.013 | 0.005 | 0.05 |
|
||||
| loaded year | 44.3 | 16.2 | 25 |
|
||||
| two years | 1.84 | 0.67 | 1 |
|
||||
| three years | 5.94 | 2.17 | 3.5 |
|
||||
| representative year | 0.24 | 0.07 | 0.15 |
|
||||
| typed obligations | 0.10 | 0.03 | 0.1 |
|
||||
| chained earned entitlements | 0.01 | 0.00 | 0.05 |
|
||||
| deliberate conflict | 0.00 | 0.00 | 0.05 |
|
||||
| loaded year | 5.51 | 1.67 | 2.5 |
|
||||
| two years | 0.93 | 0.28 | 0.5 |
|
||||
| three years | 3.11 | 0.94 | 1.5 |
|
||||
|
||||
Ratios come from the development machine (calibration about 2.74 s); the
|
||||
multipliers add roughly 1.5x headroom, rounded up, with a floor of 0.05 for the
|
||||
two tiny scenarios. A failure lists, per scenario, the measured mean, the
|
||||
budget, the multiplier and the calibration in milliseconds.
|
||||
Ratios come from the development machine's Bun protocol after #302
|
||||
(calibration 3.30 s, [record](issue-302.md#benchmark-protocol)); the multipliers
|
||||
add roughly 1.5x headroom, rounded up, with a floor of 0.05 for the two tiny
|
||||
scenarios. Typed obligations gets 0.1: its 0.1-second run would leave a 0.05
|
||||
budget only about 80 ms of headroom for runner jitter. Before #302, the
|
||||
multipliers were 0.25, 3, 0.05, 0.05, 25, 1 and 3.5. A failure lists, per
|
||||
scenario, the measured mean, the budget, the multiplier and the calibration in
|
||||
milliseconds.
|
||||
|
||||
The first CI runs of PR #287 (commit `63b50c4`) give consistent ratios on two
|
||||
runners:
|
||||
@@ -66,12 +70,14 @@ runners:
|
||||
srvhome is about 1.9 times slower than nessar-desktop, so its whole protocol
|
||||
needs roughly 400 seconds. The original 300-second `timeout` killed it during
|
||||
the loaded year, while every completed scenario was within budget. The CI
|
||||
`timeout` is therefore 600 seconds and the test timeout 570 seconds. They are
|
||||
only hang guards and do not judge speed; #286 lowers them once the speed
|
||||
target is met.
|
||||
`timeout` was therefore 600 seconds and the test timeout 570 seconds. After
|
||||
#302 the whole local protocol takes about 41 seconds, so srvhome should need
|
||||
about 80. #286 lowered the CI `timeout` to 240 seconds and the test timeout to
|
||||
210 seconds, about three times that. They are only hang guards and do not
|
||||
judge speed.
|
||||
|
||||
The protocol now runs on Bun's test runner, outside coverage:
|
||||
`timeout 600s bun test src/app/benchmark --timeout=600000`. JavaScriptCore has
|
||||
`timeout 240s bun test src/app/benchmark --timeout=240000`. JavaScriptCore has
|
||||
no equivalent of Node's `--max-old-space-size`, so the CI benchmark job enforces
|
||||
memory with a 3 GiB container limit instead of the former 2 GiB heap cap.
|
||||
Scenarios, calibration, warm-up, repetitions, determinism checks and
|
||||
|
||||
@@ -240,3 +240,41 @@ On these larger plans, ranking loses a real objective more often than on the
|
||||
small requests: 4 in 50, against 9 in 5,000. The synchronous calibration engine
|
||||
took 3,924 seconds for the 50 requests, about 78 seconds each, and the current
|
||||
engine 289 seconds, about 5.8 seconds each.
|
||||
|
||||
## Benchmark protocol
|
||||
|
||||
The unchanged benchmark protocol ran on the development machine before the
|
||||
first exact step (`b356117`, from a separate worktree) and after the frozen
|
||||
engine rebuild, one after the other in the same session, with no other
|
||||
workload:
|
||||
|
||||
```sh
|
||||
timeout 600s bun test src/app/benchmark --timeout=600000
|
||||
```
|
||||
|
||||
| Scenario | Before (s) | After (s) | Target (s) |
|
||||
| --------------------------- | ---------: | --------: | ---------: |
|
||||
| representative year | 0.45 | 0.24 | 3 |
|
||||
| typed obligations | 4.70 | 0.10 | 3 |
|
||||
| chained earned entitlements | 0.03 | 0.01 | 3 |
|
||||
| deliberate conflict | 0.01 | 0.00 | 3 |
|
||||
| loaded year | 57.25 | 5.51 | 10 |
|
||||
| two years | 2.19 | 0.93 | 5 |
|
||||
| three years | 7.03 | 3.11 | 5 |
|
||||
| calibration (`two years`) | 3.40 | 3.30 | |
|
||||
| whole test run | 232.37 | 40.78 | |
|
||||
|
||||
Each figure is the mean of the two measured calculations. All seven scenarios
|
||||
keep their outcomes, placement counts, completed stages and determinism, and
|
||||
every scenario now meets its target.
|
||||
|
||||
### Gate
|
||||
|
||||
For [#286](https://gitea.nessar.fr/nessar/vac-optimizer/issues/286), the
|
||||
multipliers now add about 1.5 times headroom to these ratios:
|
||||
0.15 for the representative year, 0.1 for typed obligations, 0.05 for the two
|
||||
tiny scenarios, 2.5 for the loaded year, 0.5 for two years and 1.5 for three years.
|
||||
srvhome is about 1.9 times slower than nessar-desktop, so it should need about
|
||||
80 seconds for the protocol. The CI `timeout` is now 240 seconds and the test
|
||||
timeout 210 seconds, about three times that. The CI benchmark has not yet run
|
||||
on the Gitea runners with these values.
|
||||
@@ -21,7 +21,7 @@ configuration; the Compose files remain manual deployment options.
|
||||
| `policy` | Tests the policy and publisher with Bun, then validates the checked-out commit, GitFlow, PR title, commits, and stable release version. |
|
||||
| `validate` | Runs ESLint, Prettier (application, `scripts/ci-*.mjs` and workflow files) and `bun check` type checks of application and test code, all on Bun with the frozen lockfile. |
|
||||
| `test` | Runs functional coverage with `bun test --isolate` (line coverage only), converts it to HTML and Cobertura, and uploads coverage for seven days even on failure. |
|
||||
| `benchmark` | Required seven-scenario protocol (one warm-up, two measured repetitions) outside coverage in a container limited to 3 GiB. It first times the frozen engine on `two years` (best of three) and fails any scenario whose mean planning time exceeds its multiplier times that calibration; the 600-second timeout is only a hang guard. |
|
||||
| `benchmark` | Required seven-scenario protocol (one warm-up, two measured repetitions) outside coverage in a container limited to 3 GiB. It first times the frozen engine on `two years` (best of three) and fails any scenario whose mean planning time exceeds its multiplier times that calibration; the 240-second timeout is only a hang guard. |
|
||||
| `build` | Builds the client-only Angular production bundle (the only check of Angular templates; `bun check` does not replace it) and the Bun static-server bundle, and retains `dist` for seven days. |
|
||||
| `security` | Runs Semgrep JavaScript/TypeScript SAST and Trivy dependency/secret scans; retains SARIF reports for seven days. Scanner execution must succeed; findings remain reports rather than a new severity gate. The scanners are not project JavaScript, so this job installs no Bun. |
|
||||
| `images` | Publishes branch/SHA images after required validation, including the benchmark, or promotes an existing SHA image for a stable tag, running the publisher with Bun. Runs only for pushes and same-repository PRs. |
|
||||
@@ -32,7 +32,7 @@ publication (`images.needs` includes `benchmark`) and the `required` aggregate.
|
||||
The relative gate absorbs a slower runner and load that lasts through the whole
|
||||
job, because calibration runs in the same job. It does not absorb load that
|
||||
starts or stops during the protocol, such as an overlapping push and
|
||||
pull_request run of the same commit; the 600-second hang guard still bounds
|
||||
pull_request run of the same commit; the 240-second hang guard still bounds
|
||||
those runs.
|
||||
|
||||
Push and PR runs are separate so PR title and rebase checks run on the PR event.
|
||||
|
||||
@@ -134,5 +134,5 @@ describe('leave planner benchmark protocol', () => {
|
||||
} finally {
|
||||
vi.stubGlobal('window', browserWindow);
|
||||
}
|
||||
}, 570_000);
|
||||
}, 210_000);
|
||||
});
|
||||
@@ -301,15 +301,15 @@ export function benchmarkReport(measurements: BenchmarkMeasurement[]): string {
|
||||
].join('\n');
|
||||
}
|
||||
|
||||
// Measured on the development machine (ratio to the frozen two-years run), tightened by #286.
|
||||
// About 1.5x the development machine's ratios to the calibration run after #302 (#286).
|
||||
export const BENCHMARK_BUDGET_MULTIPLIERS: Record<string, number> = {
|
||||
'representative year': 0.25,
|
||||
'typed obligations': 3,
|
||||
'representative year': 0.15,
|
||||
'typed obligations': 0.1,
|
||||
'chained earned entitlements': 0.05,
|
||||
'deliberate conflict': 0.05,
|
||||
'loaded year': 25,
|
||||
'two years': 1,
|
||||
'three years': 3.5,
|
||||
'loaded year': 2.5,
|
||||
'two years': 0.5,
|
||||
'three years': 1.5,
|
||||
};
|
||||
|
||||
/** One message per scenario whose mean planning time exceeds multiplier x calibration, or that has no multiplier. */
|
||||
|
||||
Reference in new issue
Block a user