test(benchmark): tighten the relative gate to the measured speed
After #302, the development machine's protocol takes about 41 seconds instead of 232. The budget multipliers now add about 1.5 times headroom to the new ratios (0.1 for typed obligations, which leaves room for runner jitter), and the hang guard drops from 600 to 240 seconds, with a 210-second test timeout: about three times the expected run on srvhome, which is about 1.9 times slower than nessar-desktop. Part of #286 and #302. The gate still needs a passing CI benchmark on two runners. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
nessarandClaude Opus 5.5 committed
1 parent
6c9ff4ae34
commit
676ea24410
7 files changed
+75
-31
No files matched your search
+24
-18
@@ -6,7 +6,7 @@
|
||||
unchanged Node acceptance protocol on the documented development laptop:
|
||||
**298.6 seconds**, including one warm-up and two measured repetitions of all
|
||||
seven scenarios, without coverage, with one worker and a 2 GiB Node heap.
|
||||
The full protocol keeps a 600-second timeout only as a hang guard (see
|
||||
The full protocol keeps a 240-second timeout only as a hang guard (see
|
||||
[Relative CI gate](#relative-ci-gate)). All expected outcomes,
|
||||
placement counts, final stages and determinism checks pass. The fixture inputs
|
||||
are unchanged; the test now prints its existing report.
|
||||
@@ -25,8 +25,8 @@ bundle never changes, so ratios stay comparable with the recorded history; the
|
||||
differential check's frozen bundle can be rebuilt independently.
|
||||
Each scenario's budget is its multiplier times that calibration; the mean
|
||||
`planningMs` of its two measured repetitions must not exceed it. A scenario
|
||||
without a multiplier fails too. `#286` tightens the multipliers once the speed
|
||||
target is met.
|
||||
without a multiplier fails too. #286 tightened the multipliers once #302 met
|
||||
the speed targets.
|
||||
|
||||
The job log shows a Markdown table with one row per scenario, printed as soon
|
||||
as that scenario finishes. Each row gives both measured planning times, their
|
||||
@@ -36,18 +36,22 @@ tab-separated report follows the table.
|
||||
|
||||
| Scenario | Mean planning (s) | Ratio to calibration | Multiplier |
|
||||
| --------------------------- | ----------------- | -------------------- | ---------- |
|
||||
| representative year | 0.40 | 0.15 | 0.25 |
|
||||
| typed obligations | 5.02 | 1.83 | 3 |
|
||||
| chained earned entitlements | 0.025 | 0.01 | 0.05 |
|
||||
| deliberate conflict | 0.013 | 0.005 | 0.05 |
|
||||
| loaded year | 44.3 | 16.2 | 25 |
|
||||
| two years | 1.84 | 0.67 | 1 |
|
||||
| three years | 5.94 | 2.17 | 3.5 |
|
||||
| representative year | 0.24 | 0.07 | 0.15 |
|
||||
| typed obligations | 0.10 | 0.03 | 0.1 |
|
||||
| chained earned entitlements | 0.01 | 0.00 | 0.05 |
|
||||
| deliberate conflict | 0.00 | 0.00 | 0.05 |
|
||||
| loaded year | 5.51 | 1.67 | 2.5 |
|
||||
| two years | 0.93 | 0.28 | 0.5 |
|
||||
| three years | 3.11 | 0.94 | 1.5 |
|
||||
|
||||
Ratios come from the development machine (calibration about 2.74 s); the
|
||||
multipliers add roughly 1.5x headroom, rounded up, with a floor of 0.05 for the
|
||||
two tiny scenarios. A failure lists, per scenario, the measured mean, the
|
||||
budget, the multiplier and the calibration in milliseconds.
|
||||
Ratios come from the development machine's Bun protocol after #302
|
||||
(calibration 3.30 s, [record](issue-302.md#benchmark-protocol)); the multipliers
|
||||
add roughly 1.5x headroom, rounded up, with a floor of 0.05 for the two tiny
|
||||
scenarios. Typed obligations gets 0.1: its 0.1-second run would leave a 0.05
|
||||
budget only about 80 ms of headroom for runner jitter. Before #302, the
|
||||
multipliers were 0.25, 3, 0.05, 0.05, 25, 1 and 3.5. A failure lists, per
|
||||
scenario, the measured mean, the budget, the multiplier and the calibration in
|
||||
milliseconds.
|
||||
|
||||
The first CI runs of PR #287 (commit `63b50c4`) give consistent ratios on two
|
||||
runners:
|
||||
@@ -66,12 +70,14 @@ runners:
|
||||
srvhome is about 1.9 times slower than nessar-desktop, so its whole protocol
|
||||
needs roughly 400 seconds. The original 300-second `timeout` killed it during
|
||||
the loaded year, while every completed scenario was within budget. The CI
|
||||
`timeout` is therefore 600 seconds and the test timeout 570 seconds. They are
|
||||
only hang guards and do not judge speed; #286 lowers them once the speed
|
||||
target is met.
|
||||
`timeout` was therefore 600 seconds and the test timeout 570 seconds. After
|
||||
#302 the whole local protocol takes about 41 seconds, so srvhome should need
|
||||
about 80. #286 lowered the CI `timeout` to 240 seconds and the test timeout to
|
||||
210 seconds, about three times that. They are only hang guards and do not
|
||||
judge speed.
|
||||
|
||||
The protocol now runs on Bun's test runner, outside coverage:
|
||||
`timeout 600s bun test src/app/benchmark --timeout=600000`. JavaScriptCore has
|
||||
`timeout 240s bun test src/app/benchmark --timeout=240000`. JavaScriptCore has
|
||||
no equivalent of Node's `--max-old-space-size`, so the CI benchmark job enforces
|
||||
memory with a 3 GiB container limit instead of the former 2 GiB heap cap.
|
||||
Scenarios, calibration, warm-up, repetitions, determinism checks and
|
||||
|
||||
@@ -240,3 +240,41 @@ On these larger plans, ranking loses a real objective more often than on the
|
||||
small requests: 4 in 50, against 9 in 5,000. The synchronous calibration engine
|
||||
took 3,924 seconds for the 50 requests, about 78 seconds each, and the current
|
||||
engine 289 seconds, about 5.8 seconds each.
|
||||
|
||||
## Benchmark protocol
|
||||
|
||||
The unchanged benchmark protocol ran on the development machine before the
|
||||
first exact step (`b356117`, from a separate worktree) and after the frozen
|
||||
engine rebuild, one after the other in the same session, with no other
|
||||
workload:
|
||||
|
||||
```sh
|
||||
timeout 600s bun test src/app/benchmark --timeout=600000
|
||||
```
|
||||
|
||||
| Scenario | Before (s) | After (s) | Target (s) |
|
||||
| --------------------------- | ---------: | --------: | ---------: |
|
||||
| representative year | 0.45 | 0.24 | 3 |
|
||||
| typed obligations | 4.70 | 0.10 | 3 |
|
||||
| chained earned entitlements | 0.03 | 0.01 | 3 |
|
||||
| deliberate conflict | 0.01 | 0.00 | 3 |
|
||||
| loaded year | 57.25 | 5.51 | 10 |
|
||||
| two years | 2.19 | 0.93 | 5 |
|
||||
| three years | 7.03 | 3.11 | 5 |
|
||||
| calibration (`two years`) | 3.40 | 3.30 | |
|
||||
| whole test run | 232.37 | 40.78 | |
|
||||
|
||||
Each figure is the mean of the two measured calculations. All seven scenarios
|
||||
keep their outcomes, placement counts, completed stages and determinism, and
|
||||
every scenario now meets its target.
|
||||
|
||||
### Gate
|
||||
|
||||
For [#286](https://gitea.nessar.fr/nessar/vac-optimizer/issues/286), the
|
||||
multipliers now add about 1.5 times headroom to these ratios:
|
||||
0.15 for the representative year, 0.1 for typed obligations, 0.05 for the two
|
||||
tiny scenarios, 2.5 for the loaded year, 0.5 for two years and 1.5 for three years.
|
||||
srvhome is about 1.9 times slower than nessar-desktop, so it should need about
|
||||
80 seconds for the protocol. The CI `timeout` is now 240 seconds and the test
|
||||
timeout 210 seconds, about three times that. The CI benchmark has not yet run
|
||||
on the Gitea runners with these values.
|
||||
Reference in new issue
Block a user