ci(benchmark): size the hang guard for the slowest runner
CI / policy (push) Successful in 6s
CI / validate (push) Successful in 53s
CI / build (push) Successful in 1m18s
CI / policy (pull_request) Successful in 9s
CI / security (push) Successful in 1m12s
CI / validate (pull_request) Successful in 1m18s
CI / benchmark (push) Successful in 3m56s
CI / test (push) Successful in 4m6s
CI / build (pull_request) Successful in 36s
CI / security (pull_request) Successful in 1m5s
CI / test (pull_request) Successful in 3m2s
CI / images (push) Successful in 1m15s
CI / required (push) Successful in 1s
CI / benchmark (pull_request) Successful in 7m18s
CI / images (pull_request) Successful in 1m50s
CI / required (pull_request) Successful in 1s

The first PR run landed on srvhome, about 1.9 times slower than
nessar-desktop. The fixed 300-second timeout killed its protocol during the
loaded year, although every completed scenario was within its relative
budget. Raise the CI timeout to 600 seconds and the test timeout to 570
seconds so they only guard against hangs, and record the first CI-runner
ratios.

Refs #285

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
nessarandClaude Opus 5.5 committed 2026-10-17 01:40:41 +02:00
1 parent 63b50c4311
commit cfdb28c90a
5 files changed
+27 -10

No files matched your search

+1 -1
View File
@@ -86,7 +86,7 @@ jobs:
- run: bun install --frozen-lockfile
- run: bun scripts/generate-langs.js
- name: Benchmark protocol with relative gate
run: timeout 300s bunx vitest run src/app/benchmark/ --maxWorkers=1
run: timeout 600s bunx vitest run src/app/benchmark/ --maxWorkers=1
build:
name: build
+22 -5
View File
@@ -6,7 +6,7 @@
unchanged Node acceptance protocol on the documented development laptop:
**298.6 seconds**, including one warm-up and two measured repetitions of all
seven scenarios, without coverage, with one worker and a 2 GiB Node heap.
The full protocol keeps a 300-second timeout only as a hang guard (see
The full protocol keeps a 600-second timeout only as a hang guard (see
[Relative CI gate](#relative-ci-gate)). All expected outcomes,
placement counts, final stages and determinism checks pass. The fixture inputs
are unchanged; the test now prints its existing report.
@@ -39,11 +39,28 @@ repetitions must not exceed it. A scenario without a multiplier fails too.
Ratios come from the development machine (calibration about 2.74 s); the
multipliers add roughly 1.5x headroom, rounded up, with a floor of 0.05 for the
two tiny scenarios. A failure lists, per scenario, the measured mean, the
budget, the multiplier and the calibration in milliseconds. The 300-second
`timeout` in CI is only a hang guard.
budget, the multiplier and the calibration in milliseconds.
TODO: measure the ratios on at least one CI runner; no runner numbers are
recorded yet.
The first CI runs of PR #287 (commit `63b50c4`) give consistent ratios on two
runners:
| Scenario | nessar-desktop | srvhome |
| --------------------------- | -------------- | ------- |
| calibration | 3.03 s | 5.71 s |
| representative year | 0.17 | 0.14 |
| typed obligations | 1.83 | 1.75 |
| chained earned entitlements | 0.011 | 0.005 |
| deliberate conflict | 0.006 | 0.003 |
| loaded year | 16.6 | killed |
| two years | 0.65 | killed |
| three years | 1.99 | killed |
srvhome is about 1.9 times slower than nessar-desktop, so its whole protocol
needs roughly 400 seconds. The original 300-second `timeout` killed it during
the loaded year, while every completed scenario was within budget. The CI
`timeout` is therefore 600 seconds and the test timeout 570 seconds. They are
only hang guards and do not judge speed; #286 lowers them once the speed
target is met.
The post-removal functional integration checks, browser QA and unchanged-input
handoff for #78 are recorded in [the #77 verification report](../trace-free-planning-verification.md).
+1 -1
View File
@@ -159,7 +159,7 @@ seconds (−33%), three years from 10.2 to 5.9 seconds and two years from 2.9 to
from 262.0 to 180.3 seconds.
Because the target is not met, the #285 multipliers stay at current speed and
the 300-second hang-guard timeout stays as it is. #286 remains open for a
the hang-guard timeout is not lowered. #286 remains open for a
separate decision on search changes.
### Remaining loaded-year profile
+2 -2
View File
@@ -21,7 +21,7 @@ configuration; the Compose files remain manual deployment options.
| `policy` | Tests the policy and publisher, then validates the checked-out commit, GitFlow, PR title, commits, and stable release version. |
| `validate` | Runs lint, formatting, and application/test type checks. |
| `test` | Runs functional coverage with one worker and a 2 GiB Node heap; uploads coverage for seven days even on failure. |
| `benchmark` | Required seven-scenario protocol (one warm-up, two measured repetitions) outside coverage with one worker and a 2 GiB Node heap. It first times the frozen engine on `two years` (best of three) and fails any scenario whose mean planning time exceeds its multiplier times that calibration; the 300-second timeout is only a hang guard. |
| `benchmark` | Required seven-scenario protocol (one warm-up, two measured repetitions) outside coverage with one worker and a 2 GiB Node heap. It first times the frozen engine on `two years` (best of three) and fails any scenario whose mean planning time exceeds its multiplier times that calibration; the 600-second timeout is only a hang guard. |
| `build` | Builds browser/server production bundles and retains `dist` for seven days. |
| `security` | Runs Semgrep JavaScript/TypeScript SAST and Trivy dependency/secret scans; retains SARIF reports for seven days. Scanner execution must succeed; findings remain reports rather than a new severity gate. |
| `images` | Publishes branch/SHA images after required validation, including the benchmark, or promotes an existing SHA image for a stable tag. Runs only for pushes and same-repository PRs. |
@@ -32,7 +32,7 @@ publication (`images.needs` includes `benchmark`) and the `required` aggregate.
The relative gate absorbs a slower runner and load that lasts through the whole
job, because calibration runs in the same job. It does not absorb load that
starts or stops during the protocol, such as an overlapping push and
pull_request run of the same commit; the 300-second hang guard still bounds
pull_request run of the same commit; the 600-second hang guard still bounds
those runs.
Push and PR runs are separate so PR title and rebase checks run on the PR event.
@@ -129,5 +129,5 @@ describe('leave planner benchmark protocol', () => {
} finally {
vi.stubGlobal('window', browserWindow);
}
}, 300_000);
}, 570_000);
});