test(benchmark): calibrate again after the protocol and judge against the slower value #308

Merged

nessar merged 1 commits from feature/benchmark-calibration-bracket into develop 2026-10-19 10:24:14 +02:00

Conversation 0 Commits 1 Files Changed 3

+51 -25

nessar commented 2026-10-19 10:11:27 +02:00

Owner

Copy link

Copy source

The benchmark gate calibrates the frozen engine once, before the scenarios. Load that starts afterwards slows the scenarios but not the yardstick. This is what failed the pull_request run of #307: two years took 1653.1 ms against a 1648.7 ms budget (0.3% over). It was an unrelated change, and the same commit's push run passed. The push and pull_request runs of one commit overlap on the single runner, next to the test, build, security and image jobs.

Change

The spec repeats the calibration after the protocol and judges every budget against the slower of the two values.

• The budgets can only loosen, never tighten, so this cannot create a new failure.
• The live rows still use the first calibration. When the second is slower, the log prints it and the summary table again with the budgets actually applied, so no row reads OVER while the job passes.
• docs/gitea-actions.md and docs/benchmark/README.md describe the new protocol. Prettier also reflowed the job table in gitea-actions.md, which accounts for most of that file's diff.

Verification

Local runs of bun test src/app/benchmark --timeout=240000, with CPU load starting after the first calibration (the failure mode in CI):

• Load that ends before the end of the protocol: fails as before (loaded year, two years, three years over budget). The second calibration is quiet, so nothing changes. This is the documented limit.
• Load that lasts through the end: calibration 3015 ms before, 7598 ms after; the three scenarios that read OVER on the first table pass on the reprinted one. Without load the benchmark passes in 45 s and prints no extra block.

Not covered

Load that ends before the second calibration is not absorbed. The multipliers are unchanged; the headroom of three years (1.36x) and loaded year (1.43x) over quiet CI runs is unchanged.

The benchmark gate calibrates the frozen engine once, before the scenarios. Load that starts afterwards slows the scenarios but not the yardstick. This is what failed the pull_request run of #307: `two years` took 1653.1 ms against a 1648.7 ms budget (0.3% over). It was an unrelated change, and the same commit's push run passed. The push and pull_request runs of one commit overlap on the single runner, next to the test, build, security and image jobs. ## Change The spec repeats the calibration after the protocol and judges every budget against the slower of the two values. - The budgets can only loosen, never tighten, so this cannot create a new failure. - The live rows still use the first calibration. When the second is slower, the log prints it and the summary table again with the budgets actually applied, so no row reads `OVER` while the job passes. - `docs/gitea-actions.md` and `docs/benchmark/README.md` describe the new protocol. Prettier also reflowed the job table in `gitea-actions.md`, which accounts for most of that file's diff. ## Verification Local runs of `bun test src/app/benchmark --timeout=240000`, with CPU load starting after the first calibration (the failure mode in CI): - Load that ends before the end of the protocol: fails as before (`loaded year`, `two years`, `three years` over budget). The second calibration is quiet, so nothing changes. This is the documented limit. - Load that lasts through the end: calibration 3015 ms before, 7598 ms after; the three scenarios that read `OVER` on the first table pass on the reprinted one. Without load the benchmark passes in 45 s and prints no extra block. ## Not covered Load that ends before the second calibration is not absorbed. The multipliers are unchanged; the headroom of `three years` (1.36x) and `loaded year` (1.43x) over quiet CI runs is unchanged.

nessar added 1 commit 2026-10-19 10:11:27 +02:00

test(benchmark): calibrate again after the protocol and judge against the slower value ... b37101435a

The relative gate calibrated once before the scenarios, so load that
started afterwards (the push and pull_request runs of one commit overlap
on the same runner, next to the test, build and image jobs) slowed the
scenarios but not the yardstick. A pull_request run failed `two years`
at 1653.1 ms against a 1648.7 ms budget this way.

The spec now repeats the calibration after the protocol and uses the
slower of the two values. The budgets can only loosen, never tighten,
and the log reprints the table with the budgets actually applied when
they change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

nessar merged commit abfbca1d45 into develop 2026-10-19 10:24:14 +02:00

nessar deleted branch feature/benchmark-calibration-bracket 2026-10-19 10:24:21 +02:00

nessar referenced this pull request 2026-10-19 10:28:59 +02:00

Complete and verify the Bun CI cutover #293

nessar referenced this pull request 2026-10-19 10:29:07 +02:00

fix: stabilize two CI test failures (Paris today, worker cancel race) #307

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

bug

dependencies

docker

documentation

duplicate

enhancement

github_actions

good first issue

help wanted

invalid

javascript

needs-info

needs-triage

question

ready-for-agent

wontfix

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate (renovate) Bot nessar (nessar)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: nessar/vac-optimizer#308