nessar/vac-optimizer
1
0 Fork 0
Code Issues 14 Pull Requests 3 Actions Packages Projects 1 Releases Wiki Activity

ci(benchmark): gate scenario speed relative to the runner #285

New Issue
Closed
opened 2026-10-16 22:39:46 +02:00 by nessar · 1 comment
nessar commented 2026-10-16 22:39:46 +02:00
Owner
Copy link
Copy source

Parent

#279

What to build

The CI benchmark judges speed relative to the runner it lands on. Before the protocol, the job times the frozen engine on the two-year scenario, best of three, in the same job. Each scenario's budget is a multiple of that calibration. A slower or busy runner no longer fails a correct change, and a real regression still fails on a fast runner. The 300-second outer timeout remains for now as a hang guard. This replaces the absolute 300-second acceptance protocol named in #78, so the policy change is documented with it.

Acceptance criteria

  • The calibration uses the frozen engine from the differential-check ticket, takes the best of three runs, and its value is printed in the job log.
  • Every scenario has a budget expressed as a multiple of the calibration. Multipliers are set from ratios measured on the development machine and at least one CI runner. The loaded year's final multiplier is about four times, matching the 10-second target. Until the speed target is met, multipliers may reflect current speed and are tightened by the acceptance ticket.
  • A scenario over budget fails the job with its measured time, budget and calibration in the message.
  • Scenarios, inputs, warm-ups, repetitions, outcome assertions and the 2 GiB heap limit are unchanged.
  • The benchmark documentation and the CI documentation describe the relative gate as the new benchmark policy, replacing the 300-second protocol.
  • The job passes on at least two different CI runners.
  • Record a decision on whether the benchmark keeps running on both push and pull_request events, since the two runs of PR #278's head commit competed on one runner.

Blocked by

## Parent #279 ## What to build The CI benchmark judges speed relative to the runner it lands on. Before the protocol, the job times the frozen engine on the two-year scenario, best of three, in the same job. Each scenario's budget is a multiple of that calibration. A slower or busy runner no longer fails a correct change, and a real regression still fails on a fast runner. The 300-second outer timeout remains for now as a hang guard. This replaces the absolute 300-second acceptance protocol named in #78, so the policy change is documented with it. ## Acceptance criteria - [ ] The calibration uses the frozen engine from the differential-check ticket, takes the best of three runs, and its value is printed in the job log. - [ ] Every scenario has a budget expressed as a multiple of the calibration. Multipliers are set from ratios measured on the development machine and at least one CI runner. The loaded year's final multiplier is about four times, matching the 10-second target. Until the speed target is met, multipliers may reflect current speed and are tightened by the acceptance ticket. - [ ] A scenario over budget fails the job with its measured time, budget and calibration in the message. - [ ] Scenarios, inputs, warm-ups, repetitions, outcome assertions and the 2 GiB heap limit are unchanged. - [ ] The benchmark documentation and the CI documentation describe the relative gate as the new benchmark policy, replacing the 300-second protocol. - [ ] The job passes on at least two different CI runners. - [ ] Record a decision on whether the benchmark keeps running on both push and pull_request events, since the two runs of PR #278's head commit competed on one runner. ## Blocked by - #280 - #281
nessar added the ready-for-agent label 2026-10-16 22:39:46 +02:00
nessar added a new dependency 2026-10-16 22:39:46 +02:00
#281 test(benchmark): report each scenario as it finishes
nessar commented 2026-10-17 02:19:56 +02:00
Author
Owner
Copy link
Copy source

Done in PR #287 (acdb193, cfdb28c, 3bbe989): relative gate with frozen-engine calibration, per-scenario multipliers and failure messages. Passed on nessar-desktop and srvhome (srvhome ~1.9x slower; CI ratios recorded in docs/benchmark/README.md). The hang guard was raised to 600 s (test timeout 570 s) because srvhome needs ~370 s. Decision recorded in docs/gitea-actions.md: keep the benchmark on push and pull_request, since the push run gates image publication.

Done in PR #287 (acdb193, cfdb28c, 3bbe989): relative gate with frozen-engine calibration, per-scenario multipliers and failure messages. Passed on nessar-desktop and srvhome (srvhome ~1.9x slower; CI ratios recorded in docs/benchmark/README.md). The hang guard was raised to 600 s (test timeout 570 s) because srvhome needs ~370 s. Decision recorded in docs/gitea-actions.md: keep the benchmark on push and pull_request, since the push run gates image publication.
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
bug
dependencies
docker
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
javascript
needs-info
needs-triage
question
ready-for-agent
wontfix
No labels ready-for-agent
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate (renovate) Bot nessar (nessar)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Blocks
#286 perf(planning): accept the speed target and lower the hang-guard timeout
nessar/vac-optimizer
Depends on
#280 test(planning): compare the engine against a frozen engine on random requests
nessar/vac-optimizer
#281 test(benchmark): report each scenario as it finishes
nessar/vac-optimizer
Reference: nessar/vac-optimizer#285