nessar/vac-optimizer
1
0 Fork 0
Code Issues 7 Pull Requests 3 Actions 1 Packages Projects 1 Releases Wiki Activity

test(benchmark): calibrate again after the protocol and judge against the slower value
CI / policy (push) Successful in 8s
Details
CI / validate (push) Successful in 30s
Details
CI / build (push) Successful in 36s
Details
CI / policy (pull_request) Successful in 8s
Details
CI / benchmark (push) Successful in 1m14s
Details
CI / validate (pull_request) Successful in 29s
Details
CI / security (push) Successful in 1m5s
Details
CI / test (push) Successful in 1m54s
Details
CI / build (pull_request) Successful in 23s
Details
CI / security (pull_request) Successful in 58s
Details
CI / images (push) Successful in 56s
Details
CI / required (push) Successful in 1s
Details
CI / benchmark (pull_request) Successful in 1m46s
Details
CI / test (pull_request) Successful in 1m49s
Details
CI / images (pull_request) Successful in 1m16s
Details
CI / required (pull_request) Successful in 2s
Details

Browse Source 
The relative gate calibrated once before the scenarios, so load that
started afterwards (the push and pull_request runs of one commit overlap
on the same runner, next to the test, build and image jobs) slowed the
scenarios but not the yardstick. A pull_request run failed `two years`
at 1653.1 ms against a 1648.7 ms budget this way.

The spec now repeats the calibration after the protocol and uses the
slower of the two values. The budgets can only loosen, never tighten,
and the log reprints the table with the budgets actually applied when
they change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
nessarandClaude Sonnet 5.5 committed 2026-10-19 10:11:26 +02:00
1 parent 7fc3457b63
commit b37101435a
3 files changed
+51 -25
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/benchmark/README.md
+4
View File
@@ -24,6 +24,10 @@ Before the protocol, in the same process, the spec times the calibration engine
a871f8f) on the `two years` scenario, best of three, and prints the result. This
bundle never changes, so ratios stay comparable with the recorded history; the
differential check's frozen bundle can be rebuilt independently.
The spec repeats the calibration after the protocol and uses the slower of the two
values, so load that starts during the protocol slows the yardstick too; the second
value can only loosen the budgets. When it does, the log prints it and the summary
table again with the budgets actually applied.
Each scenario's budget is its multiplier times that calibration; the mean
`planningMs` of its two measured repetitions must not exceed it. A scenario
without a multiplier fails too. #286 tightened the multipliers once #302 met
docs/gitea-actions.md
+15 -14
View File
@@ -16,24 +16,25 @@ synchronization, and title/body edits. Manual dispatch runs validation without
publishing images. There is no schedule or deployment job in the preserved
configuration; the Compose files remain manual deployment options.
| Job | Behavior |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `policy` | Tests the policy and publisher with Bun, then validates the checked-out commit, GitFlow, PR title, commits, and stable release version. |
| `validate` | Runs ESLint, Prettier (application, `scripts/ci-*.mjs` and workflow files) and `bun check` type checks of application and test code, all on Bun with the frozen lockfile. |
| `test` | Runs functional coverage with `bun test --isolate` (line coverage only), converts it to HTML and Cobertura, and uploads coverage for seven days even on failure. |
| `benchmark` | Required seven-scenario protocol (one warm-up, two measured repetitions) outside coverage in a container limited to 3 GiB. It first times the frozen engine on `two years` (best of three) and fails any scenario whose mean planning time exceeds its multiplier times that calibration; the 240-second timeout is only a hang guard. |
| `build` | Builds the client-only Angular production bundle (the only check of Angular templates; `bun check` does not replace it) and the Bun static-server bundle, and retains `dist` for seven days. |
| `security` | Runs Semgrep JavaScript/TypeScript SAST and Trivy dependency/secret scans; retains SARIF reports for seven days. Scanner execution must succeed; findings remain reports rather than a new severity gate. The scanners are not project JavaScript, so this job installs no Bun. |
| `images` | Publishes branch/SHA images after required validation, including the benchmark, or promotes an existing SHA image for a stable tag, running the publisher with Bun. Runs only for pushes and same-repository PRs. |
| `required` | Fails on failed, cancelled, or skipped required validation, including the benchmark. Requires image publication when expected; accepts its intentional skip for fork PRs and manual dispatch. It is shell only and needs no Bun. |
| Job | Behavior |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `policy` | Tests the policy and publisher with Bun, then validates the checked-out commit, GitFlow, PR title, commits, and stable release version. |
| `validate` | Runs ESLint, Prettier (application, `scripts/ci-*.mjs` and workflow files) and `bun check` type checks of application and test code, all on Bun with the frozen lockfile. |
| `test` | Runs functional coverage with `bun test --isolate` (line coverage only), converts it to HTML and Cobertura, and uploads coverage for seven days even on failure. |
| `benchmark` | Required seven-scenario protocol (one warm-up, two measured repetitions) outside coverage in a container limited to 3 GiB. It times the frozen engine on `two years` (best of three) before and after the protocol, and fails any scenario whose mean planning time exceeds its multiplier times that calibration; the 240-second timeout is only a hang guard. |
| `build` | Builds the client-only Angular production bundle (the only check of Angular templates; `bun check` does not replace it) and the Bun static-server bundle, and retains `dist` for seven days. |
| `security` | Runs Semgrep JavaScript/TypeScript SAST and Trivy dependency/secret scans; retains SARIF reports for seven days. Scanner execution must succeed; findings remain reports rather than a new severity gate. The scanners are not project JavaScript, so this job installs no Bun. |
| `images` | Publishes branch/SHA images after required validation, including the benchmark, or promotes an existing SHA image for a stable tag, running the publisher with Bun. Runs only for pushes and same-repository PRs. |
| `required` | Fails on failed, cancelled, or skipped required validation, including the benchmark. Requires image publication when expected; accepts its intentional skip for fork PRs and manual dispatch. It is shell only and needs no Bun. |
The benchmark runs on both push and pull_request events: the push run gates image
publication (`images.needs` includes `benchmark`) and the `required` aggregate.
The relative gate absorbs a slower runner and load that lasts through the whole
job, because calibration runs in the same job. It does not absorb load that
starts or stops during the protocol, such as an overlapping push and
pull_request run of the same commit; the 240-second hang guard still bounds
those runs.
job, because calibration runs in the same job. It calibrates before and after the
protocol and judges against the slower value, so it also absorbs load that starts
during the protocol, such as an overlapping push and pull_request run of the same
commit. It does not absorb load that ends before the second calibration; the
240-second hang guard still bounds those runs.
Push and PR runs are separate so PR title and rebase checks run on the PR event.
New runs cancel older runs for the same event/ref. Image jobs serialize per
src/app/benchmark/leave-planner-benchmark.spec.ts
+32 -11
View File
@@ -3,6 +3,7 @@ import { calculatePlanningAsyncInternal } from '../services/frozen-engine/leave-
import { createPlanningCalculation } from '../services/leave-planner.service';
import {
BENCHMARK_BUDGET_MULTIPLIERS,
type BenchmarkMeasurement,
benchmarkReport,
benchmarkScenarios,
benchmarkSummaryHeader,
@@ -71,23 +72,43 @@ describe('leave planner benchmark protocol', () => {
}
// Calibration engine (frozen at a871f8f) on 'two years', best of three: the runner-speed yardstick.
const calibrationRequest = scenarios.find((item) => item.name === 'two years')!.request;
let calibrationMs = Infinity;
for (let run = 0; run < 3; run++) {
const request = structuredClone(calibrationRequest);
const start = performance.now();
await calculatePlanningAsyncInternal({ ...request, calculationDate: '2024-01-01' });
calibrationMs = Math.min(calibrationMs, performance.now() - start);
}
const calibrate = async () => {
let best = Infinity;
for (let run = 0; run < 3; run++) {
const request = structuredClone(calibrationRequest);
const start = performance.now();
await calculatePlanningAsyncInternal({
...request,
calculationDate: '2024-01-01',
});
best = Math.min(best, performance.now() - start);
}
return best;
};
const calibrationBefore = await calibrate();
// Direct stdout writes show each row as soon as its scenario finishes.
const print = (text: string) => process.stdout.write(`${text}\n`);
print(
`\nCalibration: frozen engine on "two years", best of 3 = ${calibrationMs.toFixed(1)} ms` +
`\nCalibration: frozen engine on "two years", best of 3 = ${calibrationBefore.toFixed(1)} ms` +
'\nBudget = limit x calibration; ratio = mean / calibration.\n',
);
print(benchmarkSummaryHeader());
const measurements = await runLeavePlannerBenchmark(2, (rows) =>
print(benchmarkSummaryRow(rows, calibrationMs)),
);
const scenarioRows: BenchmarkMeasurement[][] = [];
const measurements = await runLeavePlannerBenchmark(2, (rows) => {
scenarioRows.push(rows);
print(benchmarkSummaryRow(rows, calibrationBefore));
});
// Load that starts after the first calibration slows the scenarios but not the yardstick;
// calibrating again and judging against the slower value absorbs it. Never tighter than before.
const calibrationAfter = await calibrate();
const calibrationMs = Math.max(calibrationBefore, calibrationAfter);
if (calibrationMs !== calibrationBefore) {
print(
`\nCalibration after the protocol = ${calibrationAfter.toFixed(1)} ms; budgets use the slower: ${calibrationMs.toFixed(1)} ms\n`,
);
print(benchmarkSummaryHeader());
for (const rows of scenarioRows) print(benchmarkSummaryRow(rows, calibrationMs));
}
for (const [name, placements] of [
['two years', 60],
['three years', 90],