Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bb3364e014 |
No files matched your search
@@ -24,10 +24,6 @@ Before the protocol, in the same process, the spec times the calibration engine
|
|||||||
a871f8f) on the `two years` scenario, best of three, and prints the result. This
|
a871f8f) on the `two years` scenario, best of three, and prints the result. This
|
||||||
bundle never changes, so ratios stay comparable with the recorded history; the
|
bundle never changes, so ratios stay comparable with the recorded history; the
|
||||||
differential check's frozen bundle can be rebuilt independently.
|
differential check's frozen bundle can be rebuilt independently.
|
||||||
The spec repeats the calibration after the protocol and uses the slower of the two
|
|
||||||
values, so load that starts during the protocol slows the yardstick too; the second
|
|
||||||
value can only loosen the budgets. When it does, the log prints it and the summary
|
|
||||||
table again with the budgets actually applied.
|
|
||||||
Each scenario's budget is its multiplier times that calibration; the mean
|
Each scenario's budget is its multiplier times that calibration; the mean
|
||||||
`planningMs` of its two measured repetitions must not exceed it. A scenario
|
`planningMs` of its two measured repetitions must not exceed it. A scenario
|
||||||
without a multiplier fails too. #286 tightened the multipliers once #302 met
|
without a multiplier fails too. #286 tightened the multipliers once #302 met
|
||||||
|
|||||||
@@ -17,11 +17,11 @@ publishing images. There is no schedule or deployment job in the preserved
|
|||||||
configuration; the Compose files remain manual deployment options.
|
configuration; the Compose files remain manual deployment options.
|
||||||
|
|
||||||
| Job | Behavior |
|
| Job | Behavior |
|
||||||
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
| `policy` | Tests the policy and publisher with Bun, then validates the checked-out commit, GitFlow, PR title, commits, and stable release version. |
|
| `policy` | Tests the policy and publisher with Bun, then validates the checked-out commit, GitFlow, PR title, commits, and stable release version. |
|
||||||
| `validate` | Runs ESLint, Prettier (application, `scripts/ci-*.mjs` and workflow files) and `bun check` type checks of application and test code, all on Bun with the frozen lockfile. |
|
| `validate` | Runs ESLint, Prettier (application, `scripts/ci-*.mjs` and workflow files) and `bun check` type checks of application and test code, all on Bun with the frozen lockfile. |
|
||||||
| `test` | Runs functional coverage with `bun test --isolate` (line coverage only), converts it to HTML and Cobertura, and uploads coverage for seven days even on failure. |
|
| `test` | Runs functional coverage with `bun test --isolate` (line coverage only), converts it to HTML and Cobertura, and uploads coverage for seven days even on failure. |
|
||||||
| `benchmark` | Required seven-scenario protocol (one warm-up, two measured repetitions) outside coverage in a container limited to 3 GiB. It times the frozen engine on `two years` (best of three) before and after the protocol, and fails any scenario whose mean planning time exceeds its multiplier times that calibration; the 240-second timeout is only a hang guard. |
|
| `benchmark` | Required seven-scenario protocol (one warm-up, two measured repetitions) outside coverage in a container limited to 3 GiB. It first times the frozen engine on `two years` (best of three) and fails any scenario whose mean planning time exceeds its multiplier times that calibration; the 240-second timeout is only a hang guard. |
|
||||||
| `build` | Builds the client-only Angular production bundle (the only check of Angular templates; `bun check` does not replace it) and the Bun static-server bundle, and retains `dist` for seven days. |
|
| `build` | Builds the client-only Angular production bundle (the only check of Angular templates; `bun check` does not replace it) and the Bun static-server bundle, and retains `dist` for seven days. |
|
||||||
| `security` | Runs Semgrep JavaScript/TypeScript SAST and Trivy dependency/secret scans; retains SARIF reports for seven days. Scanner execution must succeed; findings remain reports rather than a new severity gate. The scanners are not project JavaScript, so this job installs no Bun. |
|
| `security` | Runs Semgrep JavaScript/TypeScript SAST and Trivy dependency/secret scans; retains SARIF reports for seven days. Scanner execution must succeed; findings remain reports rather than a new severity gate. The scanners are not project JavaScript, so this job installs no Bun. |
|
||||||
| `images` | Publishes branch/SHA images after required validation, including the benchmark, or promotes an existing SHA image for a stable tag, running the publisher with Bun. Runs only for pushes and same-repository PRs. |
|
| `images` | Publishes branch/SHA images after required validation, including the benchmark, or promotes an existing SHA image for a stable tag, running the publisher with Bun. Runs only for pushes and same-repository PRs. |
|
||||||
@@ -30,11 +30,10 @@ configuration; the Compose files remain manual deployment options.
|
|||||||
The benchmark runs on both push and pull_request events: the push run gates image
|
The benchmark runs on both push and pull_request events: the push run gates image
|
||||||
publication (`images.needs` includes `benchmark`) and the `required` aggregate.
|
publication (`images.needs` includes `benchmark`) and the `required` aggregate.
|
||||||
The relative gate absorbs a slower runner and load that lasts through the whole
|
The relative gate absorbs a slower runner and load that lasts through the whole
|
||||||
job, because calibration runs in the same job. It calibrates before and after the
|
job, because calibration runs in the same job. It does not absorb load that
|
||||||
protocol and judges against the slower value, so it also absorbs load that starts
|
starts or stops during the protocol, such as an overlapping push and
|
||||||
during the protocol, such as an overlapping push and pull_request run of the same
|
pull_request run of the same commit; the 240-second hang guard still bounds
|
||||||
commit. It does not absorb load that ends before the second calibration; the
|
those runs.
|
||||||
240-second hang guard still bounds those runs.
|
|
||||||
|
|
||||||
Push and PR runs are separate so PR title and rebase checks run on the PR event.
|
Push and PR runs are separate so PR title and rebase checks run on the PR event.
|
||||||
New runs cancel older runs for the same event/ref. Image jobs serialize per
|
New runs cancel older runs for the same event/ref. Image jobs serialize per
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ import { calculatePlanningAsyncInternal } from '../services/frozen-engine/leave-
|
|||||||
import { createPlanningCalculation } from '../services/leave-planner.service';
|
import { createPlanningCalculation } from '../services/leave-planner.service';
|
||||||
import {
|
import {
|
||||||
BENCHMARK_BUDGET_MULTIPLIERS,
|
BENCHMARK_BUDGET_MULTIPLIERS,
|
||||||
type BenchmarkMeasurement,
|
|
||||||
benchmarkReport,
|
benchmarkReport,
|
||||||
benchmarkScenarios,
|
benchmarkScenarios,
|
||||||
benchmarkSummaryHeader,
|
benchmarkSummaryHeader,
|
||||||
@@ -72,43 +71,23 @@ describe('leave planner benchmark protocol', () => {
|
|||||||
}
|
}
|
||||||
// Calibration engine (frozen at a871f8f) on 'two years', best of three: the runner-speed yardstick.
|
// Calibration engine (frozen at a871f8f) on 'two years', best of three: the runner-speed yardstick.
|
||||||
const calibrationRequest = scenarios.find((item) => item.name === 'two years')!.request;
|
const calibrationRequest = scenarios.find((item) => item.name === 'two years')!.request;
|
||||||
const calibrate = async () => {
|
let calibrationMs = Infinity;
|
||||||
let best = Infinity;
|
|
||||||
for (let run = 0; run < 3; run++) {
|
for (let run = 0; run < 3; run++) {
|
||||||
const request = structuredClone(calibrationRequest);
|
const request = structuredClone(calibrationRequest);
|
||||||
const start = performance.now();
|
const start = performance.now();
|
||||||
await calculatePlanningAsyncInternal({
|
await calculatePlanningAsyncInternal({ ...request, calculationDate: '2024-01-01' });
|
||||||
...request,
|
calibrationMs = Math.min(calibrationMs, performance.now() - start);
|
||||||
calculationDate: '2024-01-01',
|
|
||||||
});
|
|
||||||
best = Math.min(best, performance.now() - start);
|
|
||||||
}
|
}
|
||||||
return best;
|
|
||||||
};
|
|
||||||
const calibrationBefore = await calibrate();
|
|
||||||
// Direct stdout writes show each row as soon as its scenario finishes.
|
// Direct stdout writes show each row as soon as its scenario finishes.
|
||||||
const print = (text: string) => process.stdout.write(`${text}\n`);
|
const print = (text: string) => process.stdout.write(`${text}\n`);
|
||||||
print(
|
print(
|
||||||
`\nCalibration: frozen engine on "two years", best of 3 = ${calibrationBefore.toFixed(1)} ms` +
|
`\nCalibration: frozen engine on "two years", best of 3 = ${calibrationMs.toFixed(1)} ms` +
|
||||||
'\nBudget = limit x calibration; ratio = mean / calibration.\n',
|
'\nBudget = limit x calibration; ratio = mean / calibration.\n',
|
||||||
);
|
);
|
||||||
print(benchmarkSummaryHeader());
|
print(benchmarkSummaryHeader());
|
||||||
const scenarioRows: BenchmarkMeasurement[][] = [];
|
const measurements = await runLeavePlannerBenchmark(2, (rows) =>
|
||||||
const measurements = await runLeavePlannerBenchmark(2, (rows) => {
|
print(benchmarkSummaryRow(rows, calibrationMs)),
|
||||||
scenarioRows.push(rows);
|
|
||||||
print(benchmarkSummaryRow(rows, calibrationBefore));
|
|
||||||
});
|
|
||||||
// Load that starts after the first calibration slows the scenarios but not the yardstick;
|
|
||||||
// calibrating again and judging against the slower value absorbs it. Never tighter than before.
|
|
||||||
const calibrationAfter = await calibrate();
|
|
||||||
const calibrationMs = Math.max(calibrationBefore, calibrationAfter);
|
|
||||||
if (calibrationMs !== calibrationBefore) {
|
|
||||||
print(
|
|
||||||
`\nCalibration after the protocol = ${calibrationAfter.toFixed(1)} ms; budgets use the slower: ${calibrationMs.toFixed(1)} ms\n`,
|
|
||||||
);
|
);
|
||||||
print(benchmarkSummaryHeader());
|
|
||||||
for (const rows of scenarioRows) print(benchmarkSummaryRow(rows, calibrationMs));
|
|
||||||
}
|
|
||||||
for (const [name, placements] of [
|
for (const [name, placements] of [
|
||||||
['two years', 60],
|
['two years', 60],
|
||||||
['three years', 90],
|
['three years', 90],
|
||||||
|
|||||||
Reference in new issue
Block a user