Repository navigation
feat(bench): --baseline to fail a run on a performance regression - #1073
Merged
Merged
Conversation
@max_ms is an absolute ceiling, and to survive the slowest CI runner it has to be loose -- so it only catches catastrophes and says nothing about the 30% slowdown that stays under it. Every performance campaign in this repo (#761, #798, #830, #977) actually asked "is this slower than before", and answering it meant comparing by hand. --baseline compares against the --report-json document of an earlier run (#1028), failing when a benchmark exceeds baseline * (1 + tolerance/100) with a default tolerance of 10 percent. --baseline-update records the new reference, and the table prints a per-benchmark delta so an improvement is visible too. The comparison uses the median, not the average: one descheduled iteration on a shared runner moves the mean far more than the middle value, and a gate that cries wolf gets disabled. A benchmark the baseline does not know is reported as new, and one only in the baseline as removed; neither fails, because the commit that deletes a benchmark must not fail on its absence. A missing or malformed baseline exits non-zero -- a gate quietly comparing against nothing would report every run green, which is worse than no gate. Parsed with awk, not jq: jq is optional everywhere else in bashunit, and a gate that does nothing when a tool is missing is the bug this feature exists to prevent. All arithmetic runs under LC_ALL=C (#912). Binary size budget raised to 608 KiB. Closes #1029
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤔 Background
Related #1029
@max_msis an absolute ceiling: loose enough to survive the slowest runner, so it only catches catastrophes and says nothing about the 30% slowdown that stays under it. Every performance campaign in this repo asked "is this slower than before" and answered it by hand.💡 Changes
--baseline <file>compares the run against the [feat(bench): --report-json / --report-junit for benchmark results #1028]--report-jsondocument of an earlier one, failing when a benchmark exceedsbaseline × (1 + tolerance/100);--baseline-tolerance(default 10) and--baseline-update <file>complete the workflow.awk, notjq(optional elsewhere in bashunit — a gate that does nothing when a tool is missing is the bug this prevents), all arithmetic underLC_ALL=C(7 test failures on Bash 5.3 macOS (nix-shell) #912).