Reference tables
Key generated facts from the pipeline chapters, gathered for lookup. The owning chapters hold the complete explanations; this page is a reference, not a substitute for the reasoning.
Every table here is generated from the code, so it cannot drift from the tool’s behavior.
Data
What each engine measures
See Shape of the data.
| Engine | Measures | Stored in |
|---|---|---|
criterion | Wall-clock time per iteration | nanoseconds |
callgrind | Simulated instruction and branch counts | counts |
alloc_tracker | Heap allocation, in bytes and in count, plus optional peak outstanding bytes | bytes, counts |
all_the_time | Processor time per iteration | nanoseconds |
How many series one benchmark yields
See Shape of the data.
| Engine | One benchmark yields | Which series |
|---|---|---|
criterion | 1 series | wall_time |
callgrind | 3 series | instruction_count, conditional_branches, indirect_branches |
alloc_tracker | Up to 3 series | allocated_bytes, peak_outstanding_bytes, allocation_count |
all_the_time | 1 series | processor_time |
Which engines report dispersion
See Shape of the data.
| Engine | Confidence interval | Standard deviation |
|---|---|---|
callgrind | Never — it reports a single simulated figure | Never |
criterion | Always | Recorded, never read |
alloc_tracker | Only when the operation was measured over several spans | Never |
all_the_time | Only when the operation was measured over several spans | Never |
How benchmark identity is formed
See Shape of the data.
| Engine | Segments | Note |
|---|---|---|
callgrind | package directory, module path, function, and the case id where one is given | Fully qualified |
criterion | group, function, and the parameter where the benchmark is parameterized | Carries no package name, so identical names in different crates share a series |
alloc_tracker | the operation name alone | Carries no package name, so identical operation names in different crates share a series |
all_the_time | the operation name alone | Carries no package name, so identical operation names in different crates share a series |
Storage
The stored run
See Shape of the data.
run
├── schema_version 5
├── context
│ ├── observed_at when the measurement was taken (provenance only)
│ ├── git commit, branch, and whether the tree was dirty
│ ├── environment local, or the CI provider that ran it
│ ├── toolchain target triple and compiler version
│ ├── tool_version which cargo-bench-history wrote this
│ ├── machine host hardware description (recorded, never read)
│ └── best_of repetitions run, when --best-of was used
└── results[] one per benchmark case
├── id the benchmark identity
└── metrics[] one per metric kind, each with
├── value the measurement
├── std_dev where the engine reports one (recorded, never read)
└── interval low and high, where the engine reports them
Key grammar
See Shape of the data.
v1/wordcount/objects/callgrind/x86_64-unknown-linux-gnu/a1b2c3d4e5f60718/4f2a1c9d8e7b6a5f4e3d2c1b0a9f8e7d6c5b4a39/clean.json
| Segment | Meaning |
|---|---|
v1 | Storage layout version |
wordcount | Project — which store this is |
objects | Fixed; separates records from any future sibling namespace |
callgrind | Discriminant set field: engine |
x86_64-unknown-linux-gnu | Discriminant set field: target triple |
a1b2c3d4e5f60718 | Discriminant set field: machine key |
4f2a1c9… | The commit |
clean.json | Which kind of object |
Object kinds
| File | What it is |
|---|---|
clean.json | The canonical run for that commit. At most one. |
dirty-<unix>.json | A snapshot taken with uncommitted changes. Any number. |
bless-<unix>.json | A recorded acceptance of the level at that commit. Any number. |
Collision policies
See Collection.
| Policy | Behavior | When to use it |
|---|---|---|
| default | Refuses, leaving the stored run untouched | Always, unless you have a reason not to |
--skip-existing | Leaves the stored run and reports success | Re-running a collection over a range where some commits are already done |
--overwrite | Replaces the stored run | Re-measuring a commit whose recorded run you do not trust |
What the machine key hashes
See Collection.
| Hashed | Not hashed |
|---|---|
| Processor count | Clock speeds — they vary with thermal state and power policy on one machine |
| Memory-region count | Hostname — a renamed machine keeps its history |
| Processor models | Installed memory size |
Analysis
Mode selection
See Selection.
| Context commit equals the merge base | Dirty run admitted on the context commit | Mode |
|---|---|---|
| yes | no | history |
| yes | yes | branch |
| no | no | branch |
| no | yes | branch |
Minimum evidence
See Detection.
| Detector | Needs |
|---|---|
| History change point | 10 points in the analyzed window, and 5 points on each side of the split |
| History drift | 10 points in the analyzed window |
| Branch comparison | 10 commits on the base side, collapsed to one level each |
Every gate, per detector
See Noise gates.
Each detector applies its own gates in its own order, and a candidate stops at the first gate that declines it. A gate several detectors share is one policy asked at a different point in each sequence.
change_point — a level that moved and stayed moved
| Gate | What it compares | Threshold |
|---|---|---|
split_located | Whether a candidate split exists at all. | a split must be found |
min_regime | How many points the shorter side of the split holds. | 5 points |
non_zero_delta | Whether the two regime levels differ at all. | above zero |
relative_floor | The move as a fraction of the baseline. | 3.0% |
absolute_floor | The move in the metric’s own units. | the metric’s own floor, below |
residual_noise | The move against the series’ own typical residual. | 3× the typical residual |
regime_separation | The share of before-and-after pairs that agree the level moved. | 0.85 |
interval_disjoint | The two regimes’ reported confidence intervals. | the two intervals must not overlap |
selection_adjustment | The change-point’s rank-test chance level, before the split-search correction. | corrects the level; never declines |
significance | The chance level of the rank test comparing the two regimes. | p < 0.05 |
drift — a level that is moving steadily
| Gate | What it compares | Threshold |
|---|---|---|
min_series_points | How many points the analyzed window holds. | 10 points |
significance | The chance level of the trend test across the window. | p < 0.05 |
non_zero_delta | Whether the fitted line moved across the window. | above zero |
relative_floor | The move as a fraction of the baseline. | 3.0% |
absolute_floor | The move in the metric’s own units. | the metric’s own floor, below |
residual_noise | The move against the series’ own typical residual. | 3× the typical residual |
interval_noise_band | The move against the engine’s own reported imprecision. | 2× the reported half-width |
branch — a context commit against the base ref
| Gate | What it compares | Threshold |
|---|---|---|
min_base_commits | How many base-ref commit levels the comparison window holds. | 10 commit levels |
non_zero_delta | Whether the context run lies strictly outside the observed current-base range. | above zero |
relative_floor | The move as a fraction of the baseline. | 5.0% |
absolute_floor | The move in the metric’s own units. | the metric’s own floor, below |
interval_disjoint | The current-base observations’ and context run’s reported confidence intervals. | the two intervals must not overlap |
interval_noise_band | The move against the engine’s own reported imprecision. | 2× the reported half-width |
Absolute floors
See Noise gates.
| Metric | Absolute floor | Why |
|---|---|---|
wall_time | 1 ns | A timing engine fits a slope across a run’s iterations and resolves far below a clock tick, so this is a judgement about what is worth acting on rather than a resolution limit. |
processor_time | 1 ns | A timing engine fits a slope across a run’s iterations and resolves far below a clock tick, so this is a judgement about what is worth acting on rather than a resolution limit. |
instruction_count | 5 instructions | Code layout can shift these counts by a few units between builds of identical source, so a handful of them says nothing about what the code costs. |
conditional_branches | 5 conditional branches | Code layout can shift these counts by a few units between builds of identical source, so a handful of them says nothing about what the code costs. |
indirect_branches | 5 indirect branches | Code layout can shift these counts by a few units between builds of identical source, so a handful of them says nothing about what the code costs. |
allocated_bytes | 1 byte | A fraction of a byte or of an allocation cannot happen; the floor rejects only the sub-unit moves that amortizing across a run’s iterations manufactures. |
peak_outstanding_bytes | 1 byte | A peak counts whole bytes; the floor rejects sub-byte differences introduced by averaging span peaks. |
allocation_count | 1 allocation | A fraction of a byte or of an allocation cannot happen; the floor rejects only the sub-unit moves that amortizing across a run’s iterations manufactures. |
Reporting
Why a series was not judged
See Multiplicity and coverage.
| Reason | What it means | What to do |
|---|---|---|
ghost | A series not measured at the analyzed context commit. | Nothing, if the benchmark was removed or its package was not built. Otherwise check that it still runs at the analyzed context commit. |
ignored | A series ignored by configuration. | Nothing, if the exclusion is intentional. Inspect the raw data with examine, or remove the matching prefix from [ignore].benchmarks to analyze it. |
too_few_points | A series with too few points in the analyzed window. | Wait. The series is judged once enough commits have been measured. |
too_few_points_since_blessing | A series with too few points since being blessed. | Wait. A blessing discards the evidence before it, so the count restarts. |
not_measured_on_branch | A series not measured on the branch. | Run the benchmark on the branch, or accept that this one is out of scope for the comparison. |
too_few_base_commits | A series with too few base-ref commits to compare against. | Measure more of the base ref. A comparison needs a base window to compare against. |
too_few_base_commits_since_blessing | A series with too few base-ref commits remaining since being blessed. | Wait for more base-ref measurements after the blessing. Earlier evidence was intentionally excluded. |
current_base_regime_unresolved | A series whose current base regime is unresolved. | Inspect the recent base history. It appears to have moved, but the new level has too little support for an honest comparison. |
Coverage states
See Multiplicity and coverage.
| State | Verdict on a silent run | What the silence covers |
|---|---|---|
no_series | Nothing was analyzed, so no change could be detected. | Nothing: no series was accounted for. The empty-outcome hint explains why. |
nothing_in_scope | Nothing was in analysis scope, so nothing was judged. | Nothing: every accounted series was absent at the analyzed context commit or ignored by configuration. |
nothing_judged | Nothing was judged, so no change could be detected either way. | Nothing: in-scope series existed but none could be judged; the breakdown says which evidence floor they fell short of. |
partial | No notable changes detected among the series that were judged. | The judged series only: no reportable move among them, and no claim about the in-scope series that went unjudged. |
full | No notable changes detected. | The whole in-scope suite: every in-scope series was judged, so this is the only silent state with no coverage qualification. The verdict stays no notable changes detected. |
Output formats
See Reporting. The JSON excerpt there is illustrative of the report shape, not a complete field catalog.
| Output | How to request it | What it is for | What it carries |
|---|---|---|---|
| Text | the default; --no-text suppresses it | Reading a report in a terminal | every finding; omits the per-reason census when findings exist |
| Markdown | --markdown <path> | Pasting into a pull request or an issue | every finding and a final Coverage section with the complete per-reason census |
| JSON | --json <path> | Automation | the complete census always, and every finding; omits the per-commit chart series, which is presentation rather than data |
| Condensed summary | --markdown-summary <path> (analyze only) | A size-limited destination, such as a pull request comment or a rolling issue body | at most the 10 findings of greatest magnitude, flattened so the per-set grouping is dropped — lossy by design |
| Outcome | --outcome <path> (analyze only) | Selecting an automation message without parsing JSON | one stable wire name: findings, clean, insufficient_baseline, nothing_in_scope, partial |
JSON is the complete machine-readable result: it always carries every finding and the full per-reason census, including the ghost count. It is not an observation archive — the per-commit series behind the charts is obtained with examine. Full Markdown always includes the same coverage counts and reasons in its final Coverage section. Text omits the full reason breakdown when findings exist. Configured exclusions remain visible beside findings on every surface. The condensed summary is lossy by design, so it is the one output that must not be automated against: a check reading it cannot distinguish findings that were capped away from findings that were never made. The outcome file repeats only JSON’s top-level outcome field; it carries no report details.
Comparison-base lag
See Reporting.
| Reason | The line the report prints | What it means |
|---|---|---|
| discriminant set mismatch | Warning: comparison base is 7 commits behind base (discriminant set mismatch) | Newer base data exists, but it was measured under a different machine key. Measurements are never compared across machine keys, so the comparison reached back to the newest base commit this partition covers. A rotating CI pool is the usual cause. |
| no base data at more recent commits | Warning: comparison base is 1 commit behind base (no base data at more recent commits) | No base-side run for the series exists at any more recent commit. The comparison base is simply the newest base data there is, and collection on the base branch is what would move it forward. |
The lag is advisory. It never changes which findings are reported and never affects the exit code; what it changes is how much weight a marginal branch finding deserves, because a comparison against a base state several commits old is exactly that.
Terms
Every term the pipeline chapters define is in the Glossary.