Insights

Everything before this chapter explains what the tool does. This one is about what you do with the result.

Start with the question you need to answer:

• I have a finding — something was reported and you need to act.
• Reading a silent report — no findings were reported, and you need to know what was actually judged.
• I expected a finding and got none — the checklist for tracking down where a real regression was dropped.

I have a finding

flowchart TD
    F["A finding"] --> Q0{"Which mode<br/>produced the report?"}
    Q0 -->|"branch"| BR["It compares your tip<br/>to the base"]
    Q0 -->|"history"| Q1{"Which method?"}
    Q1 -->|"change point"| CP["It names a nearby commit"]
    Q1 -->|"drift"| DR["It names a window"]
    CP --> Q2{"Did anything else move<br/>at the same commit?"}
    Q2 -->|"many series"| SHARED["Look for a shared cause:<br/>code or environment"]
    Q2 -->|"one or a few"| CODE["Suspect the local change"]
    SHARED --> CORR["Correlate metrics with<br/>the diff and run metadata"]
    CORR --> ACC["Fix, or bless if intended"]
    CODE --> FIX["Fix, or bless if intended"]
    DR --> WIN["Examine the series;<br/>look for many small increments"]
    BR --> TIP["Re-run the tip;<br/>check for a lag warning"]

Note that the method named on a finding tells you which comparison produced it: change point and drift come from history mode, branch excursion from branch mode. The report’s header tells you which mode ran.

A change point

The finding names a commit somewhere near the change, not the exact first commit that introduced the new level. A specific causal commit is not always identifiable; if collection is sparse, the cause is somewhere in the gap before the named commit.

1. Look at the series. cargo bench-history examine --benchmark <qualified-id> --metric <name> prints every stored point. Both values come from the finding. The chart in the report is a summary; this is the data.
2. Check whether the named commit has a neighbor gap. If the previous observation is twenty commits back, your suspect list is those twenty commits, not one.
3. Check what else moved. Many simultaneous steps suggest a shared cause, not a specific one. The cause may be shared code such as an allocator, runtime or dependency, or an environment change such as the toolchain, runner, or hardware. Correlate the affected metrics with the commit diff and run metadata before attributing it.
4. Decide. Fix it, or accept it with bless, which re-baselines the series so it stops being re-reported.

A drift

A drift names a window, not a commit, because that is what the evidence supports: no single commit is responsible.

The usual cause is genuinely incremental — a container that grew, an abstraction that gained a layer per change, a loop that got one more thing to do each month. That is exactly the class of regression that comparing against the previous release will never catch, which is why the tool looks for it.

1. Examine the series and look at the shape. A true drift climbs steadily.
2. If it climbs in visible steps instead, the drift detector won on residuals but the real story is several small change points. Bisect: analyze with --context set to a commit partway through the window, and see where the level sat then.
3. Check the window’s length. A drift over a long window with sparse observations may be two distant levels with nothing in between. The chart shows the gaps to scale.

A branch finding

Branch mode judges only the analyzed context commit, against the base ref’s observed current-regime range. Your branch’s intermediate commits are ignored — only the context state is evaluated.

1. Re-run the benchmark on the context commit. Branch findings rest on a small sample, often a single run. Inspect the new engine output directly: ordinary CI collection uses --skip-existing, which preserves existing stored measurements. Replacing stored results requires an explicit collect --overwrite.
2. Look for a comparison-base lag warning. On a rotating CI pool, your runner’s machine key may only have base data from several commits back. The comparison is still valid, but it is against an older base state than you might assume.
3. Check whether the base itself recently moved. If the base stepped a few commits ago and your branch merely matches the new level, that is worth knowing before you go looking through your own diff.
4. Read the comparable-base count literally. It tells you how many eligible base commits produced at least as much report-wide out-of-range movement. It is historical context, not a confidence percentage.

Reading a silent report

“No notable changes detected” applies only to metric series that were actually judged. One benchmark can contribute several metric series, so these counts are not benchmark counts.

Where to find coverage

The GitHub issue or pull-request comment contains a condensed summary, not the full report. When present, Download the full report bundle links to the full outputs:

• Full Markdown (report.md in the bundle, or the path passed to --markdown) ends with a Coverage section. It lists the coverage state, accounted-for, in-scope, judged and unjudged counts, and a count for each unjudged reason. It is present even when findings exist.
• JSON (report.json in the bundle, or the path passed to --json) carries the same account under census; the breakdown is census.reasons.
• Text on stdout and condensed Markdown summaries show the in-scope judged ratio in their headers when anything is in scope. They list unjudged reasons only when there are no findings, while configured ignored-series counts remain visible beside findings.

The warning about in-scope series that could not be judged refers to the full Markdown Coverage section or the JSON census, not to additional detail hidden in the comment. Publication can omit the bundle link. In that case, obtain the full outputs from the workflow that produced the comment, or rerun analyze with the reported commit and discriminant filters, adding --markdown report.md --json report.json. Use the same stored measurements and selection to reproduce the report.

For individual series identities, rerun the same analysis with --verbose: its diagnostics name each unjudged series, the evidence it carried and the rule that declined it.

A silent analysis in Markdown and JSON

These examples are generated from the same analysis using the production renderers. The Markdown example is file content, not terminal stdout; the JSON example contains its matching census object.

Full Markdown report — written by --markdown.

# Benchmark history analysis: textproc

- Commit: 9f2c4a1d3b5e708aab12cd34ef5678901234567a
- Mode: history
- Runs analyzed: 128 (4d17b0c93ea2 → 9f2c4a1d3b5e)
- In-scope series judged: 3 of 6
- Regressions: 0

No notable changes detected among the series that were judged.

Judged 3 of 6 in-scope series; no reportable move survived the gates.

Not judged: 2 series not measured at the analyzed context commit; 1 series ignored by configuration; 3 series with too few points in the analyzed window.

## Coverage

- State: `partial`
- Metric series accounted for: 9
- In scope: 6
- Judged: 3
- Unjudged (including out-of-scope series): 6

| Unjudged reason | Metric series |
| --- | --- |
| not measured at the analyzed context commit | 2 |
| ignored by configuration | 1 |
| with too few points in the analyzed window | 3 |

JSON report — the same analysis from --json, with counts under census.

{
  "project": "textproc",
  "tip_commit": "9f2c4a1d3b5e708aab12cd34ef5678901234567a",
  "tip_dirty": false,
  "mode": "history",
  "outcome": "partial",
  "notable": false,
  "runs": 128,
  "series": 6,
  "regressions": 0,
  "ghosts_excluded": 2,
  "census": {
    "total": 9,
    "in_scope": 6,
    "judged": 3,
    "unjudged": 6,
    "coverage": "partial",
    "reasons": [
      {
        "reason": "ghost",
        "count": 2
      },
      {
        "reason": "ignored",
        "count": 1
      },
      {
        "reason": "too_few_points",
        "count": 3
      }
    ]
  },
  "findings": [],
  "sets": [
    {
      "engine": "criterion",
      "target_triple": "x86_64-unknown-linux-gnu",
      "machine_key": "a1b2c3d4",
      "runs": 128,
      "series": 6,
      "regressions": 0
    }
  ]
}

Interpreting the counts and reasons

The in-scope count excludes ghosts: metric series whose benchmark was not measured at the analyzed context commit. A pull request often benchmarks only affected packages while the analysis reads a wider history, so an untouched package’s benchmarks can be ghosts. They still appear in the accounted-for and unjudged totals and in the reason breakdown. They do not prevent full coverage of the in-scope suite.

Configured ignore prefixes also exclude matching series from the in-scope count while preserving them in the total and reason breakdown. They describe an intentional policy choice, not missing baseline evidence. Removing a prefix restores eligibility of the stored measurements. If the suite contains only ignored series and ghosts, its outcome is nothing_in_scope, not an all-clear.

The following reasons explain why series were not judged:

Judged 0 of N means nothing was tested, not that measured levels stayed flat. Check the reason breakdown: history may still be accumulating, a blessing may have limited usable evidence, or current measurements may be absent. When nothing is in scope, the report says so instead of printing a ratio. When no series was accounted for, it says nothing was analyzed.

The JSON census.coverage field and Markdown’s Coverage state use the same tokens: no_series, nothing_in_scope, nothing_judged, partial and full. Only full means every in-scope series was judged. Even then, silence means no reportable move survived the gates, not proof that nothing changed.

Findings do not imply full coverage: a report can identify a regression while leaving other series unjudged. Platform coverage is separate again. A missing collection platform can leave the available series fully judged without establishing an all-clear for every platform.

The tool reports that a measured level moved, not why it moved. Attributing a move and deciding whether to accept it remain your judgment, recorded with bless.

Unreliable or inconsistent hardware

Symptoms:

• Many unrelated benchmarks step at the same commit without a corresponding shared-code change.
• A series oscillates between two levels rather than wandering around one.
• Findings appear and disappear between runs with no code changes.
• The report’s coverage drops, or comparison-base lag warnings appear.

Diagnosis:

• cargo bench-history list discriminants shows how your history is partitioned. Several machine keys where you expected one means the pool is rotating.
• cargo bench-history machine-key --verbose explains what the current machine hashed to and why.

Remedies:

• backfill on each machine so each key accumulates its own usable history.
• For a one-off infrastructure step you accept, bless --all re-baselines everything at once.

Unreliable benchmarks

A noisy benchmark is not merely unpleasant — it is actively harmful to detection. In history mode the residual gate requires a move to exceed a multiple of the series’ typical residual. In branch mode every current-regime extreme expands the observed range the context must clear. Noise does not just add false alarms; it hides real regressions.

How to recognize one:

• The series oscillates between two levels instead of varying around one. Usually a code path that gets taken sometimes: a cache that sometimes hits, an allocation that sometimes triggers a resize.
• Occasional points far from the rest. Usually the operating system — a scheduler decision, a page fault, another process.
• Drift that tracks nothing in your code. Often a data structure that grows across iterations.
• Values that differ run to run with no code or environment change. Usually unseeded randomness — most often the hashers behind HashMap/HashSet, whose seed is randomized per process, but also any randomized algorithm or sampled input — reshuffling the work each time.

Fixes, roughly in order of effect:

1. Measure instruction counts instead of wall time where you can. Callgrind’s counts are the least noisy thing available, at the cost of not measuring what the user experiences.
2. Pin instruction layout. An unrelated dependency bump can move a hot loop across a cache line boundary and produce a phantom regression with no source change. See Measurement stability.
3. Use --best-of N to take the minimum of several runs. Note the caveat below.
4. Remove the variable path from the benchmark itself: pre-warm caches, pre-size buffers, fix the input.
5. Seed all randomness. Every random choice a benchmark makes must come from a fixed seed, including the hashers behind HashMap and HashSet (their default seed is randomized per process) — otherwise each run measures a subtly different workload.

--best-of is a protocol, not a flag. The stored value depends on how many repetitions you ran, so changing N shifts every series at once and reads as a suite-wide step. Pick a value and keep it. If you must change it, expect a step and bless it.

Multithreaded and OS-waiting benchmarks

These deserve their own mention because they are the worst case.

A benchmark that waits on another thread, a lock, the filesystem, or the network is measuring the operating system’s scheduling decisions at least as much as your code. The scatter is large, and it is not symmetric — waits have a floor and a long tail.

The consequence follows from the residual gate: the band widens until only enormous moves clear it. The benchmark still runs, still records, still looks like coverage in the census — and detects almost nothing.

What to do instead: measure the single-threaded work directly, and test concurrency behavior with something other than a benchmark. If you must measure a concurrent path, instruction counts degrade far more gracefully than wall time.

I expected a finding and got none

Work down this list. Each step names the chapter that explains it.

1. Was the series judged at all? Read the full Markdown report’s final Coverage section, the JSON census, or the verbose per-series reasoning. An unjudged series is never a silent skip — the reason is stated. → Multiplicity and coverage
2. Is it a ghost? If the benchmark did not run at the analyzed commit, every one of its metrics was dropped before detection. This is the single most common cause of a surprising silence. → Reconstruction
3. Is there enough history? Both sides of a step need enough commits. A benchmark added recently cannot produce a change point yet. A series that has just cleared the detection minimum is judged, but a lone step there cannot survive a large family’s correction. → Detection, Multiplicity and coverage
4. Does the move clear both floors? Relative and absolute. A large percentage of a tiny number does not qualify. → Noise gates
5. Is the series too noisy? In history mode, compare the move with the series’ residual scatter. In branch mode, check whether historical extremes expanded the observed current-base range. → Noise gates
6. What does the report-wide context say? For history, a larger judged family sets a stricter false-discovery bar — strict enough that a series at the detection minimum cannot report a lone step in a large family. For branch mode, read how many comparable base commits tied or exceeded the branch score. → Multiplicity and coverage
7. Was it an improvement in history mode? History mode watches for regressions only, so a move that made things faster never becomes a finding there. → Reporting
8. Was it blessed? A blessing re-baselines the series, so moves before the blessed commit are no longer considered. → Reconstruction

If none of these explains it, the fastest way to see exactly which check declined the candidate is to look at the series directly with examine and compare it against the gate order in Noise gates.