Introduction

cargo-bench-history is a Cargo subcommand that maintains a long-lived history of benchmark results and analyzes it for trends that snapshot-only tooling cannot see — for example “benchmark X has been getting incrementally slower over the past 12 months” or “scenario Y regressed after commit Z, visible only in hindsight against noisy data”.

Most benchmark tooling only reports the current run, or at best compares against the previous local run. cargo-bench-history instead stores every run as an immutable record — on the local filesystem or in an Azure Blob container — and reconstructs per-benchmark series in git first-parent commit order, so historical trends become analyzable.

flowchart LR
  subgraph Producers
    C[Criterion]
    G[Callgrind]
    A[alloc_tracker]
    T[all_the_time]
  end
  C --> CO[collect]
  G --> CO
  A --> CO
  T --> CO
  CO -->|immutable runs| ST[(Storage: local / Azure)]
  ST --> AN[analyze]
  AN --> F[Findings report]

Who this book is for

This book is the user-facing guide: how to install the tool, wire up storage, collect and backfill history, and read the analysis. It complements — but does not replace — two other sources:

• The API documentation on docs.rs for the library surface.
• The in-repository design document for the internal architecture and rationale.

How it fits together

1. Install the tool and, optionally, write a starter config with install.
2. backfill seeds history by benching a range of past commits, so analysis has a trend to work with.
3. collect runs the workspace benchmarks for the current commit and adds the results to that history.
4. analyze reconstructs per-benchmark series and reports regressions and drift.
5. examine, list, prune, and bless inspect, preview, tidy, and re-baseline the recorded history.

Installation

Install with cargo binstall to fetch a prebuilt binary on supported targets (transparently building from source elsewhere):

cargo binstall cargo-bench-history

Or build from source with:

cargo install cargo-bench-history

Once installed, the tool is available as a Cargo subcommand:

cargo bench-history --help

Next steps

• Follow the Getting started walkthrough for the end-to-end local flow.
• Choose where results are stored — see Storage backends.
• Browse the per-command reference under Commands.

Getting started

This walkthrough uses local filesystem storage. For the cloud backend, see Storage backends.

Before you begin cargo-bench-history records whatever your existing benchmarks emit — it does not create benchmarks for you. Your project needs at least one benchmark a supported engine can harvest (most commonly Criterion benches under benches/), or collect runs but stores nothing. See Benchmark engines for the full list.

# Write a starter .cargo/bench_history.toml (documents the optional cloud backend).
cargo bench-history install

# Bootstrap history from completed commits. Ending at HEAD~1 leaves the current commit for
# the collect step below; a single run on its own has nothing to compare against.
cargo bench-history backfill --local=./bench-history <from-commit> HEAD~1

# Run the workspace benchmarks for the current commit and add the results to that history.
# Drop --local to store in the cloud backend from the config file.
cargo bench-history collect --local=./bench-history

# On a noisy runner, use this instead of the preceding collect command to run the suite
# three times and keep the best (minimum) value per metric.
# cargo bench-history collect --local=./bench-history --best-of 3

# Analyze the recorded history for regressions and drift.
cargo bench-history analyze --local=./bench-history

# Inspect the raw per-commit data points behind a finding, to correlate a change with the
# commit that caused it (both --benchmark and --metric are required).
cargo bench-history examine --local=./bench-history \
    --benchmark my_pkg/my_group/my_case --metric instruction_count

# Print this machine's hardware fingerprint (the key every engine's history is
# partitioned by).
cargo bench-history machine-key

For workloads you want to retain but not analyze automatically, configure benchmark ignore prefixes. The starter configuration includes a commented example.

What just happened

A single collect on its own has nothing to compare against — the value of the tool comes from history. backfill seeds that history from existing commits, collect adds the current commit, and analyze reconstructs each benchmark’s series in git first-parent order to report level shifts and slow drift.

For why a single run is not enough, and how series are ordered and compared, see Comparability and partitioning and Analysis.

Storage backends

Every run is stored as an immutable record. Where those records live is chosen at run time, not baked into the committed configuration:

• Local filesystem — pass --local=<path>, or a bare --local to take the path from the CARGO_BENCH_HISTORY_STORAGE environment variable.
• Azure Blob — configure a cloud backend in .cargo/bench_history.toml and omit --local.

A local path is machine-dependent, so it is never stored in the shared, version-controlled config file. --local always overrides a configured cloud backend.

Backend selection precedence

For every storage-backed command:

1. An explicit --local=<path>.
2. A bare --local — the path from the storage environment variable (unset or empty is an error).
3. Otherwise, the configured cloud backend.
4. Otherwise, a configuration error.

collect --no-store is the one exception: it skips selection entirely for dry runs.

The Azure Blob backend

The cloud backend is configured in .cargo/bench_history.toml — run install to generate a fully commented starter file. Authentication is via Microsoft Entra ID (OAuth); there are no static credentials in the config.

Use setup-azure to provision storage and a shared Azure managed identity with GitHub federation, or export its self-contained deployment bundle for review. It requires no benchmark checkout. Follow the generated starter configuration for the storage fields.

Read-through cache (cloud backend, CI)

The read commands (analyze, examine, list, prune) load the whole in-selection history before reconstructing a series. Against the cloud backend that is one download per object, so CI would re-fetch everything on every run.

A --cache=<dir> flag (with an environment fallback) enables an on-disk read-through cache that mirrors fetched object bodies. Because stored records are immutable per key, a cached body is trusted indefinitely; a small per-project marker invalidates the cache after the rare delete or overwrite. The cache is meaningful only with the cloud backend, so it conflicts with --local. In GitHub Actions, persist the cache directory with the standard Actions cache so each run pays the network cost only for objects it has never seen.

GitHub automation notes

• analyze needs a resolvable git repository with enough history to find the merge-base with the base branch. On a shallow clone, deepen it (git fetch --unshallow, or fetch-depth: 0 in actions/checkout) or pass an explicit --base.
• Run CI collection with --skip-existing for append-only behavior: a same-commit re-run still benchmarks every engine (so a broken benchmark is caught) but writes nothing, keeping caches valid across runs.
• PR and trunk collection use the same store so branch analysis can read both the head and its baseline. Git topology keeps unrelated PR commits out of trunk analysis; storing a branch measurement does not add it to the trunk series.

GitHub automation

Use the prebuilt benchmark-history workflows to collect benchmark history and turn it into GitHub reports:

• The history workflow collects measurements on pushes to main, analyzes the accumulated history and maintains a rolling issue for findings.
• The pull-request workflow considers all workspace packages and automatically selects the benchmarks affected by a PR. It compares their results against the PR’s base and maintains a report comment.
• The backfill workflow fills gaps across a commit range, skipping measurements already stored for the runner’s platform and machine key. It only stores data; it does not analyze or publish reports.

The workflows handle checkout, tool installation and collection; history and PR also handle analysis and report publication. Their default collection platforms are Linux and Windows. Apple Silicon macOS is also supported: add platforms: ubuntu-latest,windows-latest,macos-latest to each caller’s with: block to include it. Your benchmarks and their dependencies must support the selected platforms. You add the caller files and connect shared storage.

Benchmark ignore prefixes in the selected configuration govern analysis on the checked-out revision. They do not shrink collection scope or change platform-completeness accounting. Use an action/tool release supporting the setting. An all-ignored analysis is inconclusive, not evidence that a previous regression recovered.

First, run cargo bench --workspace --all-features to check your benchmarks. If you are adding benchmarks, start with a supported benchmark engine.

1. Set up Azure storage

Follow setup-azure to provision storage and a workflow identity for your repository, using main as the history branch. Keep its printed storage account, container, managed identity client ID and Azure tenant ID for the following steps. For other storage choices, see Storage backends.

Create .cargo/bench_history.toml in your repository root. Set project.id to a name for your workspace’s benchmark history, and copy the account and container from the setup output:

[project]
id = "my-workspace"

[storage.azure]
account = "myhistoryaccount"
container = "bench-history"

In Settings → Secrets and variables → Actions → Variables, create these repository variables from the same output:

These are identifiers, not secrets. The workflows use short-lived identity tokens to authenticate; you do not need a stored credential or an azure/login step. GitHub token requests retry transient service and connection failures with bounded backoff. Authentication and permission rejections are not retried, and exhausting retries reports a failure without switching credentials.

2. Add the history workflow

Create .github/workflows/benchmark-history.yml:

name: Benchmark history

on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  history:
    if: github.ref == 'refs/heads/main'
    permissions:
      contents: read
      actions: read
      id-token: write
      issues: write
    uses: folo-rs/cargo-bench-history-action/.github/workflows/history.yml@v2
    with:
      azure-client-id: ${{ vars.AZURE_CLIENT_ID }}
      azure-tenant-id: ${{ vars.AZURE_TENANT_ID }}

The workflow collects your workspace’s benchmarks and stores the measurements in Azure. The permissions let it read the repository and run artifacts, authenticate to Azure, and publish findings or workflow-failure alerts as issues.

Commit the configuration and workflow to main. Open Actions → Benchmark history to watch the first run. When it finishes, download the bench-history-report-… artifact from the run’s Artifacts section. It contains the full Markdown report, JSON report and condensed summary.

To run it manually, select Run workflow on that Actions page and choose main.

3. Add the pull-request workflow

Create .github/workflows/benchmark-pr.yml:

name: Benchmark pull request

on:
  pull_request:
    branches: [main]
    types: [opened, synchronize, reopened, closed]

jobs:
  benchmark:
    if: github.event.pull_request.head.repo.full_name == github.repository
    permissions:
      contents: read
      actions: read
      id-token: write
      pull-requests: write
    uses: folo-rs/cargo-bench-history-action/.github/workflows/pr.yml@v2
    with:
      azure-client-id: ${{ vars.AZURE_CLIENT_ID }}
      azure-tenant-id: ${{ vars.AZURE_TENANT_ID }}

Commit this file to main, then open a PR that changes benchmarked code, using a branch in your repository. The workflow posts a pending comment, runs the benchmarks, and updates that comment with the result and a link to the pr-bench-history-report-… artifact. Pushing another commit updates the same comment; closing the PR cancels any outstanding benchmark run.

The PR workflow skips forks. With GitHub’s default settings, GitHub’s servers remove id-token: write from fork PR jobs, so those jobs cannot request the identity token needed to access Azure. Editing the workflow or approving its run does not restore that permission. Workflows running in the fork itself use the fork’s identity, which does not match your Azure trust. Azure’s PR trust rule names your repository, not the PR’s source repository: GitHub’s permission restriction, not that Azure rule or a workflow condition, prevents fork access.

4. Fill gaps with backfill

Backfill is useful when hosted runners leave sparse per-machine history or when you want to seed earlier commits. Create .github/workflows/benchmark-backfill.yml:

name: Benchmark backfill

on:
  schedule:
    - cron: '0 2 * * *'
  workflow_dispatch:
    inputs:
      to:
        description: Optional range-end override
        required: false
        type: string

jobs:
  backfill:
    if: github.ref == 'refs/heads/main'
    permissions:
      contents: read
      id-token: write
    uses: folo-rs/cargo-bench-history-action/.github/workflows/backfill.yml@v2
    with:
      azure-client-id: ${{ vars.AZURE_CLIENT_ID }}
      azure-tenant-id: ${{ vars.AZURE_TENANT_ID }}
      lookback: 14 days
      minimum-age: 24 hours
      to: ${{ inputs.to }}

This example runs nightly and permits manual dispatch on main, matching the Azure branch trust configured above. Shared preparation chooses the newest first-parent commit at least minimum-age old, then finds its oldest first-parent ancestor within lookback of preparation time. The optional to input overrides the age-based end selection; the lookback still runs from preparation time, not from the selected end. If that history is older than the horizon, backfill measures just the selected end. If no automatic end is eligible, the workflow explains the no-op and starts no benchmark jobs.

Both duration inputs are required for rolling selection. They accept friendly spans such as 24 hours or 24 hours ago, and ISO spans such as P14D; their magnitudes are resolved from one UTC clock snapshot, including calendar units. lookback must be nonzero; minimum-age may be zero.

To select an explicit range instead, replace the duration inputs with from and to, each a nonblank, single-line commit reference. Do not combine from with lookback or minimum-age. The start must be a first-parent ancestor of the end. Preparation freezes the range to full commit SHAs and execution checks out its end. The core tool visits the inclusive range newest-first, using each historical workspace rather than filtering against benchmarks present at the invocation head.

Already-stored commits are skipped in the current platform/machine partition, so you can resume an interrupted run by dispatching the same range. Each platform job has a six-hour ceiling and runs independently of other platforms’ failures. Backfill runs queue without deduplicating or cancelling earlier invocations; project/platform queues also serialize config paths that identify the same storage project.

Set max-commits: '1' to attempt at most one missing commit per platform job, or choose another positive integer. Omitted or empty input leaves replay unlimited. The limit applies after skipping recorded commits in the current partition and does not shorten the inclusive range. Failed, empty and write-time duplicate attempts count, while pre-check skips do not. Each attempt completes all repetitions and engine storage, and a bounded pass exits normally with a deferred-work count after cleanup. This bounds work, not elapsed time.

Build or benchmark failures stop by default. Set ignore-errors: true to continue past individual failing commits; infrastructure errors still fail. The workflow preserves unsuccessful job conclusions and has no whole-job failure-suppression option. A hosted-runner timeout is not a successful bounded pass.

Backfill accepts the same measurement and installation settings as the other workflows: platforms, working-directory, config, exclude, bench, best-of, all-features, no-default-features, features, rustflags, install-method and source-path. Match your history caller’s feature and repetition settings so measurements remain comparable. There is no package allowlist or overwrite option in the reusable workflow.

Configuration, source-installed tools and the optional .github/actions/bench-history-setup/action.yml hook come from the invocation checkout, not the historical commits. The shared workflow invokes that fixed hook before benchmarking; no setup-path input is needed. A nested working-directory selects the same relative project directory in each historical worktree.

rustflags appends rustc arguments using Cargo’s whitespace-separated RUSTFLAGS syntax, without shell quoting. Existing encoded Cargo arguments take precedence over ambient RUSTFLAGS and keep their argument boundaries. An empty input preserves the environment. Set the same compiler options for history, PR and backfill; custom setup hooks need only prepare repository-specific build prerequisites.

The reusable workflow calculates the range; your caller supplies only parameters and a schedule. It emits no receipts, report artifacts or public outputs, and has no analysis or publication sink; later history runs analyze the stored measurements. See backfill for core range, storage and measurement semantics.

5. Read the reports

Findings describe benchmark changes and are advisory: they do not fail the workflow. Read the report’s coverage information alongside its findings. A complete all-clear requires enough comparable data and successful collection on every expected platform.

The first runs often have too little history to compare against. Their publication state is inconclusive, not clean. An inconclusive report can also mean that only some benchmarks could be judged or that collection coverage was incomplete. The report’s analysis outcome and coverage explain the cause; inconclusive publication does not mean analysis failed. Let the history workflow collect measurements as new commits reach main, or seed earlier commits with backfill.

Only findings create the rolling Benchmark history findings for … issue. An initial clean or inconclusive history report therefore leaves no issue; inspect its artifact instead. Once a findings issue exists, a later complete clean report marks it all-clear but leaves it open. Inconclusive results cannot clear it. PRs receive a comment even for clean or inconclusive results. If no benchmark packages were affected, the comment explains that nothing was benchmarked.

Execution failures are different: inspect the failed Actions run and its diagnostics. For a failed history run, the workflow also attempts to publish a Benchmark history workflow failed for … alert issue. Later successful runs do not close these alerts.

Benchmark engines

cargo-bench-history supports four benchmark engines. It does not run them by name — it enables the combined environment they all need and harvests whichever produced output. What drives the data model is whether each measurement carries a confidence interval (which decides how dispersion is gated). No engine is exempt from run-to-run noise, and every engine’s numbers are machine-dependent in practice — even simulated instruction counts and allocation figures vary with the host, because libraries dispatch to different code paths on different microarchitectures — so every engine is partitioned by machine key.

An interval needs something to vary across, so the two operation engines report one only where an operation was measured more than once. A single-span operation records its value alone, and analysis then judges it the same way it judges Callgrind’s counts — by comparing across commits rather than by reading a per-measurement interval. Nothing is lost: an interval can only ever suppress a finding, never create one.

Why no engine is deterministic

Re-running Callgrind with the exact same binary and input often prints the same count. A history series does not hold those inputs fixed: every commit rebuilds the benchmark, and a toolchain, dependency, target-feature, profile, or benchmark-input change can produce a different instruction stream. Even the same compiler version can make different code-generation choices around inlining, ordering, and layout. Those environmental changes are valid observations in the timeline, but they mean no engine’s historical values should be treated as mathematically exact.

What Callgrind persists

Collection supports Gungraun summary schemas 6 and 7. Both supply the current run’s aggregate counts; baseline-only values and per-part copies are not collected.

For Callgrind, layout-sensitivity is decisive at microbenchmark scale, so only the build-stable events are persisted: the instruction count (Ir) and the two branch-execution counts (Bc, Bi) — they count what the code did. The cache-simulation counts, the derived EstimatedCycles, and the branch-misprediction counts reflect where the code and data landed in memory and swing by tens of percent between two builds of identical source, so they are never parsed or persisted.

Allocation peaks

Available allocation peaks are collected and analyzed automatically as peak_outstanding_bytes. This is allocator-requested memory relative to the start of a span, not RSS or the simultaneous total across worker threads.

Benchmarks that accumulate memory between iterations should use operation.measure_thread().no_peak().iterations(n) in alloc_tracker. Their peak depends on the batch size the harness selects and can produce false regression alerts even with a narrow confidence interval. Varying iteration workloads can also make the batch maximum depend on batch size; freeing pre-existing allocations or transferring allocations across threads can make the reported peak unsuitable for comparison.

Opting out withholds the operation’s peak from both human and machine-readable producer reports, without losing bytes allocated or allocation count. Process-scoped measurements already withhold peaks. Missing peaks are not zero measurements, and opting out does not erase historical data. Use the opt-out consistently across repetitions of a --best-of run, which must all report the same metric set.

Shared shape

Despite differing in units and noise, all four engines reduce to the same shape: a stable benchmark identity → a set of named numeric metrics. Every persisted metric is lower-is-better, so a rise is always a regression and a fall an improvement.

Comparability and partitioning

The central tenet: partition only by what makes results fundamentally incomparable; record everything else as metadata so the analysis can see its effect over time.

Discriminant sets

Two results are comparable exactly when they were measured in the same project and their discriminant sets match. A discriminant set is { engine, target_triple, machine_key }:

• engine — different units and semantics never mix.
• target_triple — even simulated counts are not comparable across architectures.
• machine_key — always present: a fingerprint of the host hardware the benchmark ran on. Every engine is partitioned by it, because every engine’s numbers vary with the hardware in practice. See Collection.

The project — workspace identity, configured and defaulting to the repository directory name — partitions too, but it sits one level above: it selects which store is being read, so results from two projects never meet in the first place. That is why it is not part of the discriminant set proper.

Deliberately metadata, not partition — so a change shows up as a timeline step, which is the whole point of the tool — are the toolchain versions, OS/libc, commit, branch, the environment provider, and the --best-of repetition count the run’s values were reduced from. A rustc bump therefore appears as a step in the series rather than silently forking history.

Read the repetition count differently from the rest, though. Because the reduction keeps a minimum, and the expected minimum of N samples falls as N rises, changing it moves the recorded level of every benchmark noisy enough to be affected — a broad step across the suite at one commit is the signature of the count changing, not of the code changing. Each run records the count it was reduced from, so which protocol produced a value stays recoverable from the stored data. See collect --best-of.

Ordering the timeline

A series is built per (discriminant set, benchmark identity, metric) and ordered by git first-parent topology, not by wall-clock time. Every run does carry a single wall-clock observation timestamp, but it is provenance only and never orders anything — a benchmarked commit’s position on the timeline is its committer date, read from the git graph at analyze time, so a rebase or amended date can never leave a stale timestamp behind.

flowchart LR
  RR[BenchmarkResult] --> RS[Run]
  CTX[RunContext: git + env + toolchain + machine + best-of] --> RS
  RS -->|collect stores| ST[(Storage)]
  ST -->|analyze reconstructs by topology| SE[Series] --> F[Findings]

Benchmark identity

A benchmark identity is an ordered, non-empty list of string segments; each engine adapter decides the segments, and reports render the full form with segments joined by /. Renaming a benchmark starts a new series. Because identity is per discriminant set, the same benchmark can exist under several triples or machine keys at once.

Blessing scopes

A blessing records acceptance of a benchmark change and sets a new analysis baseline at a chosen commit. Its blessing scope selects which discriminant sets receive that acceptance, using engine, target-triple and machine-key restrictions. An unrestricted field matches every value, including values encountered later. Benchmark-id prefixes select the affected benchmark identities separately from this scope.

A blessing can apply to several partitions without combining their measurements. Each matching series is re-baselined independently, even when its partition has no measurement at the chosen commit. See bless / unbless for the command syntax and defaults.

Measurement stability

cargo-bench-history compares a benchmark against its own past. Use a consistent measurement protocol so changes in compiler layout or sampling do not obscure changes in the measured work.

For native timing benchmarks, use the compiler configuration below, keep the repetition count fixed, and exclude workloads whose results are dominated by environmental decisions.

Recommended compiler configuration

Use the following configuration for LLVM-based Rust benchmark builds. This is the configuration used for normal measurements in the Folo workspace:

# bash / zsh
RUSTFLAGS="-Cllvm-args=-align-all-functions=6 -Cllvm-args=-align-all-nofallthru-blocks=6" cargo bench

# PowerShell
$env:RUSTFLAGS = "-Cllvm-args=-align-all-functions=6 -Cllvm-args=-align-all-nofallthru-blocks=6"
cargo bench

These examples assume no other compiler flags are configured. Preserve unrelated options when adding this policy to an existing build. CARGO_ENCODED_RUSTFLAGS takes precedence over RUSTFLAGS; if it is already set, add the options to that encoded argument list instead.

Apply the configuration through the Cargo-built Rust dependencies as well as the benchmark executable, and use it consistently in local, CI and backfill measurements. The values are log2 exponents: 6 selects the 64-byte instruction-cache-line size of x86-64.

Why this configuration

Instruction-fetch and decoding behavior can depend on code placement. The linker orders functions in the final binary, while the compiler determines the layout inside each function. An unrelated dependency change can move a function; a change to untimed setup can move a measured loop within it. A persistent timing shift can therefore occur even when the measured instructions have not changed.

A basic block is a straight-line group of instructions inside a function. A block is reached by fall-through when execution continues into it from the preceding block without a jump.

Function-entry alignment preserves cache-line-relative positions when a byte-identical function is relocated. It does not freeze internal positions when code generation changes, which is why the recommendation also includes selective internal-block alignment.

-align-all-nofallthru-blocks=6 aligns blocks without fall-through predecessors, keeping its padding off those executed paths while stabilizing selected internal positions.

Do not substitute -align-all-blocks=6. It also aligns blocks reached by fall-through, which can execute padding inside the measured path and change its instruction count and timing.

The recommended configuration increases executable code size, especially when applied through dependencies. Accept that footprint cost as part of the measurement protocol rather than mixing benchmark-only and whole-build configurations in one history. It reduces layout sensitivity; it does not stabilize data placement or eliminate every source of native variation.

The options affect the LLVM code being compiled. They do not realign prebuilt libraries, hand-written assembly, or C dependencies built without equivalent options.

Use one protocol throughout the history

cargo-bench-history measures whatever your benchmarks produce; it does not inject compiler flags or dictate a build profile. Put the recommended configuration in the build path that every collection uses, such as a command recipe or CI configuration.

Treat a deliberate compiler-policy change as a measurement-baseline change: build flags are not part of a result’s identity, so measurements from different policies share a series. Use bless to record accepted shifts at the introducing commit, with the benchmark selection and blessing scope matching the expected change. Use --all only when acceptance intentionally covers every benchmark, not to dismiss unrelated findings.

Branch comparisons honor that baseline boundary too. They retain the blessed base commit and exclude earlier evidence; affected series remain unjudged until enough measurements exist at or after the boundary.

Keep backfilled ranges within a consistent compiler policy. Flags reach backfill from its invocation, not from the historical commit being measured, so a current build configuration also applies when collecting older commits.

The alignment recommendation addresses native timing. Instruction-count-only build paths need not add native-timing stability flags; keep their comparisons consistent separately.

Keep the repetition count fixed

Choose a repetition count for each machine and project and keep it fixed. collect --best-of N runs the suite N times and stores the per-metric minimum. The expected minimum falls as the sample count rises, so changing N can shift the recorded level without changing the benchmarked code.

The shift depends on each metric’s noise: a deterministic metric does not move, and any single observation may remain unchanged. Every stored run records the count it was reduced from, so the measurement protocol remains recoverable from the data.

Exclude environment-dominated workloads

Concurrent work, thread creation, live system queries and affinity changes can be dominated by environmental decisions. Repetition and noise gates can reduce random jitter but cannot identify the cause of a persistent shift. Keep these workloads as manual experiments when they cannot support useful automated comparisons.

Use configured benchmark ignores to exclude such measurements from analysis while retaining collection and raw inspection. This is a project policy, not an automatic classification of all multithreaded benchmarks as invalid. Ignoring excludes a benchmark from comparisons; a blessing instead accepts a baseline change in a benchmark you still want analyzed.

Analysis

A series is built per (discriminant set, benchmark identity, metric), ordered by git first-parent topology. The goal is high signal-to-noise: report level shifts and trends that are real and stay silent on measurement jitter. Because no engine is deterministic, the detector treats every metric as noisy and never trusts a value as exact.

History finding methods

History mode evaluates two finding methods for each series, and the resulting findings are ranked together by descending relative move:

1. Change-point (step) — the primary finding. A single most-likely level shift is located, and the report names a commit somewhere near that split. Persistence is built in, so a single-commit blip cannot trip it.
2. Monotonic drift — a separate finding type for slow trends, established by a trend test and sized by an outlier-resistant slope.

When both fire on one series, the better-fitting model wins, so sharp steps route to the change-point method and smooth ramps to drift, and the two never double-report one event.

This page is the mental model. For the mechanism — which test, which threshold, in which order, with worked examples — see Data pipeline.

Noise-aware gating

The gates exist to suppress noise — movement the measurement itself manufactures — and nothing else. They are not a filter for changes you might find uninteresting. A level shift caused by a new runner, a toolchain bump, or a hardware refresh is a genuine move of the measured level and is reported; deciding that its cause makes it acceptable is your judgment, recorded with bless. Every floor is therefore tuned to the measurement noise and to the smallest magnitude worth acting on — never to whether someone would call a cause acceptable, because a gate wide enough to hide an infrastructure step would hide the regressions that share its shape.

A candidate change-point is reported only when several gates all hold: the two regimes must be statistically distinguishable, the move must clear a practical-magnitude floor, it must stand above the series’ own scatter, and the two regimes must genuinely separate rather than merely differ on average. Where points carry confidence intervals, those act as an additional veto that can only suppress a candidate, never manufacture one. A change-point also needs a minimum run of points on each side, so a series too short to hold two such regimes is not analysed at all.

The practical-magnitude floor has two parts and a move must clear both. A relative floor demands a minimum percentage. An absolute floor, in the metric’s own units, demands a minimum magnitude — a handful of instructions is build layout shifting rather than work done, a fraction of a nanosecond is not worth acting on however confidently it was measured, and a fraction of an allocation cannot happen. Both apply to every metric: without the absolute floor, a benchmark whose baseline is a couple of nanoseconds turns scheduling jitter into a double-digit percentage “regression”, and without the relative floor a large baseline would flag on a move that is noise at its scale.

Each detector applies its own gates in its own order, and the thresholds differ by mode and by metric. The Noise gates chapter walks every one of them, with the current values and worked examples.

Branch excursion method

Branch mode asks a different question from history mode — not “did this series change somewhere” but “is this context commit outside every value recently observed in the base ref’s current regime?” Two properties follow from that:

• A commit is one observation. Several stored runs at one commit are re-measurements of a single build on a single runner, not independent evidence about the base level, so each commit’s runs collapse to that commit’s median before anything is compared. The comparison window is counted in commits for the same reason — a run-counted window would shrink to a handful of commits wherever a repository records several runs each. History mode does not collapse this way: it ranks the series’ raw points, so a commit carrying several stored runs weighs on it once per run.
• One new observation, not a second sample. The context commit is judged against the observed range of the current base regime. It reports only when the value is strictly below every observed base value or strictly above every one, and reports the excess beyond the nearest range edge. It does not invent a probability distribution for the one branch observation.

Branch mode also holds its relative floor above history’s — a pull-request comment is read by everyone who touches the branch, so a false alarm costs more there. Where the engine reports per-point dispersion, further suppression-only vetoes apply.

Neither mode attaches a confidence score to a finding. History reports have calibrated chance levels internally; branch reports instead state how many comparable base commits produced at least as much report-wide out-of-range movement. Reports rank individual findings by the size of the move.

The Detection chapter has the full comparison of the two modes, including how selector and reference observations prevent the base commit being judged from choosing its own regime boundary.

Controlling false discoveries

A repository has many benchmarks × metrics, so a per-series result is not enough to understand a whole report. History candidates enter a false-discovery-rate procedure whose family is every history series this analysis judged, including series that produced no candidate.

One consequence is worth knowing up front: a history finding has to clear a stricter bar for a large judged family than for a small one. The report’s judged count is the denominator to inspect. Branch mode cannot honestly apply that procedure to one observation. It keeps factual excursions and compares the complete branch report with the same analysis run on eligible base commits in turn. See Multiplicity and coverage for both controls.

Configured ignore prefixes remove benchmarks from analysis without discarding their measurements. Ignored series and ghosts sit outside the in-scope coverage denominator, but remain disclosed in the report’s account. If those exclusions leave nothing in scope, the outcome is nothing_in_scope, not an all-clear.

For practical interpretation, see Reading a silent report.

Analysis modes

The same stored history answers two very different questions, so analyze runs in one of two modes, auto-detected from git topology (there is no flag to force a mode):

• history — the base-branch view: long-range change-point and drift detection; reports regressions only.
• branch — the feature-branch view: judges the context commit’s latest state against the base ref, reporting both directions. The branch’s own intermediate history is ignored.

Comparison-base lag

Branch mode compares the context commit against the recent base-ref points for the same project and the same discriminant set — same engine, target triple, and machine key. Measurements are never compared across machine keys, so on rotating CI pools, where the newest base-ref commits may have run on a different machine, the context run’s key can have usable base data only a few commits behind the base ref. The comparison quietly reaches back in history.

analyze discloses this per affected discriminant set, naming how far behind the comparison base is and why:

• discriminant set mismatch — a newer base-ref run for the benchmark and metric exists, but under a different machine key. This is pool rotation, not a gap in coverage.
• no base data at more recent commits — no newer base-ref run exists for that series at all.

The warning is advisory metadata: it explains what the context commit was actually compared against and never changes which findings are reported or the exit code.

Re-baselining

A long history should not keep re-flagging an event you have already dealt with. A blessing re-baselines a history series from the blessed commit forward, so the pre-blessing step is no longer re-flagged while earlier points still feed the chart. In branch mode it is a hard evidence boundary: the blessed base commit remains, but every earlier base observation is excluded from regime selection, range construction, and historical comparison.

Blessing is how you dispose of a shift that is real but not about your code — a runner swap, a toolchain bump, a deliberate tradeoff. The detector reports that the level moved; you record, once and against the commit it happened at, that you accept it.

Report formats

The canonical formats — text (to stdout), full Markdown, and JSON — carry the same findings and compose from one pass. JSON is the machine-readable form: a flat, globally-ranked findings list where each finding is self-describing. A consumer keys off a top-level “notable” flag and reads each finding’s direction, magnitude, and attribution. Separately, analyze can render a condensed Markdown summary — a lossy excerpt for a size-limited downstream consumer.

Full Markdown ends with a Coverage section listing counts and reasons for unjudged metric series, whether or not findings exist. JSON carries the same account under census. Text and condensed summaries give the judged ratio in their headers and the reason breakdown only when there are no findings. Insights shows matching Markdown and JSON examples and explains how to interpret incomplete coverage.

Human-readable findings include a compact, topology-accurate chart: one column per first-parent commit from the first observation onward, so a commit with no measurement renders as a gap (a broken line) rather than being collapsed away. Leading gaps are trimmed and interior gaps kept, and a trailing gap up to the analyzed context commit is the visual form of the “no newer data” disclosure — a benchmark not measured on the most recent commits. History mode shows the full selected series, including pre-blessing context; branch mode shows the comparison baseline and a bounded recent tail ending at the context commit, dropping the interior branch commits and drawing the comparison-base lag as the empty columns between the newest base observation and the context commit, so the commit being judged remains visible without compressing months of history into the same chart.

There is no severity classification: a finding’s magnitude is conveyed by its relative-change percent, and which findings warrant action is left to human or agent judgment.

Data pipeline

The data pipeline follows a single number all the way through the tool: from the moment a benchmark engine writes it to a file, to the sentence in a report that says it moved.

The rest of this guide teaches the mental model. This part is the mechanism, in full, with the numbers. It exists for two readers:

• You have a finding that does not make sense, and the concept chapters have not settled it. Chapter Insights is the triage entry point; it links back into whichever stage you need.
• You maintain this tool, or you have to trust it, and you want to check that what it does matches what it claims. Every stage below states what it computes and against which threshold, so you can reproduce a verdict by hand.

The stages

flowchart TD
    E["Benchmark engines<br/>criterion · callgrind · alloc_tracker · all_the_time"]
    E -->|"one file per engine"| C

    subgraph collection ["Collection"]
        C["collect / backfill"] --> S[("Stored runs<br/>one object per engine per commit")]
    end

    S --> SEL

    subgraph analysis ["Analysis"]
        SEL["Selection<br/>which objects are eligible"]
        SEL --> REC["Reconstruction<br/>runs become series"]
        REC --> DET["Detection<br/>locate, fairly test, and size a move"]
        DET --> GAT["Noise gates<br/>is the move real?"]
        GAT --> FDR["Multiplicity control<br/>is it real given everything else tested?"]
    end

    FDR --> REP["Reporting<br/>findings, ranked, in three formats"]

Each stage preserves one thing, and naming those is the fastest way to understand the shape of the whole:

The correction for trying many possible history change points is part of Detection, not a separate pipeline stage. It makes the step detector’s chance level fair before the significance gate weighs it. Multiplicity control later answers the report-wide question: history applies its false-discovery correction across series, while branch mode compares the complete report with historical base turns.

How to read the pipeline chapters

The chapters are ordered along the pipeline, and each one assumes only the ones before it. Reading them in order builds the whole picture; stopping anywhere leaves you with a correct partial one.

Two conventions run throughout.

Terms are defined before they are used. Every chapter that introduces a term opens with a short table defining it in plain language, and the Glossary collects them all with the textbook name alongside, for when you want to read further. Where a plain description works as well as the technical name, these chapters use the description.

Generated evidence is computed, not invented. Every behavior-bearing table, figure, computed example, configured value, and serialized excerpt is produced by cargo-bench-history-figures from the production type, key builder, adapter fixture, or implementation-owned inspection surface that owns the behavior. Completeness and derivation tests pin those links — enum ALL coverage, exhaustive matches, real key-builder assertions, and fixture-backed adapter checks — and freshness tests fail if the checked-in includes drift from regenerated output. Genuinely explanatory, non-behavioral prose stays ordinary Markdown.

This is also why the examples are small: they are meant to be checkable by hand, not realistic.

Where this fits

Shape of the data

Before anything can be analyzed, it has to be measured, reduced to a common form, and stored. This chapter is that form: what one benchmark produces, what survives into the store, and what the store looks like.

It is the only place in this guide that documents the storage layer. If you are debugging why two results are not being compared, or what a directory of JSON files means, this is the chapter.

Terms used here

One benchmark is not one number

The most common surprise: a benchmark does not produce a timeseries. It produces several — one per metric kind the engine reports.

So four benchmark identities in a suite can easily be eight or more series, and every one of them is judged independently. That is why a store’s series count is larger than the benchmark count, and why the false-discovery family grows faster than you might expect.

What each engine measures

Everything is lower-is-better. There is no metric where a larger number is an improvement, which is what lets the tool talk about regressions and improvements without knowing what a benchmark does.

What Callgrind deliberately throws away

Callgrind reports far more than three counts — cache hits at each level, estimated cycles, branch mispredictions. Only the instruction count and the two branch counts are stored.

The discarded ones are simulated and highly sensitive to code layout: they can swing by tens of percent between builds of identical source, because the simulator’s model of the cache depends on where functions landed in memory. Storing them would add series that regularly report large moves for no reason — the precise failure this tool exists to avoid.

Which number is taken

Each adapter picks one value per metric, and the choice matters more than it looks:

• Criterion prefers the regression slope over the mean where a slope is available. The slope is fitted across iteration counts and is much less sensitive to a slow first iteration.
• alloc_tracker and all_the_time take the per-iteration slope, never the recorded totals. The optional alloc_tracker peak uses the producer’s weighted span-peak estimate and its own confidence interval.
• Callgrind takes the new side of the pair it reports.

Two consequences worth knowing. The stored record does not say which estimator was used, so a Criterion benchmark that stops producing a usable slope falls back to the mean silently — and because the mean and the slope disagree, that switch can read as a step that no code caused. And for the required allocation totals and processor-time metrics, a missing or non-finite slope drops that operation from the run rather than falling back to anything, so a benchmark can simply disappear from a series, with no error to say why.

An unavailable allocation peak drops only that metric, preserving the operation’s bytes and allocation count. Peak recording is default-on; benchmarks with unsuitable memory lifetimes disable it in the producer with ThreadSpan::no_peak(), as described under Allocation peaks.

Dispersion: what the engine tells you about its own precision

Some engines report a confidence interval alongside the value.

The rule that matters is the same throughout the pipeline: a confidence interval can only ever take a candidate away. No gate uses one to create or strengthen a candidate. So an engine that reports none is not held to a weaker standard — it is judged on its own between-commit scatter instead, which is arguably the more relevant quantity anyway.

Benchmark identity

An identity is an ordered list of name segments, rendered with slashes. Each adapter builds it from what its engine knows:

Renaming a benchmark therefore starts a new series and retires the old one — the tool has no way to know the two are related. See Reconstruction.

Note which identities carry no package attribution: Criterion (group and function only) and the operation engines alloc_tracker and all_the_time (the operation name alone). Only Callgrind is fully qualified. So two identically-named benchmarks in different crates of one workspace share a series. See Limits.

What a stored run holds

One run is one engine’s whole output at one commit, written as a single object.

run
├── schema_version          5
├── context
│   ├── observed_at         when the measurement was taken (provenance only)
│   ├── git                 commit, branch, and whether the tree was dirty
│   ├── environment         local, or the CI provider that ran it
│   ├── toolchain           target triple and compiler version
│   ├── tool_version        which cargo-bench-history wrote this
│   ├── machine             host hardware description (recorded, never read)
│   └── best_of             repetitions run, when --best-of was used
└── results[]               one per benchmark case
    ├── id                  the benchmark identity
    └── metrics[]           one per metric kind, each with
        ├── value           the measurement
        ├── std_dev         where the engine reports one (recorded, never read)
        └── interval        low and high, where the engine reports them

The context is provenance. Only some of it drives behavior:

That last row is not an oversight — see Limits.

The observation timestamp deserves emphasis: when a benchmark was measured has no bearing on where it sits in a series. Position comes from git topology, read at analysis time. A commit benchmarked out of order, or backfilled months later, lands exactly where its commit sits.

Where it all lives

A run’s storage key is a path with a fixed grammar:

v1/wordcount/objects/callgrind/x86_64-unknown-linux-gnu/a1b2c3d4e5f60718/4f2a1c9d8e7b6a5f4e3d2c1b0a9f8e7d6c5b4a39/clean.json

Concrete partition directories hold runs and any partition-local blessing records:

Project-level blessing records store their blessing scope explicitly and live under v1/<project>/objects/blessings/<commit>/bless-<issued_unix_nanos>.json. Each record carries benchmark prefixes and a required scope object containing engine, target_triple and machine_key arrays. An empty array leaves that axis unrestricted. The scope is matched at query time rather than expanded into the partitions present at issuance. Partition-local blessing records derive their exact scope from their keys and remain effective. See bless / unbless for acceptance and revocation.

Each path segment is sanitized: characters outside a safe set are replaced, and the result is lowercased. That is what keeps a key usable as both a filesystem path and a blob name — but it also means identifiers differing only in case or punctuation collapse into one partition.

Bodies are compressed at the storage boundary, so a local store is a tree of gzipped JSON.

Atomicity, and what it does not cover

One stored object is written whole or not at all, and never rewritten: normal collection is append-only. That is what keeps caches valid and history trustworthy.

The unit of atomicity is one engine’s output at one commit — not the whole collection. Engines are stored one at a time, so a run killed part-way through can leave a commit holding Criterion data but not Callgrind. Nothing is corrupt, but the commit is incomplete, and a later backfill will consider it already covered.

If a collection was interrupted, re-collect that commit with --overwrite rather than assuming backfill will notice.

Taking the minimum of several runs

--best-of N runs the whole suite N times and keeps, per benchmark case and per metric kind, the smallest value.

Two details follow from “per metric kind”:

• A stored result can mix metrics chosen from different repetitions — the instruction count from run 2, the branch count from run 3.
• The winning metric is kept whole, so its confidence interval travels with the value it belongs to. Intervals are never merged or recomputed.

The minimum is a useful heuristic here because shared-runner contention commonly adds delay. Keeping the smallest observed value reduces the influence of transient positive interference, without guaranteeing that the chosen value is disturbance-free.

--best-of is a protocol, not a flag. The stored value depends on N, so changing it shifts every series at once and reads as a suite-wide step. See Insights.

What this stage hands on

A store: immutable objects, keyed by partition and commit, each holding one engine’s reduced output. Next: Collection, which is how they get there.

Collection

Collection is how measurements become history. Two commands do it: collect measures the commit you are on, and backfill reconstructs a range of commits you never measured.

This chapter is what each actually does, and — more usefully — what shapes the store ends up with.

Terms used here

What collect does

flowchart TD
    P["Probe the environment<br/>git commit · dirty? · toolchain · machine key"]
    P --> R["Run cargo bench<br/>once, or N times under --best-of"]
    R --> H["Harvest each engine's output directory<br/>ignoring anything older than this run"]
    H --> D["Reduce to the common form<br/>one value per benchmark, per metric"]
    D --> S["Store one object per engine"]

The tool does not run engines by name. It enables the environment all four need and harvests whichever produced output, so a workspace with only Criterion benchmarks simply yields Criterion data.

The harvest is freshness-gated

Engines write into target/, and target/ persists between runs. Without a check, a collection would happily re-store last week’s output for a benchmark that failed to run this time — and that stale value would enter history as if it had been measured at the current commit.

So harvested files must be newer than the run that was supposed to produce them. A benchmark that did not run this time contributes nothing, rather than contributing a lie.

The consequence is worth stating: an engine that produced no output stores nothing, silently. There is no error, because “this workspace has no Callgrind benchmarks” and “the Callgrind benchmarks all failed” look identical from here. Coverage can flag a missing engine only for a benchmark already present in history: an earlier observation of a benchmark that is missing now surfaces at analysis time either as a ghost — a benchmark history remembers but the analyzed commit no longer measures, so it is dropped before detection — or, where some observation survives but too few to judge, as an unjudged series the census still counts and names a reason for. A benchmark that was never recorded leaves no trace at all: collection has no object, no series, and no way to tell “no benchmarks” from “a failed engine”. Verifying that every expected engine ran is a separate collection-time check, not something analysis coverage can guarantee.

Collisions: what happens when a run already exists

Normal collection is append-only. Storing over an existing clean run is refused by default.

Why refuse by default: history is only trustworthy if a recorded measurement stays put. Silently replacing one would make a series depend on how many times someone re-ran a command.

Dirty runs

A measurement taken with uncommitted changes is stored separately, under a key that includes the observation time rather than replacing the commit’s clean run.

That keeps three things true at once:

• the clean run at that commit is untouched;
• several dirty snapshots on the same commit coexist;
• the analysis can tell them apart and apply its admission rules.

Only two dirty snapshots taken in the same second collide.

Dirty runs are what make the tool useful before you commit — you can measure a change in progress — and they are also what makes uncommitted work on the base branch analyze as a branch.

A dirty run is stored like any other, including to shared cloud storage. But it is only ever read back to judge local, in-progress work: an official history analysis of a base branch never admits one, so a dirty snapshot cannot leak into anyone else’s baseline. See dirty admission.

Target triples and machine keys

The target triple comes from the toolchain that compiled the benchmark.

The machine key is a short hash over the host’s hardware identity. What goes into it is chosen deliberately:

The exclusions are the interesting part. Clock speeds are excluded because they vary with thermal state and power policy on the same physical machine — including them would rotate the key spontaneously and shatter the history. Hostname is excluded so a renamed machine keeps its history.

machine-key --verbose prints exactly what was hashed and why, which is the fastest way to diagnose a pool that is rotating keys unexpectedly.

What is stored per commit

Collection stores runs, not series. A run is one engine’s whole output at one commit — every benchmark it measured, and every metric of each — written as a single object. The series these pipeline chapters describe do not exist in the store at all: they are cut across runs later, at analysis time, one metric of one benchmark read against every commit (see Reconstruction).

flowchart LR
    subgraph r1["run @ commit 1"]
        direction TB
        p1a["bench_a · instructions"]
        p1b["bench_a · branches"]
        p1c["bench_b · instructions"]
    end
    subgraph r2["run @ commit 2"]
        direction TB
        p2a["bench_a · instructions"]
        p2b["bench_a · branches"]
        p2c["bench_b · instructions"]
    end
    p1a -. "one series" .-> p2a
    p1b -.-> p2b
    p1c -.-> p2c

Read the boxes top to bottom and you have a run: everything one engine measured at one commit. Read a row left to right and you have a series: one metric, across commits. So a single run contributes one point to many series, and a single series draws one point from many runs. The rest of this chapter is about the runs — how they land on commits, and what shape the store ends up with.

How runs land on commits

One partition’s objects for a commit sit together under that commit. There is no requirement that commits be measured in order, or that every commit be measured at all.

Read that picture carefully, because most collection problems are visible in it:

• Gaps are commits nobody measured. Ordinary and expected — but the detectors cannot see how wide they are.
• A row that starts late is a machine that joined the pool later.
• A row that stops is a benchmark that was deleted, or a machine that left.
• Two half-filled rows are the signature of a rotating CI pool, and the usual cause of comparison-base lag.

backfill

backfill measures a range of commits you never measured, by checking each one out in an isolated worktree and running the suite there.

It differs from collect in ways that matter:

• It plans over a first-parent range, newest first, so the most useful history arrives first if you stop it early.
• It skips commits already covered in the partition it is about to write, so re-running it is cheap and safe.
• It probes the toolchain inside the worktree, so a commit that pinned a different compiler is measured with that compiler.
• It works in a detached worktree, so your working tree is untouched.

Filling the gaps a runner pool leaves

A rotating CI pool hands each commit to whichever machine was free. The pool as a whole measures every commit, but each individual machine key ends up with only the commits that happened to land on it — a sparse, half-filled history:

flowchart LR
    c1 --> c2 --> c3 --> c4 --> c5 --> c6
    c1 -. "ran on" .-> A
    c3 -.-> A
    c5 -.-> A
    c2 -. "ran on" .-> B
    c4 -.-> B
    c6 -.-> B
    A["machine A history<br/>c1 · c3 · c5 (c2, c4, c6 missing)"]
    B["machine B history<br/>c2 · c4 · c6 (c1, c3, c5 missing)"]

This is the case backfill exists for, and the key point is that another machine’s data is not coverage for yours. Counts and timings are never compared across machine keys, so a partition with holes stays a partition with holes until that machine fills them.

The fix is to run backfill on each machine over the shared range. Each fills its own partition; each ends up with a history dense enough to judge.

What can go wrong, and how it looks

What collection hands on

A store of immutable objects. Analysis reads it next, starting with Selection — which decides which of them are even eligible.

Selection

Selection answers one question: which stored runs are eligible for this analysis?

It answers it entirely from storage keys and git topology. No excluded object is ever fetched or read — which is what keeps analysis fast against a store holding years of history, and why the stage exists as a stage at all.

Terms used here

The funnel

flowchart TD
    A["Every object in the store"] --> B["Discriminant filter<br/>engine · target triple · machine key"]
    B --> C["On the analyzed history?<br/>first-parent ancestry of the context"]
    C --> D["Dirty admission<br/>which side of the merge base?"]
    D --> E["--since cutoff<br/>by committer date"]
    E --> F["Sort by key, assign ordinals"]
    F --> G["Fetch and parse only these"]

The funnel below traces one worked query — a host-local analyze — against this store. Two of its inputs drive the removals that would otherwise look arbitrary: the discriminant filters resolve to this host’s own partitions, so a foreign partition measured on another machine is dropped; and --since resolves to a cutoff date part way through the history, dropping everything committed before it. Each row is one selection stage and what it removed.

32 runs removed and 22 runs account for all 54 run objects this store held. Only those survivors are fetched and parsed; every other run was decided on its storage key and the commit’s place in the topology alone.

The store in this worked example holds only runs — stored benchmark measurements (see Shape of the data). A blessing, a recorded acceptance of a change (see Reconstruction), is a separate object kind stored alongside them: it is set apart during discriminant selection and follows its own path, so it never enters the run-only topology, dirty-admission, and window stages counted here.

The grid draws the analyzed first-parent line, so the runs the on-history stage removed have no column in it — having no place on that line is exactly why they were removed. Partitions are labeled by engine, target triple, and machine key, with the triples shortened to fit.

Target, base, and the split

Two refs frame every analysis:

• the context (--context, default HEAD) whose history is analyzed, and
• the base (--base, default the detected default branch).

The tool walks the context’s first-parent ancestry and splits it at the merge base with the base. Everything at or before the merge base is base-side; everything after is target-side.

flowchart LR
    b0["·"] --> b1["·"] --> MB["merge base"]
    MB --> t1["·"] --> T["context = HEAD"]
    MB -. "base branch continues" .-> B["base tip"]
    subgraph baseside["base-side"]
        b0
        b1
        MB
    end
    subgraph targetside["target-side"]
        t1
        T
    end

Everything in the left box is base-side (at or before the merge base); everything in the right box is target-side (after it).

If the base cannot be resolved, or shares no ancestor with the context, that is a hard error — not a silent fallback. The usual cause is a shallow CI clone; fetch full history, or pass an explicit --base.

Mode is derived, not chosen

From the shape of the history it is handed, selection derives an analysis mode: the question the rest of the pipeline will answer about the series. There are two, and later stages — detection above all — branch on the one selection chose:

• History mode — does a level move somewhere in an official branch’s own history?
• Branch mode — is the analyzed context commit off the base it would merge into?

There is no flag to force the choice. The tool reads two signals:

The second signal is the surprising one. A run recorded from a dirty working tree on top of the base tip selects branch mode, because that is what it is: work that is not in the base’s history yet, sitting on top of it. An unnamed feature branch.

Note it is the admitted run that decides this, not the state of your checkout. A dirty checkout with no dirty run in the analysis still analyzes as history.

Dirty admission

A dirty run is one measured with uncommitted changes. Where it is allowed depends on which side of the merge base it sits:

Why base-side dirty runs are rejected: the base is the reference the analysis compares against. A measurement of somebody’s uncommitted experiment is not a description of the base’s history, and letting it in would corrupt the baseline for everyone.

--no-dirty removes dirty runs from the target side as well, so no dirty run is admitted anywhere — the base side already rejects them regardless.

The window

--since drops whole runs older than a cutoff, judged by each commit’s committer date.

The default is where the two modes diverge, and it is worth knowing which you are in:

• History mode applies a six-month look-back by default, so a scheduled trend watch does not silently widen as history accumulates.
• Branch mode applies no default. A branch is judged against its base’s recent commits, and a time window could only starve that comparison of evidence.

The cutoff is deliberately one-sided. There is no --until, because --context already anchors the newest edge of the timeline — a topology-first tool moves the tip by naming a commit, not by naming a date.

A --since shorter than your collection cadence will quietly turn a healthy store into a report full of unjudged series. The report’s coverage line is where that shows up.

Discriminant filters

Three discriminant filters restrict which discriminant sets participate: --engine, --target-triple, and --machine-key. Each is repeatable and unioned.

Omitting one auto-detects from the host — except --engine, which defaults to all of them, since a machine can produce output from several engines at once. The literal all disables a filter entirely.

Because auto-detection means the same command can search different data on different machines, every query prints a one-line effective selection summary to stderr, naming what was actually searched and which parts were defaulted. It is not opt-in.

Ordering is fixed before any parallelism

Runs that survive the preceding selection stages are sorted by storage key and assigned their positions before any object is fetched. Loading is then parallel, but the result is not affected by which worker finished first.

Two analyses of the same store therefore produce byte-identical reports. That is what makes a report diffable and a regression in the tool itself detectable.

What the other commands share

analyze, list runs, and examine run this same selection pipeline, so list genuinely previews the selected raw data. The columns below name concepts that later chapters introduce — benchmark prefixes, the ghost filter, and configured exclusions and blessings — so a linear reader is not expected to know them yet; they are forward references. The divergences are real, though, and worth knowing:

examine deliberately skips ghosts, configured ignores and blessings: it is a drill-down onto raw recorded data, and hiding points there would defeat its purpose. prune is a maintenance command with its own selection rules — notably no default --since — because deleting data by accident is much worse than analyzing too little.

Ignore matching needs benchmark IDs from the payload, so it happens after loading, not in this key-only selection stage. It leaves the selected run count unchanged.

A base merged into the branch is supported

Branch mode builds its comparison window from the base ref’s own recent first-parent history, anchored at the base ref. Whether the branch was rebased onto the base, the base was merged into the branch, or both happened during development does not change that comparison window.

The context commit’s first-parent line is still the line that list, examine, and prune display or maintain, and it is still divided at its fork point — the newest commit it shares with the base ref — into base-side history and the branch’s own commits. When the base was merged in, the merge base sits off that line, so the fork point is the newest shared commit rather than the merge base itself; base-side history is identified and preserved either way, so prune never deletes a shared commit without the base-branch opt-in.

The merge base is consulted only as a shared-history check. If no merge base can be resolved at all, the repository checkout is too shallow or the refs are disjoint, and selection stops with the existing merge-base-unavailable error.

What selection hands on

An ordered set of eligible objects, and the topology needed to place them. Next: Reconstruction.

Reconstruction

Selection decided which stored runs are eligible. This stage turns them into the thing detection actually looks at: series.

Reconstruction orders observations and narrows the series detection can see. Its exclusions make this stage an important place to investigate a report that surprises you.

Terms used here

From runs to series

A stored run is a snapshot: one engine’s whole output at one commit. A series is the opposite cut through the same data — one measurement, across every commit.

A series is identified by three things together:

• the discriminant set it was measured under,
• the benchmark identity, and
• the metric kind.

So a single Callgrind benchmark contributes three series, and a run holding twenty benchmarks scatters points across sixty of them.

Each resulting point carries its value, its position in the first-parent topology, whether it came from a dirty working tree, and the confidence interval the engine reported if it reported one.

Note where the position comes from: git, read at analysis time — not from anything stored in the run. When a measurement was taken has no bearing on where it sits in the series.

Ordering

Points sort by topological position, then clean before dirty, then by the order the objects were selected.

The clean-before-dirty rule matters at the analyzed tip. A dirty snapshot is newer work than the clean run at the same commit, so it sorts last and becomes the series’ latest state, which is what branch mode judges.

Gaps are holes, not zeroes

A commit with no observation produces no point. Nothing is interpolated, and nothing is filled in.

The two panes are the same series twice. The first is what the history holds: each observation at the commit it was measured at, gaps drawn to scale, so a steady climb reads as the straight line it is even though most commits were never measured. The second is what the detector receives: a bare sequence, with no notion of a commit it holds no observation for. The same climb now reads as uneven steps, because the gaps the detector cannot see are exactly where the biggest jumps happened.

The consequence is the single most important thing to understand about this stage, and it is easy to miss:

The detectors cannot see how far apart two observations are.

Ten observations spread across a thousand commits and ten across ten consecutive commits are indistinguishable to every detector and every gate. The report’s charts, like the history pane, are drawn against topology and do show gaps to scale — which is why a suspicious finding is always worth looking at as a chart: it shows you what the statistics never knew.

Why not interpolate? A gap means nothing was measured, not that nothing changed. Interpolating would invent data, and inventing data in the input to a change detector manufactures changes.

Ghost elimination

A benchmark with no observation at the analyzed commit is dropped entirely — every one of its metric series, before anything else happens.

A couple of details matter:

• Any run at that commit counts as presence, including one from a dirty working tree.
• It runs before blessings and before detection, so a ghost never enters the false-discovery family and never dilutes the correction.

Why: re-reporting a benchmark that no longer exists is noise, and a deleted benchmark’s last few commits look exactly like a regression that was never fixed.

The cost: a benchmark measured on ninety-nine of a hundred commits, but not the tip, produces nothing at all. If a run failed at the tip, the report will be quieter than it should be — and the census is where that shows up. This is the most common reason for a surprising silence; see Insights.

Renaming a benchmark does the same thing, from the tool’s point of view: the old identity stops being measured and becomes a ghost, and the new one starts a fresh series with no history.

Configured exclusions

An ignore prefix deliberately keeps a benchmark out of analysis while leaving its measurements stored. It removes every matching metric series before blessings and detection, so neither history correction nor branch-wide comparison can depend on it.

Ghost elimination precedes this policy: a benchmark absent at the context is accounted for as a ghost, not counted again as ignored. The census records each remaining exclusion as ignored. Neither exclusion applies to list or examine.

Blessings

A blessing records that you looked at a change and accepted it. It re-baselines the series: detection sees only the commits from the blessed one onward.

The full series is kept for charting, so the chart still shows you the whole story while detection judges only the active part. A blessed series that is left with too few points to judge is reported with its own distinct reason, rather than being lumped in with series that were simply too short.

A few details worth knowing:

• The evidence line depends on the mode. History applies the blessing to the context history. Branch mode applies it to the base ref’s first-parent history, because that is the evidence the branch is judged against. In both cases the blessed commit remains and earlier evidence is excluded.
• Benchmark identity selection uses prefixes. The recorded prefixes are matched against the slash-joined benchmark identity (group/case, package/module/fn, …), so blessing foo/bar also accepts foo/barbaz. Bear that in mind when names share a stem.
• A blessing applies within its scope. Its blessing scope selects the eligible discriminant sets. Each matching series is re-baselined separately; acceptance across several machine keys or targets never combines their measurements.

One run per commit

A series carries at most one point per commit. A clean run is stored once per commit and never rewritten, so it cannot contribute two. The only commit that can hold more than one run is one with dirty snapshots — repeated measurements of an uncommitted working tree — and those are admitted only on the target side (see Selection).

History mode admits no dirty runs, so every commit it reconstructs contributes exactly one point. Branch mode judges only the context commit, and only that commit’s latest run: its clean run, or the newest dirty snapshot taken on top of it. An earlier run at the context commit is a superseded state, not what is being evaluated, so it takes no part. Either way the context is a single observation, judged against a base that is itself one clean point per commit.

What reconstruction hands on

A set of series, each an ordered sequence of points, with ghosts and configured ignores removed and blessings positioned on the mode’s evidence line.

Next: Detection, which locates possible moves and corrects for its own search before handing candidates onward.

Detection

Detection is the stage that asks one question of a reconstructed series: did its level move? Everything before this chapter arranged the data. Everything after it decides whether to believe the answer.

This chapter covers what the tool looks for, how it looks, and — just as important — what it deliberately does not claim.

Terms used here

What counts as a signal

A signal is a level that moved, established by a named statistical test and sized by an estimator that a few odd measurements cannot throw off.

That definition is narrow on purpose, and three exclusions follow from it.

• A signal is not a cause. The tool reports that a level moved. Why it moved — a code change, a compiler upgrade, a noisy neighbor on the build machine — is your judgment, recorded with bless.
• A signal is not one surprising measurement. A level has to persist to be a level.
• A signal is not yet a finding. Everything this chapter produces is a candidate. The noise gates and the group-wide correction still have to agree, and most candidates do not survive both.

Two shapes, two detectors

Performance changes arrive in two shapes, and each needs a different instrument.

flowchart TD
    S["A reconstructed series"] --> M{"Which mode?"}
    M -->|"history"| H1["Look for a step:<br/>where did the level change?"]
    M -->|"history"| H2["Look for a drift:<br/>is it sliding one way?"]
    H1 --> A{"Both found<br/>something?"}
    H2 --> A
    A -->|"yes"| B["Keep whichever model<br/>fits the series better"]
    A -->|"only one"| C["Keep it"]
    M -->|"branch"| D["Compare the context commit against<br/>the base ref's current level"]

A step is the common case: something changed at one commit, and every commit after it sits at a new level. A drift is the case that snapshot-to-snapshot comparison can never catch — nothing changed much at any single commit, but a hundred commits later the benchmark is measurably slower.

Finding a step

The step detector works in three moves.

1. Locate. A split search scans every place the series could have changed level and picks the single most likely one.
2. Test fairly. A rank comparison asks, two-sided, whether the before and after regimes differ. The result is corrected for having chosen the strongest of many possible splits.
3. Size. Each side’s median becomes its level, and the difference between them is the move. Medians rather than averages, so one outlier cannot invent a step or hide one.

The report names a commit somewhere near the split — the detector’s best estimate of where the new level begins, not a guarantee of the first commit that introduced it. Use examine to inspect the series and narrow the location if a specific commit can be identified.

The split search’s own chance level is not used as evidence. It only locates the boundary. The rank comparison decides whether the regimes differ, and the correction below makes that comparison fair despite the search that chose its boundary.

Here is a series that steps, and the detector’s own answer for it.

Reported. A regression of +29.57% via change point, somewhere near commit10.

The level moved from 100.4 to 130.1.

The split is where the level changed, not where the largest single jump happened.

A series can contain more than one visible step. The split search still chooses one boundary, and later gates judge that single candidate.

Reported. A regression of +54.07% via change point, somewhere near commit15.

The level moved from 102.2 to 157.4.

The split search chooses one boundary even though another step remains visible.

Making the winning split a fair test

The step detector chooses a split because it looked strongest. Treating that winner as though someone had named it before seeing the data would give it an unfair advantage: even an unchanged noisy series offers many chances for one split to look persuasive by accident.

flowchart LR
    S["One unchanged,<br/>noisy series"] --> A["Try split A"]
    S --> B["Try split B"]
    S --> C["Try split C …"]
    A --> W{"Keep the<br/>strongest"}
    B --> W
    C --> W
    W --> L["A lucky winner<br/>can look convincing"]

Detection corrects that advantage by asking how often chance could produce a winner at least this strong. It has two conservative ways to answer. A mathematical bound over every possible split can certify an especially clear step immediately. If only one exact split is possible, that fixed comparison is already fair. Otherwise the detector constructs a bounded conditional orbit, checks every member, and runs the entire split search again. Frequent equally strong rearranged winners explain the apparent step; rare ones support it.

flowchart LR
    O["Observed order"] --> G["Magnitude + noise gates"]
    G --> FIT{"Step fits at least<br/>as well as drift?"}
    FIT -->|no| D["Use qualified drift"]
    FIT -->|yes| A["Bound every possible<br/>split mathematically"]
    A -->|clear enough| AC["Search-adjusted<br/>step chance level"]
    A -->|needs more resolution| S["Try the next member of the<br/>complete conditional orbit"]
    S --> C{"Is the answer forced,<br/>or is the group complete?"}
    C -->|no| S
    C -->|yes| AC
    AC --> TWO["Account for trying<br/>step and drift"]
    D --> TWO
    TWO --> CAN["Detection candidate"]

History tries both the step and drift shapes, then keeps whichever model fits the series better. That gives chance another opportunity to offer a lucky answer, so each detector’s chance level is doubled before its significance gate. The correction applies whether only one detector raises a candidate or both do; it accounts for giving either detector an opportunity to report, and does not change which model wins the fit comparison.

Reusing the actual measurements preserves repeated values and quantization: an integer counter with many ties is judged against rearranged histories with those same ties. The rearrangement group is deterministic, includes the observed order, and is enumerated completely. When every distinct ordering fits the work budget, all of them are used instead. The same series therefore produces the same verdict on every platform without relying on a lucky random sample. Where direct rank counting is practical it is exact; where it is not, observed and rearranged histories use the same approximation, and their exact conditional comparison supplies the trustworthy chance level.

The calculation is deliberately bounded. Ordinary unchanged series can stop once the winners already seen prove that the final answer cannot pass. Clear changes often need no rearrangements because the mathematical bound settles the question. Ambiguous cases may require the complete orbit, whose size has a fixed maximum rather than growing without limit with the number of benchmarks. If that orbit cannot distinguish an isolated candidate from chance at the strict group-wide threshold, the tool stays silent; it does not turn missing resolution into confidence. The bound prevents runaway work; it does not promise that every difficult series is cheap.

This correction belongs inside Detection because it repairs how the history step detector searched within one series. It is not the later multiplicity control, which accounts for testing many series. Branch mode does not use this chance-level machinery. It selects the latest supported base regime under one shared false-boundary budget, then makes a factual observed-range comparison. Its report-wide historical comparison supplies finite context rather than a probability — see Limits.

Finding a drift

The drift detector also works in three moves, but every one of them differs.

1. Test first. A one-way-trend check counts how often the series rises against how often it falls, across every pair of points. A series wandering at random produces a balance; a drifting one does not.
2. Fit. An outlier-resistant slope is fitted from the middle of all the pairwise slopes, so a few odd measurements cannot tilt the line.
3. Size. The move is the fitted line’s rise across the window — not the difference between the first and last measurements, which would be at the mercy of whatever those two points happened to do.

A drift is attributed to the whole window rather than to a commit, because that is what the evidence supports: no single commit is responsible. So the finding names the range it accumulated over — rendered accumulated <oldest> → <newest> — rather than a single commit. Do not go looking at one commit’s diff.

The range is always the whole analyzed window, even when the drift occupies only part of it: a series that held flat, drifted for a stretch, then held flat again is still reported over its full range, because the detector does not try to pin down where the slope began or ended. examine prints every point, which is where to see the shape within the range and correlate it with commits yourself.

Reported. A regression of +10.40% via drift, accumulated from commit0 to commit29.

The level moved from 98.5 to 108.7.

No single commit is responsible, so the finding names the window.

When both fire

A real step also registers as a weak trend, and a real drift can look like a step if you split it down the middle. So both detectors run on every series in history mode, and when both produce a candidate the tool keeps whichever model leaves its points closer to it.

A step fitted through a stepped series leaves almost nothing unexplained; a sloped line through the same series leaves every point off by something. The comparison is made on exactly that — how far a typical point sits from each fitted model — and the smaller one wins. A tie goes to the step.

One event is therefore never reported twice, and the shape that gets reported is the one that actually describes the series.

What does not count

Two series that look like they moved and do not report. These matter more than the positive cases: they are what the tool’s usefulness rests on.

A single elevated point

Nothing reported. A level has to persist; one point is an event, not a level.

A level has to persist. Both sides of a step must hold enough commits for their medians to mean anything, so a spike near the end of a series has no after-side to speak of. This is the first gate a step candidate meets, and the one that makes the tool usable at all — a build machine having a bad minute is the single most common thing a benchmark series does.

A series that is simply noisy

Nothing reported. The split search still nominates its best candidate here; everything downstream exists to reject it.

Nothing changed here, and nothing is reported. Note what would happen without care: the split search still nominates some split, because it always nominates its best candidate. Everything downstream exists to reject it.

A level that moved and came back

An excursion that has since returned to where it started is silent too, however large it was while it lasted.

Nothing reported. The current level has already returned to the baseline.

That is a deliberate narrowing rather than a gap. By the time such a series is analyzed, its current level already agrees with its baseline, so there is nothing to act on — and a report that mixes “your code is slower now” with “your code was briefly slower some time ago” makes the first harder to see.

The measurements are still there. examine prints every recorded point of a series, without judging any of them, which is where to look when you want to know what a benchmark has been doing rather than what it is doing now.

Branch mode asks a different question

History mode asks “did this series change at some point?”. Branch mode asks something much narrower: is the analyzed context commit off the base ref’s current level?

That difference in question drives every difference in method.

The branch’s own intermediate commits are discarded. Only the analyzed context commit is the state being evaluated, so only that commit is judged.

The comparison uses the observed range of the base ref’s selected current regime. The context run is an excursion only when it lies strictly below the lowest observed value or strictly above the highest. Equality with either edge is inside. That is a deliberately modest statement: there is only one context observation, so the tool does not pretend it has a second sample or a probability model for unseen values.

In the figures below, the shaded value band is that observed range. Its edges are the real cutoff, and the reported magnitude is the excess beyond the nearest edge. Every verdict shown beneath a figure is the real one. The base window is the recent base-ref first-parent commit sample for the same discriminant set, capped at 128 commits and anchored at the base ref, not at the merge base.

Reported. A regression at the context commit, outside every observed current-base value.

The context run is 130; the observed range is 95.9-103.5, and the excess beyond its nearest edge is 26.49.

The context run is slower than every observation in the current base regime.

The same base window, with a tip that agrees with it:

Nothing reported. The context run is inside the observed current-base range, so there is nothing to report.

When the base itself recently moved

A base window spanning a genuine level shift describes two regimes. Using both would allow older operating conditions to hide a branch excursion from the current one.

With at least 20 base commits, branch mode alternates the observed base levels in chronological order between a selector lane and a reference lane. Observation order keeps both lanes useful even when the benchmark ran on only a sparse subset of first-parent commits. Only selectors may locate boundaries. The reference lane is reserved for the report-wide historical comparison, so a base commit cannot help choose the boundary it is later judged against.

The selector lane searches recursively for supported changes under one shared, selection-adjusted error budget. Each accepted split must leave full regimes, clear the practical floors, and separate the two populations almost completely. The current regime begins at the first selector observation known to follow the boundary; an interleaved reference observation whose side is ambiguous is not used. If a strongly separated recent group is still too short to establish a regime, the series is reported as unjudged rather than compared with stale history.

A base regression that was later reverted needs a second look. The point where such a stretch ends looks like no change at all beside the matching observations further back, so the search finishes by re-examining whatever follows the newest boundary it accepted, against that stretch alone and with the same search on both sides of an unsupported split. Without that step the reverted observations would stay in the comparison range and hide a branch move of the same size.

Reported. An improvement at the context commit, outside every observed current-base value.

The context run is 100; the observed range is 124.6-135.2, and the excess beyond its nearest edge is 24.65.

The context value matches the older regime but is faster than every observation in the current regime.

Histories with 10–19 base commits skip this selection and use the full observed range. They can support the factual range comparison, but not a defensible decision to discard an older regime.

Historical extremes remain evidence

A base window is a sample of what the base ref measures, and a shared machine occasionally contributes a reading that measures something else — another job on the same host, a thermal event, a noisy neighbour. Such a reading is not a level and not a trend: it stands well clear of its neighbours in one direction and is gone at the very next commit.

Branch mode keeps it. Removing a historical extreme would make the range narrower and permit a stronger claim by hiding contrary evidence. A context value inside the full observed current-regime range cannot honestly be described as faster or slower than everything the base recently did.

Nothing reported. The context run remains inside the observed range, including the disturbed reading, so the evidence does not support an excursion.

The quiet verdict under the figure is therefore intentional. The observed extreme costs sensitivity, but the alternative would be an unsupported finding.

Comparing the whole branch report with history

An excursion says what the recorded range proves about one series. It does not say how remarkable the complete report is when many series are inspected. Branch mode answers that separately by turning eligible base commits into historical branch-like turns:

flowchart LR
  B["real branch turn"] --> BR["candidate: branch tip\nreferences: all current-base values"]
  H["historical turn"] --> HR["candidate: one reference-lane base commit\nreferences: remaining base values + real branch tip"]
  BR --> S["apply the same range and practical gates\nsum normalized excess across a shared series family"]
  HR --> S
  S --> C["count historical turns tied with or above the branch score"]

Only stable series that share the same candidate base commits enter one rectangular family. Adding the real branch value to each historical turn makes the two directions symmetric. The report can then say, for example, “None of 10 comparable base commits showed as much out-of-range movement as this branch.” It does not call that count confidence or use it to erase factual excursions. If too few comparable commits exist, the excursions remain and the report says the wider comparison could not be formed.

History chance levels, not a confidence score

A history finding does not carry a confidence or certainty number. Every reported finding has already cleared its test, so any such number would read as near-certainty on all of them: a finding whose chance level is one in a million and one that barely cleared would round to the same reassuring figure. It would rank almost nothing, so the report omits it and ranks by size of move instead.

The detector still uses chance levels as internal decision inputs; they are not retained as finding fields or printed by --verbose. This chapter explains them so the verdict logic can be understood, not to provide another number for ranking findings. A chance level is how poorly chance explains the pattern — and it is easy to over-read. Four things it is not:

• Not the probability that the finding is correct. It says chance is a poor explanation. It says nothing about whether the cause is your code or the machine.
• Not a dial you tune. Every reported finding has already cleared its gate, so the ranking of importance is left to the size of move, not to how small a chance level came out.
• Not the family-wide result. A history change point has already been corrected for its internal split search. The group-wide correction runs afterwards to account for how many series were tested and does not feed back into this chance level.
• Not a branch score. Branch mode uses observed ranges and historical report turns rather than a per-series chance level.

Minimum evidence

Below these thresholds a series is not evaluated at all. It is counted as unjudged with a stated reason rather than silently skipped — see Multiplicity and coverage.

The minimum exists because a handful of points cannot tell a real level shift from ordinary scatter: with too few observations every test is dominated by chance, and any “finding” from them would be noise dressed as signal. So the bar is a deliberate judgment — a floor chosen to refuse data too sparse to trust — rather than a value the statistics derive on their own. Set it too low and the report fills with accidents; the thresholds here are picked to keep that from happening while still judging any series with a genuine history behind it.

Clearing this floor puts a series in the judged family. In history mode that is not the same as being able to report a lone step: a rank comparison on so few points has a floor, and the group-wide correction’s rank-1 bar shrinks as the family grows. See Multiplicity and coverage.

What detection hands on

A history candidate carries the series, method, direction and size of the move, attribution, and selection-adjusted chance level. A branch candidate carries the selected observed range and excess beyond its nearest edge, with no manufactured chance level.

None of them is a finding yet. Next: Noise gates.

Noise gates

Detection produces history candidates whose chance levels already account for the detector’s within-series choices, and branch candidates that lie outside an observed current-base range. This chapter is the set of checks that decides which candidates are worth your attention.

Every gate answers the same underlying question in a different way: is this move larger than what could be explained by measurement process noise? A benchmark that reports 100 ns will typically not report exactly 100 ns the next time. The gates exist to keep that fact from becoming a stream of false alarms — and every one of them is a filter: a gate can only ever remove a candidate, never create one or strengthen another.

Terms used here

What the gates are, and are not

The gates weigh evidence, never cause.

This is worth stating plainly because it is the most common misreading. If a compiler upgrade makes your benchmark 8% slower, that is a real move and the tool will report it. The gates will not suppress it for being “not your fault” — they cannot know whose fault it is, and a tool that guessed would be worse than one that does not try.

Deciding a reported move is acceptable is your call, and bless is how you record it. What the gates remove is movement that is not there: jitter, sampling luck, and moves too small to act on.

The questions the gates ask

There are more gates than there are questions, because several detectors ask the same question at a different point in their own sequence. The questions are the part worth learning:

The diagnostic ladder starts with Detection’s own setup and test rows so it can explain the whole verdict in one place. selection_adjustment is one of those rows: it is the within-series correction described in Detection, surfaced here for visibility. It changes the chance level and never declines a candidate by itself.

Each detector has its own sequence

There is no single gate order. Each detector applies the gates its own question needs, in the order that makes sense for it. A change point checks inexpensive magnitude and noise evidence before running its bounded search correction; a branch comparison first establishes that the context value is outside the observed range, then checks the excess beyond the nearest edge.

A candidate stops at the first gate that declines it. Nothing after that runs, which is why a gate ladder is often short. That is an accurate picture, not a truncated one.

Each detector applies its own gates in its own order, and a candidate stops at the first gate that declines it. A gate several detectors share is one policy asked at a different point in each sequence.

change_point — a level that moved and stayed moved

drift — a level that is moving steadily

branch — a context commit against the base ref

Two things in those tables are easy to miss:

• Branch mode uses a higher relative floor than history mode. A branch comparison rests on a single context observation, and a pull request that cries wolf gets ignored, so the bar is raised.
• The interval checks are not applied uniformly. Disjointness is checked wherever two regimes are compared — a change point, or a context commit against its base. The noise band belongs to drift and branch. A detector applies an interval check only where the comparison it makes has a meaningful interval to check.

Gate logic: is the move big enough to matter? (relative_floor, absolute_floor)

Two floors, and a candidate must clear both.

Relative — a fraction of the baseline. This is the one people expect.

Absolute — a floor in the metric’s own units, which exists because a percentage means nothing at small magnitudes. A benchmark running at 3 ns per iteration that moves to 3.2 ns has moved by more than 6% — comfortably past the relative floor — and told you nothing you can act on.

The floors differ per metric because the reasons differ:

This pair is also what makes the tool behave sensibly across scales: the same proportional move on a benchmark a thousand times larger clears the absolute floor easily, and is reported.

Gate logic: is it bigger than what this series does anyway? (residual_noise)

This is the primary noise check, and the one that does the most work. It needs nothing from the engine — no confidence interval, no repeat runs — because it measures the series against itself.

For history, the tool fits the candidate’s own model to the series (a step for a change point, a line for a drift), then measures how far a typical point sits from that model. The move must be several times that distance. Branch mode uses its stricter observed-range rule instead.

The shaded band is what the series does anyway. A move inside it is indistinguishable from the series being itself.

A note on the multiplier. The typical residual is a plain median distance, not a figure rescaled to behave like a standard deviation. So “three times the typical residual” is not “three standard deviations” — for well-behaved data it is considerably stricter. Do not translate the multiple into a familiar sigma figure and reason from that.

This gate is also why a noisy benchmark is worth fixing rather than tolerating: it widens the band, and the band is what real moves have to clear. See Insights.

Gate logic: do the two sides genuinely separate? (regime_separation)

A candidate can pass a significance test and still be an artifact. The classic case is a series that oscillates between two levels: split it anywhere and the two sides differ, and with enough points a significance test will happily call that difference real.

So the tool asks a second, different question: of every possible before-and-after pairing, what share agree the level moved? That pairwise directional frequency is the agreement share — a direction-aware effect size, not another significance test.

A genuine step: nearly every pairing agrees.

A series that revisits both levels leaves a substantial share of pairings contradicting the split, and the candidate is declined — which matches what a human reads off the chart, namely “noisy, unchanged”.

The important property is that this gate does not weaken as the series grows. A significance test gets easier to pass with more data, because more data makes a smaller difference detectable. An agreement share does not: it measures how completely the two sides separate, which more data estimates more precisely rather than inflating.

Gate logic: does the engine’s own precision explain the move? (interval_disjoint, interval_noise_band)

Where the engine reports a confidence interval, further checks apply. Both can only ever take a candidate away — no interval ever creates one or relaxes another gate.

Disjointness. If the two regimes’ intervals overlap, the engine is telling you it cannot distinguish these levels.

Noise band. The move must exceed a multiple of the engine’s own reported imprecision, independently of how the two levels compare.

Engines that report no interval — Callgrind always, and alloc_tracker or all_the_time for a single-span measurement — simply skip these. History still has its residual gate; branch mode still requires an excursion beyond every value actually observed in the current regime.

Two gates that fail open

Worth knowing, because it is surprising: if the tool cannot compute a gate’s input, the candidate passes rather than being declined.

A sample of a single point contributes no residual, so the residual check has nothing to compare against. A sample with no dispersion produces no interval statistic. In both cases the move is trusted.

The reasoning is that the alternative — declining anything we cannot check — would silently suppress findings on exactly the sparse series a user is least likely to be watching. But it does mean a candidate can reach the report with fewer gates actually applied than its ladder suggests.

Reading a gate ladder

Each detector’s gates form a ladder: the candidate meets them in order and stops at the first that declines it. The tables below show one row per gate, in the order the detector applies them. The demand is what the gate required, the computed value is what the candidate supplied, and the verdict states the outcome. A numeric gate passes when the computed value clears the demand. A boolean gate must simply hold. Once a gate declines, the rows below it show not run.

History compares the step and drift models after their cheap gates. If drift fits better, the step ladder can end with selection adjustment and significance marked not run even though no earlier step gate declined: the detector chose the better-fitting drift before paying for step calibration.

A candidate that clears every gate:

And one that is declined:

Declined by residual_noise. The detector computed 15.97 ns, against a demand of 17.38 ns. The gates below it never ran.

Reading a declined ladder is the fastest way to answer “why was my regression not reported?” — and Insights turns it into a checklist.

What the gates hand on

The candidates that survived. History values carry selection-adjusted chance levels; branch values carry the observed range and excess beyond its nearest edge. One question remains, and it is not about any individual candidate: given how many things were tested, should we believe this one?

Multiplicity and coverage

A gate-passing result has cleared every check that looks at its series individually. History detection has also corrected for choices made within that series, such as searching across possible change points. Each mode then asks a different report-wide question:

• History: given how many series were tested, does this candidate survive the group-wide false-discovery correction?
• Branch: among the comparable base-history arrangements, how many showed at least as much out-of-range movement as this branch?

The history answer decides whether the candidate is reported. The branch answer is finite historical context for an excursion that has already been established from the observed current-base range; it never suppresses that factual excursion.

This chapter explains those questions and their consequence — the report’s account of what it actually covered.

Terms used here

History: test enough things and something will look surprising

Each gate-passing candidate cleared a test at a chance level below some threshold. Read that threshold literally: it is the rate at which unchanged series are expected to produce a candidate anyway.

A large unchanged store is therefore likely to produce chance candidates even when the gates are working. As the judged family grows, the expected number of false candidates grows, and so does the probability of seeing at least one.

So a per-series check is not enough. The tool applies a group-wide correction: sort every candidate the mode can report by chance level, and require the strongest to clear a strict bar, the next a slightly looser one, and so on down. What Benjamini-Hochberg controls is the false-discovery rate averaged over many analyses — the expected share of reported findings that are wrong is held below the target, because every rejection is displayed. It is not a promise about any single report: one report can still carry a higher wrong share than the target, and only across many analyses does the average settle back to it.

The family is the 12 series this run judged. The threshold at rank k is k / 12 of the target expected false-discovery proportion, 10.0%, and the largest passing rank keeps the preceding prefix.

The history family is every series that could be tested

This is the part that carries the weight, and it is easy to get wrong.

The correction divides by the number of hypotheses tested, not the number that produced a candidate. If the tool fed the correction only the candidates that survived the gates, it would be dividing by a number that already excludes everything that came out quiet — and the correction would be almost inert.

So the family is every series the analysis judged, including the great majority that raised nothing at all.

One consequence follows immediately, and it is the honest cost of the design: a finding has to clear a stricter bar for a large judged family than for a small one. The same benchmark, the same regression, the same data may report when this analysis judged a small family and stay silent when it judged a large one. The report’s judged count is the denominator.

The same series, the same measurements, the same chance level. A larger family is more likely to produce chance candidates somewhere in the set, so this rank threshold is 166.7× as strict.

That is not a defect to be worked around. It is the cost of controlling the expected false-discovery proportion across repeated analyses when you are testing thousands of things at once. The alternative — a fixed per-series bar — is a report where the expected number of false alarms grows with the size of your suite, which is precisely the failure that makes people stop reading benchmark reports.

The detection minimum is not a reporting guarantee

The detection minimum is the shortest history that can hold two regimes. Being judged is not the same as being able to produce a finding by itself.

A rank comparison on a handful of points has a floor: even a perfect step — every later point above every earlier one — can only be so surprising, because there are only so many ways to assign those ranks. The group-wide correction then asks that lone strongest candidate to clear a bar that shrinks as the judged family grows. At the detection minimum those two numbers miss each other once the family is larger than a handful of series.

A series of 10 points whose later half sits entirely above its earlier half has chance level 0.0159. Rank 1 in a family of 6 still admits it; rank 1 in a family of 7 does not.

The series is still judged. It still counts in the denominator, which tightens the bar for every other series. It just cannot be the only finding in a large family, no matter how large the step.

The family of 2000 used in the staircase above first admits a lone perfect midpoint step at 20 points.

A short series that moves alongside other genuine findings can still be reported: later ranks carry looser bars. The bound above is specifically for a lone regression, which is also the case most worth catching.

Why history filters direction first

In history mode, improvements are neither displayed nor corrected. They leave the candidate set before the correction runs, so only regressions enter the sorted list.

The judged family does not shrink; the denominator remains every series the analysis judged. Only the directions being corrected change. Suppose ten series were judged, and two produced candidates: an improvement with a very low chance level, and a regression with a middling one.

Both orders divide by the same 10 judged series. The tool filters, then corrects: the regression is at rank 1, where it is dropped. Correcting both directions first would leave it at rank 2, where it is kept, before the display would hide the improvement.

Filtering first is the stricter order. It shortens the sorted list, which moves the regression to an earlier rank and therefore a stricter bar. That costs findings: a regression the other order would have kept can be suppressed.

That extra sensitivity is not legitimate for a regressions-only report. It is borrowed from a discovery the reader never sees: the improvement helps the regression pass, then disappears from the display. The full rejected set may still have a false-discovery bound, but the displayed findings do not.

The order is conservative in another way too. The detectors’ chance levels are two-sided, so an unchanged series has at most half that chance to raise a candidate in one named direction. The bar the correction sets is therefore met with room to spare, which is why the target is an upper bound rather than a level the tool aims at.

Branch: compare the complete report with base history

Branch mode has one context observation per series, so it does not manufacture per-series chance levels or feed excursions into the history correction. It keeps every excursion that the observed current-regime range and practical gates support.

To show how remarkable the complete report is, it evaluates comparable reference-lane base commits as historical branch-like turns. Each turn holds out one base commit as the candidate, uses the remaining base observations plus the real branch value as references, applies the same range and noise gates, and sums normalized excess across a shared rectangular series family.

The report states how many historical turns tied with or exceeded the branch score. “None of 10 comparable base commits showed as much out-of-range movement as this branch” means exactly what it says; it is not relabelled as confidence. The factual findings remain visible when no report-wide family can be formed, so limited history reduces the context available to the reader rather than silently changing what was observed.

What a report judged

The same testability decision that builds history’s correction family and branch mode’s eligible series drives the report’s coverage line. In branch mode, that verdict includes unresolved current-base regimes, so the analysis and the report cannot disagree about whether a series was judged.

Text and Markdown headers give the in-scope judged count when anything is in scope. Full Markdown also ends with a Coverage section listing all counts and unjudged reasons; JSON carries that account under census, on every analyze run. How each format presents it is the Reporting chapter’s subject.

A series that was not judged is always accounted for with a reason — never silently skipped.

Reading a silent report

“No notable changes” is not the same statement as “nothing regressed”. It means the judged series produced no reportable move, and the coverage line tells you how much of the accounted-for data was in scope and judged.

Full is the only silent state with no coverage qualification: every in-scope series was judged. The verdict remains “no notable changes detected” for those judged series: no reportable move survived the gates.

This is why automation should gate on coverage rather than on an empty findings list. A run that judged nothing at all reports no findings, and a naive check reads that as success. For report locations and worked examples, see Reading a silent report in Insights.

Ghosts are counted apart. A benchmark the analyzed commit no longer measures is excluded from the coverage denominator, though it is still counted in the totals and named in the breakdown.

The reason is practical. A pull request typically benchmarks only the packages it touches, while analysis reads the whole store — so every untouched package leaves a ghost behind. Counting them would make a perfectly healthy run read as “12 of 2000 judged”, and a ratio that alarming on a good run is a ratio people learn to ignore. See Reconstruction for what makes a ghost.

Configured ignores are also outside the denominator, but remain in the total and per-reason account as ignored. They disclose an intentional policy choice rather than missing baseline evidence. If every measured series is ignored, the report has coverage and outcome nothing_in_scope; it does not claim an all-clear or recommend collecting more history for those exclusions.

What this stage hands on

The findings, and the census. Both go to Reporting — which, as it turns out, gives the census the same prominence as the findings themselves, for the reasons above.

Reporting

Everything so far produced findings and a census. This chapter is how they reach you, and how to read what you get.

Terms used here

Ranking

Findings are ordered by descending relative move, then by analysis method, then by a stable identity tie-break so two runs over the same data produce the same order.

There is deliberately no severity classification. Whether a 4% regression in a hot path matters more than a 40% regression in a rarely-used one is a judgment about your project, and the tool does not have the information to make it. It sorts by the one quantity it can measure, and leaves the ranking of importance to you.

Note in particular that history findings carry no confidence score to sort by. Every reported history finding already cleared its test, so any such number would be uniformly high and would rank almost nothing — see Detection. Branch findings report an observed range plus report-wide historical context instead of a probability score.

Reading a finding

A finding, as the text report prints it:

http_parse/case
  +29.57% wall_time
    regression via change point · 100.4 → 130.1 · somewhere near commit10
 135 ┤           ╭─╮╭─╮╭╮           
 126 ┤         ╭─╯ ╰╯ ╰╯╰─          
 116 ┤         │                    
 106 ┤ ╭╮ ╭─╮╭─╯                    
  97 ┼─╯╰─╯ ╰╯

The chart is drawn against topology, not against the observations. One column per commit, so a gap is a gap and the trailing stretch after the last observation is visible as one. That matters: a chart that packed the observations together would hide exactly the sparseness that detection cannot see.

Where a series has more commits than the chart has columns, commits are grouped before plotting. Grouping first is deliberate — the alternative attenuates an isolated observation surrounded by gaps into nothing.

Analysis outcomes

The analysis outcome is the result of a successful analysis, combining findings with the series coverage state. Findings take precedence: an analysis with any findings has outcome findings, even when some series could not be judged.

Without findings, a fully judged in-scope suite is clean. An in-scope suite with no judged series is insufficient_baseline; a suite with some judged and some unjudged series is partial. When no series entered analysis, or every accounted series was absent at the analyzed context commit or ignored by configuration, the outcome is nothing_in_scope.

These outcomes describe the series available to analysis, not an inventory of collection jobs. Platform coverage is separate: matrix automation must verify which expected platforms completed collection before presenting a complete all-clear. A missing collection leg can coexist with a clean analysis of the available series. Analysis failure is an execution error, not an outcome.

The formats

The tool emits reports and their derived outputs in these forms, each requested independently:

• Text — the default terminal report.
• Markdown (--markdown <path>) — every finding plus complete coverage counts and reasons, for a pull request or an issue.
• JSON (--json <path>) — the complete machine-readable result.
• Condensed summary (--markdown-summary <path>, analyze only) — a short, capped Markdown digest for a size-limited destination such as a pull request comment.
• Outcome (--outcome <path>, analyze only) — the stable analysis outcome for selecting a message without parsing the full JSON report.

Full Markdown carries every finding and ends with a Coverage section containing the complete counts and unjudged-reason breakdown, including ghosts. JSON always carries the complete census under census, which makes it the machine-readable signal: each finding self-describing, plus every unjudged reason. Text omits that breakdown when findings exist. JSON deliberately omits the per-commit chart series — that is presentation, and examine is the way to get the underlying points. The condensed summary is lossy by design — capped, and flattened so the per-set grouping is dropped — so it is the one output you must not automate against. The outcome file repeats only the JSON report’s top-level outcome field; it carries no report details. The full table is under Where output goes.

Here is the same analysis in text and JSON. The JSON excerpt illustrates the shape automation reads, not a complete field catalog. For matching full Markdown and JSON examples, see Reading a silent report.

Text — the default output, printed to standard output.

Analyzed project textproc (history mode)
  commit: 9f2c4a1d3b5e708aab12cd34ef5678901234567a
  runs: 128 (4d17b0c93ea2 → 9f2c4a1d3b5e)  in-scope series judged: 5 of 8  ignored by configuration: 1 series  regressions: 2

criterion/x86_64-unknown-linux-gnu/a1b2c3d4
  runs: 128  regressions: 2
  filter: --engine criterion --target-triple x86_64-unknown-linux-gnu --machine-key a1b2c3d4

http_parse/case
  +29.57% wall_time
    regression via change point · 100.4 → 130.1 · somewhere near commit10
 135 ┤           ╭─╮╭─╮╭╮           
 126 ┤         ╭─╯ ╰╯ ╰╯╰─          
 116 ┤         │                    
 106 ┤ ╭╮ ╭─╮╭─╯                    
  97 ┼─╯╰─╯ ╰╯                     

index_build/case
  +10.40% wall_time
    regression via drift · 98.5 → 108.7 · accumulated commit0 → commit29
 113 ┤                    ╭╮    ╭── 
 109 ┤          ╭╮╭╮╭─╮   ││ ╭╮╭╯   
 105 ┤ ╭╮     ╭─╯╰╯╰╯ ╰─╮╭╯╰─╯╰╯    
 101 ┼╮│╰─────╯         ╰╯          
  97 ┤╰╯

JSON — the same analysis, written by --json.

{
  "project": "textproc",
  "tip_commit": "9f2c4a1d3b5e708aab12cd34ef5678901234567a",
  "tip_dirty": false,
  "mode": "history",
  "outcome": "findings",
  "notable": true,
  "runs": 128,
  "series": 8,
  "regressions": 2,
  "ghosts_excluded": 2,
  "census": {
    "total": 11,
    "in_scope": 8,
    "judged": 5,
    "unjudged": 6,
    "coverage": "partial",
    "reasons": [
      {
        "reason": "ghost",
        "count": 2
      },
      {
        "reason": "ignored",
        "count": 1
      },
      {
        "reason": "too_few_points",
        "count": 3
      }
    ]
  },
  "findings": [
    {
      "engine": "criterion",
      "target_triple": "x86_64-unknown-linux-gnu",
      "machine_key": "a1b2c3d4",
      "segments": [
        "http_parse",
        "case"
      ],
      "kind": "wall_time",
      "method": "change_point",
      "direction": "regression",
      "baseline": 100.36830218253652,
      "latest": 130.05171405014943,
      "relative_delta": 0.29574488381430086,
      "commit": "commit10"
    },
    {
      "engine": "criterion",
      "target_triple": "x86_64-unknown-linux-gnu",
      "machine_key": "a1b2c3d4",
      "segments": [
        "index_build",
        "case"
      ],
      "kind": "wall_time",
      "method": "drift",
      "direction": "regression",
      "baseline": 98.50313347697937,
      "latest": 108.7433855201543,
      "relative_delta": 0.10395864254986495,
      "commit": "commit29",
      "window_start": "commit0"
    }
  ],
  "sets": [
    {
      "engine": "criterion",
      "target_triple": "x86_64-unknown-linux-gnu",
      "machine_key": "a1b2c3d4",
      "runs": 128,
      "series": 8,
      "regressions": 2
    }
  ]
}

Findings never fail the build

The exit code reflects whether the analysis ran. A report full of regressions exits successfully; a report that could not reach its storage backend does not.

Why: a benchmark tool that breaks builds on a measurement gets disabled, and a disabled tool detects nothing. Findings are advisory by design.

What this means for automation: write the JSON report to a file with --json. Its top-level outcome field carries the analysis outcome; the notable boolean is true exactly for findings. Use --outcome <path> when this value is all the caller needs, and retain separate collection evidence for matrix-wide claims. The full JSON remains necessary for counts, reasons and findings.

The coverage line is not decoration

Text and Markdown headers state how many in-scope series were judged when anything was in scope. JSON records those counts in census. This terminal example shows a silent report:

A report with nothing to say still says how far that silence reaches:

Analyzed project textproc (history mode)
  commit: 9f2c4a1d3b5e708aab12cd34ef5678901234567a
  runs: 128 (4d17b0c93ea2 → 9f2c4a1d3b5e)  in-scope series judged: 3 of 6  regressions: 0
No notable changes detected among the series that were judged.
  Judged 3 of 6 in-scope series; no reportable move survived the gates.
  Not judged: 2 series not measured at the analyzed context commit; 1 series ignored by configuration; 3 series with too few points in the analyzed window.

The judged ratio heads each human-readable report that had anything in scope. Text prints the per-reason breakdown only when there are no findings, as here. Full Markdown always includes it in the final Coverage section, and JSON always carries it under census.reasons.

This is given the same prominence as the findings on purpose. “No notable changes” means nothing changed among the series this run was able to judge, and without the denominator you cannot tell that statement apart from “the benchmarks did not run”.

The final Coverage section in full Markdown and the JSON census always contain the reasons series went unjudged, including when findings exist. Text and condensed summaries list the full reason breakdown only when there are no findings. Configured exclusions also receive a compact disclosure beside findings, so an intentional reduction in analysis scope stays visible. When in-scope series remain unjudged, the condensed summary’s coverage bullet points to the full report for details, without a separate GitHub warning. See Reading a silent report for matching Markdown and JSON examples.

Comparison-base lag

Branch mode compares the analyzed context commit against the base ref’s recent commits within the same discriminant set. On a rotating CI pool, the newest base-ref commits may only carry data under a different machine key — so your runner compares against base data from several commits back.

The empty stretch after the comparison base is the lag: commits that exist on the base ref, but not for the context run’s machine key.

The report says so, naming how far behind and which of two reasons applies:

The lag is advisory. It never changes which findings are reported and never affects the exit code; what it changes is how much weight a marginal branch finding deserves, because a comparison against a base state several commits old is exactly that.

It is advisory. It never changes which findings are reported, and never affects the exit code. What it changes is how much weight you should give a marginal branch finding: a comparison against a base state ten commits old is a comparison against a base state ten commits old.

Branch range and historical context

A branch finding does not borrow history-mode change-point wording. It states the measured context value, the lowest and highest values observed in the selected current-base regime, and the excess beyond the nearest edge. “Slower than all 20 current-base observations” is a claim about those recorded observations, not a probability estimate.

The discriminant-set section then gives the report-wide historical comparison, for example:

None of 10 comparable base commits showed as much out-of-range movement as this branch
(12 series compared).

When too little shared history exists, the report says the comparison could not be formed and still prints every factual range excursion. JSON exposes the same distinction: finding-level branch range metadata and set-level branch_comparison counts.

Where output goes

The text report goes to stdout. Diagnostics, the effective-selection summary and verbose notes go to stderr.

Warnings are the exception, and it is deliberate: a comparison-base lag notice, or the note that a dirty run was admitted on the base tip, is rendered into the report body rather than onto stderr. Those qualify the findings printed beside them, so a reader who captured only the report would otherwise lose the caveat that changes how to read it.

The other formats are written to files rather than to a stream — --markdown, --json, --markdown-summary and --outcome each take a path, and --no-text suppresses the stdout report when you want only those.

JSON is the complete machine-readable result: it always carries every finding and the full per-reason census, including the ghost count. It is not an observation archive — the per-commit series behind the charts is obtained with examine. Full Markdown always includes the same coverage counts and reasons in its final Coverage section. Text omits the full reason breakdown when findings exist. Configured exclusions remain visible beside findings on every surface. The condensed summary is lossy by design, so it is the one output that must not be automated against: a check reading it cannot distinguish findings that were capped away from findings that were never made. The outcome file repeats only JSON’s top-level outcome field; it carries no report details.

What comes next

Nothing — this is the end of the pipeline. If a report left you with a question rather than an answer, Insights is the triage chapter.

Insights

Everything before this chapter explains what the tool does. This one is about what you do with the result.

Start with the question you need to answer:

• I have a finding — something was reported and you need to act.
• Reading a silent report — no findings were reported, and you need to know what was actually judged.
• I expected a finding and got none — the checklist for tracking down where a real regression was dropped.

I have a finding

flowchart TD
    F["A finding"] --> Q0{"Which mode<br/>produced the report?"}
    Q0 -->|"branch"| BR["It compares your tip<br/>to the base"]
    Q0 -->|"history"| Q1{"Which method?"}
    Q1 -->|"change point"| CP["It names a nearby commit"]
    Q1 -->|"drift"| DR["It names a window"]
    CP --> Q2{"Did anything else move<br/>at the same commit?"}
    Q2 -->|"many series"| SHARED["Look for a shared cause:<br/>code or environment"]
    Q2 -->|"one or a few"| CODE["Suspect the local change"]
    SHARED --> CORR["Correlate metrics with<br/>the diff and run metadata"]
    CORR --> ACC["Fix, or bless if intended"]
    CODE --> FIX["Fix, or bless if intended"]
    DR --> WIN["Examine the series;<br/>look for many small increments"]
    BR --> TIP["Re-run the tip;<br/>check for a lag warning"]

Note that the method named on a finding tells you which comparison produced it: change point and drift come from history mode, branch excursion from branch mode. The report’s header tells you which mode ran.

A change point

The finding names a commit somewhere near the change, not the exact first commit that introduced the new level. A specific causal commit is not always identifiable; if collection is sparse, the cause is somewhere in the gap before the named commit.

1. Look at the series. cargo bench-history examine --benchmark <qualified-id> --metric <name> prints every stored point. Both values come from the finding. The chart in the report is a summary; this is the data.
2. Check whether the named commit has a neighbor gap. If the previous observation is twenty commits back, your suspect list is those twenty commits, not one.
3. Check what else moved. Many simultaneous steps suggest a shared cause, not a specific one. The cause may be shared code such as an allocator, runtime or dependency, or an environment change such as the toolchain, runner, or hardware. Correlate the affected metrics with the commit diff and run metadata before attributing it.
4. Decide. Fix it, or accept it with bless, which re-baselines the series so it stops being re-reported.

A drift

A drift names a window, not a commit, because that is what the evidence supports: no single commit is responsible.

The usual cause is genuinely incremental — a container that grew, an abstraction that gained a layer per change, a loop that got one more thing to do each month. That is exactly the class of regression that comparing against the previous release will never catch, which is why the tool looks for it.

1. Examine the series and look at the shape. A true drift climbs steadily.
2. If it climbs in visible steps instead, the drift detector won on residuals but the real story is several small change points. Bisect: analyze with --context set to a commit partway through the window, and see where the level sat then.
3. Check the window’s length. A drift over a long window with sparse observations may be two distant levels with nothing in between. The chart shows the gaps to scale.

A branch finding

Branch mode judges only the analyzed context commit, against the base ref’s observed current-regime range. Your branch’s intermediate commits are ignored — only the context state is evaluated.

1. Re-run the benchmark on the context commit. Branch findings rest on a small sample, often a single run. Inspect the new engine output directly: ordinary CI collection uses --skip-existing, which preserves existing stored measurements. Replacing stored results requires an explicit collect --overwrite.
2. Look for a comparison-base lag warning. On a rotating CI pool, your runner’s machine key may only have base data from several commits back. The comparison is still valid, but it is against an older base state than you might assume.
3. Check whether the base itself recently moved. If the base stepped a few commits ago and your branch merely matches the new level, that is worth knowing before you go looking through your own diff.
4. Read the comparable-base count literally. It tells you how many eligible base commits produced at least as much report-wide out-of-range movement. It is historical context, not a confidence percentage.

Reading a silent report

“No notable changes detected” applies only to metric series that were actually judged. One benchmark can contribute several metric series, so these counts are not benchmark counts.

Where to find coverage

The GitHub issue or pull-request comment contains a condensed summary, not the full report. When present, Download the full report bundle links to the full outputs:

• Full Markdown (report.md in the bundle, or the path passed to --markdown) ends with a Coverage section. It lists the coverage state, accounted-for, in-scope, judged and unjudged counts, and a count for each unjudged reason. It is present even when findings exist.
• JSON (report.json in the bundle, or the path passed to --json) carries the same account under census; the breakdown is census.reasons.
• Text on stdout and condensed Markdown summaries show the in-scope judged ratio in their headers when anything is in scope. They list unjudged reasons only when there are no findings, while configured ignored-series counts remain visible beside findings.

The warning about in-scope series that could not be judged refers to the full Markdown Coverage section or the JSON census, not to additional detail hidden in the comment. Publication can omit the bundle link. In that case, obtain the full outputs from the workflow that produced the comment, or rerun analyze with the reported commit and discriminant filters, adding --markdown report.md --json report.json. Use the same stored measurements and selection to reproduce the report.

For individual series identities, rerun the same analysis with --verbose: its diagnostics name each unjudged series, the evidence it carried and the rule that declined it.

A silent analysis in Markdown and JSON

These examples are generated from the same analysis using the production renderers. The Markdown example is file content, not terminal stdout; the JSON example contains its matching census object.

Full Markdown report — written by --markdown.

# Benchmark history analysis: textproc

- Commit: 9f2c4a1d3b5e708aab12cd34ef5678901234567a
- Mode: history
- Runs analyzed: 128 (4d17b0c93ea2 → 9f2c4a1d3b5e)
- In-scope series judged: 3 of 6
- Regressions: 0

No notable changes detected among the series that were judged.

Judged 3 of 6 in-scope series; no reportable move survived the gates.

Not judged: 2 series not measured at the analyzed context commit; 1 series ignored by configuration; 3 series with too few points in the analyzed window.

## Coverage

- State: `partial`
- Metric series accounted for: 9
- In scope: 6
- Judged: 3
- Unjudged (including out-of-scope series): 6

| Unjudged reason | Metric series |
| --- | --- |
| not measured at the analyzed context commit | 2 |
| ignored by configuration | 1 |
| with too few points in the analyzed window | 3 |

JSON report — the same analysis from --json, with counts under census.

{
  "project": "textproc",
  "tip_commit": "9f2c4a1d3b5e708aab12cd34ef5678901234567a",
  "tip_dirty": false,
  "mode": "history",
  "outcome": "partial",
  "notable": false,
  "runs": 128,
  "series": 6,
  "regressions": 0,
  "ghosts_excluded": 2,
  "census": {
    "total": 9,
    "in_scope": 6,
    "judged": 3,
    "unjudged": 6,
    "coverage": "partial",
    "reasons": [
      {
        "reason": "ghost",
        "count": 2
      },
      {
        "reason": "ignored",
        "count": 1
      },
      {
        "reason": "too_few_points",
        "count": 3
      }
    ]
  },
  "findings": [],
  "sets": [
    {
      "engine": "criterion",
      "target_triple": "x86_64-unknown-linux-gnu",
      "machine_key": "a1b2c3d4",
      "runs": 128,
      "series": 6,
      "regressions": 0
    }
  ]
}

Interpreting the counts and reasons

The in-scope count excludes ghosts: metric series whose benchmark was not measured at the analyzed context commit. A pull request often benchmarks only affected packages while the analysis reads a wider history, so an untouched package’s benchmarks can be ghosts. They still appear in the accounted-for and unjudged totals and in the reason breakdown. They do not prevent full coverage of the in-scope suite.

Configured ignore prefixes also exclude matching series from the in-scope count while preserving them in the total and reason breakdown. They describe an intentional policy choice, not missing baseline evidence. Removing a prefix restores eligibility of the stored measurements. If the suite contains only ignored series and ghosts, its outcome is nothing_in_scope, not an all-clear.

The following reasons explain why series were not judged:

Judged 0 of N means nothing was tested, not that measured levels stayed flat. Check the reason breakdown: history may still be accumulating, a blessing may have limited usable evidence, or current measurements may be absent. When nothing is in scope, the report says so instead of printing a ratio. When no series was accounted for, it says nothing was analyzed.

The JSON census.coverage field and Markdown’s Coverage state use the same tokens: no_series, nothing_in_scope, nothing_judged, partial and full. Only full means every in-scope series was judged. Even then, silence means no reportable move survived the gates, not proof that nothing changed.

Findings do not imply full coverage: a report can identify a regression while leaving other series unjudged. Platform coverage is separate again. A missing collection platform can leave the available series fully judged without establishing an all-clear for every platform.

The tool reports that a measured level moved, not why it moved. Attributing a move and deciding whether to accept it remain your judgment, recorded with bless.

Unreliable or inconsistent hardware

Symptoms:

• Many unrelated benchmarks step at the same commit without a corresponding shared-code change.
• A series oscillates between two levels rather than wandering around one.
• Findings appear and disappear between runs with no code changes.
• The report’s coverage drops, or comparison-base lag warnings appear.

Diagnosis:

• cargo bench-history list discriminants shows how your history is partitioned. Several machine keys where you expected one means the pool is rotating.
• cargo bench-history machine-key --verbose explains what the current machine hashed to and why.

Remedies:

• backfill on each machine so each key accumulates its own usable history.
• For a one-off infrastructure step you accept, bless --all re-baselines everything at once.

Unreliable benchmarks

A noisy benchmark is not merely unpleasant — it is actively harmful to detection. In history mode the residual gate requires a move to exceed a multiple of the series’ typical residual. In branch mode every current-regime extreme expands the observed range the context must clear. Noise does not just add false alarms; it hides real regressions.

How to recognize one:

• The series oscillates between two levels instead of varying around one. Usually a code path that gets taken sometimes: a cache that sometimes hits, an allocation that sometimes triggers a resize.
• Occasional points far from the rest. Usually the operating system — a scheduler decision, a page fault, another process.
• Drift that tracks nothing in your code. Often a data structure that grows across iterations.
• Values that differ run to run with no code or environment change. Usually unseeded randomness — most often the hashers behind HashMap/HashSet, whose seed is randomized per process, but also any randomized algorithm or sampled input — reshuffling the work each time.

Fixes, roughly in order of effect:

1. Measure instruction counts instead of wall time where you can. Callgrind’s counts are the least noisy thing available, at the cost of not measuring what the user experiences.
2. Pin instruction layout. An unrelated dependency bump can move a hot loop across a cache line boundary and produce a phantom regression with no source change. See Measurement stability.
3. Use --best-of N to take the minimum of several runs. Note the caveat below.
4. Remove the variable path from the benchmark itself: pre-warm caches, pre-size buffers, fix the input.
5. Seed all randomness. Every random choice a benchmark makes must come from a fixed seed, including the hashers behind HashMap and HashSet (their default seed is randomized per process) — otherwise each run measures a subtly different workload.

--best-of is a protocol, not a flag. The stored value depends on how many repetitions you ran, so changing N shifts every series at once and reads as a suite-wide step. Pick a value and keep it. If you must change it, expect a step and bless it.

Multithreaded and OS-waiting benchmarks

These deserve their own mention because they are the worst case.

A benchmark that waits on another thread, a lock, the filesystem, or the network is measuring the operating system’s scheduling decisions at least as much as your code. The scatter is large, and it is not symmetric — waits have a floor and a long tail.

The consequence follows from the residual gate: the band widens until only enormous moves clear it. The benchmark still runs, still records, still looks like coverage in the census — and detects almost nothing.

What to do instead: measure the single-threaded work directly, and test concurrency behavior with something other than a benchmark. If you must measure a concurrent path, instruction counts degrade far more gracefully than wall time.

I expected a finding and got none

Work down this list. Each step names the chapter that explains it.

1. Was the series judged at all? Read the full Markdown report’s final Coverage section, the JSON census, or the verbose per-series reasoning. An unjudged series is never a silent skip — the reason is stated. → Multiplicity and coverage
2. Is it a ghost? If the benchmark did not run at the analyzed commit, every one of its metrics was dropped before detection. This is the single most common cause of a surprising silence. → Reconstruction
3. Is there enough history? Both sides of a step need enough commits. A benchmark added recently cannot produce a change point yet. A series that has just cleared the detection minimum is judged, but a lone step there cannot survive a large family’s correction. → Detection, Multiplicity and coverage
4. Does the move clear both floors? Relative and absolute. A large percentage of a tiny number does not qualify. → Noise gates
5. Is the series too noisy? In history mode, compare the move with the series’ residual scatter. In branch mode, check whether historical extremes expanded the observed current-base range. → Noise gates
6. What does the report-wide context say? For history, a larger judged family sets a stricter false-discovery bar — strict enough that a series at the detection minimum cannot report a lone step in a large family. For branch mode, read how many comparable base commits tied or exceeded the branch score. → Multiplicity and coverage
7. Was it an improvement in history mode? History mode watches for regressions only, so a move that made things faster never becomes a finding there. → Reporting
8. Was it blessed? A blessing re-baselines the series, so moves before the blessed commit are no longer considered. → Reconstruction

If none of these explains it, the fastest way to see exactly which check declined the candidate is to look at the series directly with examine and compare it against the gate order in Noise gates.

Limits

What the pipeline deliberately does not do, why, and what to do instead.

Some are deliberate trade-offs and others are limits of the current statistical model. Knowing them is what keeps you from reading more into a report than its evidence supports.

The first few are stated in full in the chapter that owns them; they are listed here so that one page answers “what will this tool not tell me?”.

Very long histories use the newest 1,000 points

Detection is designed for histories from dozens to a few hundred measurements. When a series grows beyond 1,000 points, only its newest 1,000 influence the verdict. The older measurements remain stored and visible in charts; they simply stop pulling the detector toward performance levels that are no longer current.

What to do: use examine when you need the full recorded shape. To judge an older era rather than the current one, move the analysis context to a commit in that era.

A short series cannot report a lone finding in a large family

The detection minimum is the shortest history that is judged. That series still counts toward the false-discovery family, but a lone perfect step at that length cannot clear the group-wide bar once the family grows past a handful of series.

Why: a rank comparison on so few points has a floor, and the rank-1 bar shrinks with the family. Those two numbers miss each other well before a production-scale suite.

What to do: collect more history on the series you care about. The coverage chapter shows how family reach grows with length. See Multiplicity and coverage.

Distance between commits is invisible

Detection is positional: a series with ten observations spread across a thousand commits and one with ten across ten consecutive commits are, to the detectors, the same series.

What to do: keep collection dense enough that a drift finding’s window means what you think it means, and read the chart, which does draw gaps to scale. See Reconstruction.

A benchmark missing at the tip disappears entirely

Ghost elimination looks at exactly one commit: the analyzed one. A benchmark measured on ninety-nine of a hundred commits, but not the tip, contributes nothing.

What to do: if a run failed at the tip, collect it again rather than reading the report. The census names ghosts, so the report always tells you this happened. See Reconstruction.

Analysis does not fail because it found something

The exit code reflects whether the analysis ran, not what it found: a report full of regressions still exits successfully. Whether a finding should fail a build is a decision for the automation that reads the report — nothing about a finding makes it fail one on its own, and the tool does not signal analysis failure merely because findings exist. Findings carry no severity classification either; they are ranked by size of move and nothing else.

What to do: read the JSON report and decide in your own automation, gating on coverage as well as on findings. See Reporting.

Benchmark identity can collide, and the estimator is invisible

Criterion identities carry no package attribution, so two identically-named benchmarks in different crates share a series; and the stored record does not say whether a value came from the slope or the mean.

What to do: keep benchmark names unique across the workspace. See Shape of the data.

One shared change can produce many findings

There is no cross-series aggregation. A change that slows down shared code or shared runtime behavior surfaces once per affected series, independently judged.

The group-wide correction has a consequence here that runs against intuition. It compares each candidate against a bar that rises with its rank, so co-occurring findings are judged against more generous bars than a single isolated one would be. Many simultaneous findings are therefore easier to report, which means a wall of findings is weaker per-finding evidence than its length suggests, not stronger.

Why: aggregating would require knowing which series share a cause, which is exactly the attribution the tool refuses to guess at.

What to do: read a report with many simultaneous findings as one event rather than a count, and look for the common cause before investigating any of them individually. That pattern is itself diagnostic — see Insights.

No certainty score is reported

The report gives no confidence or certainty score. History detection reaches its findings through chance levels; branch detection reaches its findings by comparing the context run against an observed range, which produces no chance level at all. The two are not measurements of the same thing, so there is no scale both could be placed on. The group-wide correction history applies afterwards does not turn its chance levels into a comparable finding score either.

What to do: rank findings by size of move, which is the comparable quantity the report provides.

Branch historical comparison is finite context, not probability

Branch mode reports an excursion when the context value lies outside the observed current-base range. Its report-wide historical comparison then counts comparable base commits whose complete report score tied with or exceeded the branch score.

That count is limited by the base evidence available after windowing, regime selection, and blessings. It does not estimate all possible future benchmark outcomes and is not reported as a p-value or confidence. With too few shared reference-lane commits, no report-wide comparison is shown; factual per-series excursions remain.

What to do: read the count literally and inspect the chart or examine when the base range appears to contain more than one operating condition. Gather more ordinary base measurements when you need a broader empirical comparison.

A recent blessing can leave branch evidence short

A blessing is an intentional evidence boundary in both modes. Branch mode keeps the blessed base commit and excludes every earlier one, even when that leaves too few base commits to judge the branch.

Why: using evidence the blessing explicitly retired would silently undo the user’s decision. The report names the shortfall rather than falling back across the boundary.

Cross-machine comparison is never attempted

Results from different machine keys are never compared, only reported side by side. If your CI pool rotates machines, some series will have less base data than the commit count suggests.

What to do: the tool discloses this as a comparison-base lag warning rather than silently comparing. Run backfill on each machine so every real hardware partition accumulates a usable history.

Some stored facts are never read back

The store records more than the analysis uses:

• std_dev, which Criterion reports, is stored and never consulted. Analysis uses confidence intervals for dispersion checks, history residuals for history noise, and observed ranges for branch excursions.
• machine — the full hardware description behind the machine key — is provenance for a human, not an input.
• best_of records how many repetitions a run took, and does not affect analysis.

Why: they cost almost nothing to record and are impossible to recover retroactively. A future version can use them; a missing history cannot be back-filled with facts nobody wrote down.

Glossary

Every term the pipeline chapters define, in one place.

The chapters are written for engineers rather than statisticians. Where a plain description works as well as the textbook name, the description is what the prose uses — but the textbook name is recorded here too, because it is what you need if you want to read further about a method.

Terminology is deliberately narrow here: dispersion is the engine-reported precision attached to one point, while scatter is the between-commit variation of a reconstructed series. The terms are not aliases.

A note on what these are not

The definitions above are working descriptions, precise enough to follow what this tool does. They are not textbook definitions, and several deliberately gloss over conditions that do not arise here — a rank comparison, for instance, is described as evidence of a difference rather than as the assumptions the test makes about the data.

Where the distinction matters to how the tool behaves, the chapter says so at that point.

Reference tables

Key generated facts from the pipeline chapters, gathered for lookup. The owning chapters hold the complete explanations; this page is a reference, not a substitute for the reasoning.

Every table here is generated from the code, so it cannot drift from the tool’s behavior.

Data

What each engine measures

See Shape of the data.

How many series one benchmark yields

See Shape of the data.

Which engines report dispersion

See Shape of the data.

How benchmark identity is formed

See Shape of the data.

Storage

The stored run

See Shape of the data.

run
├── schema_version          5
├── context
│   ├── observed_at         when the measurement was taken (provenance only)
│   ├── git                 commit, branch, and whether the tree was dirty
│   ├── environment         local, or the CI provider that ran it
│   ├── toolchain           target triple and compiler version
│   ├── tool_version        which cargo-bench-history wrote this
│   ├── machine             host hardware description (recorded, never read)
│   └── best_of             repetitions run, when --best-of was used
└── results[]               one per benchmark case
    ├── id                  the benchmark identity
    └── metrics[]           one per metric kind, each with
        ├── value           the measurement
        ├── std_dev         where the engine reports one (recorded, never read)
        └── interval        low and high, where the engine reports them

Key grammar

See Shape of the data.

v1/wordcount/objects/callgrind/x86_64-unknown-linux-gnu/a1b2c3d4e5f60718/4f2a1c9d8e7b6a5f4e3d2c1b0a9f8e7d6c5b4a39/clean.json

Object kinds

Collision policies

See Collection.

What the machine key hashes

See Collection.

Analysis

Mode selection

See Selection.

Minimum evidence

See Detection.

Every gate, per detector

See Noise gates.

Each detector applies its own gates in its own order, and a candidate stops at the first gate that declines it. A gate several detectors share is one policy asked at a different point in each sequence.

change_point — a level that moved and stayed moved

drift — a level that is moving steadily