Shape of the data

Before anything can be analyzed, it has to be measured, reduced to a common form, and stored. This chapter is that form: what one benchmark produces, what survives into the store, and what the store looks like.

It is the only place in this guide that documents the storage layer. If you are debugging why two results are not being compared, or what a directory of JSON files means, this is the chapter.

Terms used here

One benchmark is not one number

The most common surprise: a benchmark does not produce a timeseries. It produces several — one per metric kind the engine reports.

So four benchmark identities in a suite can easily be eight or more series, and every one of them is judged independently. That is why a store’s series count is larger than the benchmark count, and why the false-discovery family grows faster than you might expect.

What each engine measures

Everything is lower-is-better. There is no metric where a larger number is an improvement, which is what lets the tool talk about regressions and improvements without knowing what a benchmark does.

What Callgrind deliberately throws away

Callgrind reports far more than three counts — cache hits at each level, estimated cycles, branch mispredictions. Only the instruction count and the two branch counts are stored.

The discarded ones are simulated and highly sensitive to code layout: they can swing by tens of percent between builds of identical source, because the simulator’s model of the cache depends on where functions landed in memory. Storing them would add series that regularly report large moves for no reason — the precise failure this tool exists to avoid.

Which number is taken

Each adapter picks one value per metric, and the choice matters more than it looks:

• Criterion prefers the regression slope over the mean where a slope is available. The slope is fitted across iteration counts and is much less sensitive to a slow first iteration.
• alloc_tracker and all_the_time take the per-iteration slope, never the recorded totals. The optional alloc_tracker peak uses the producer’s weighted span-peak estimate and its own confidence interval.
• Callgrind takes the new side of the pair it reports.

Two consequences worth knowing. The stored record does not say which estimator was used, so a Criterion benchmark that stops producing a usable slope falls back to the mean silently — and because the mean and the slope disagree, that switch can read as a step that no code caused. And for the required allocation totals and processor-time metrics, a missing or non-finite slope drops that operation from the run rather than falling back to anything, so a benchmark can simply disappear from a series, with no error to say why.

An unavailable allocation peak drops only that metric, preserving the operation’s bytes and allocation count. Peak recording is default-on; benchmarks with unsuitable memory lifetimes disable it in the producer with ThreadSpan::no_peak(), as described under Allocation peaks.

Dispersion: what the engine tells you about its own precision

Some engines report a confidence interval alongside the value.

The rule that matters is the same throughout the pipeline: a confidence interval can only ever take a candidate away. No gate uses one to create or strengthen a candidate. So an engine that reports none is not held to a weaker standard — it is judged on its own between-commit scatter instead, which is arguably the more relevant quantity anyway.

Benchmark identity

An identity is an ordered list of name segments, rendered with slashes. Each adapter builds it from what its engine knows:

Renaming a benchmark therefore starts a new series and retires the old one — the tool has no way to know the two are related. See Reconstruction.

Note which identities carry no package attribution: Criterion (group and function only) and the operation engines alloc_tracker and all_the_time (the operation name alone). Only Callgrind is fully qualified. So two identically-named benchmarks in different crates of one workspace share a series. See Limits.

What a stored run holds

One run is one engine’s whole output at one commit, written as a single object.

run
├── schema_version          5
├── context
│   ├── observed_at         when the measurement was taken (provenance only)
│   ├── git                 commit, branch, and whether the tree was dirty
│   ├── environment         local, or the CI provider that ran it
│   ├── toolchain           target triple and compiler version
│   ├── tool_version        which cargo-bench-history wrote this
│   ├── machine             host hardware description (recorded, never read)
│   └── best_of             repetitions run, when --best-of was used
└── results[]               one per benchmark case
    ├── id                  the benchmark identity
    └── metrics[]           one per metric kind, each with
        ├── value           the measurement
        ├── std_dev         where the engine reports one (recorded, never read)
        └── interval        low and high, where the engine reports them

The context is provenance. Only some of it drives behavior:

That last row is not an oversight — see Limits.

The observation timestamp deserves emphasis: when a benchmark was measured has no bearing on where it sits in a series. Position comes from git topology, read at analysis time. A commit benchmarked out of order, or backfilled months later, lands exactly where its commit sits.

Where it all lives

A run’s storage key is a path with a fixed grammar:

v1/wordcount/objects/callgrind/x86_64-unknown-linux-gnu/a1b2c3d4e5f60718/4f2a1c9d8e7b6a5f4e3d2c1b0a9f8e7d6c5b4a39/clean.json

Concrete partition directories hold runs and any partition-local blessing records:

Project-level blessing records store their blessing scope explicitly and live under v1/<project>/objects/blessings/<commit>/bless-<issued_unix_nanos>.json. Each record carries benchmark prefixes and a required scope object containing engine, target_triple and machine_key arrays. An empty array leaves that axis unrestricted. The scope is matched at query time rather than expanded into the partitions present at issuance. Partition-local blessing records derive their exact scope from their keys and remain effective. See bless / unbless for acceptance and revocation.

Each path segment is sanitized: characters outside a safe set are replaced, and the result is lowercased. That is what keeps a key usable as both a filesystem path and a blob name — but it also means identifiers differing only in case or punctuation collapse into one partition.

Bodies are compressed at the storage boundary, so a local store is a tree of gzipped JSON.

Atomicity, and what it does not cover

One stored object is written whole or not at all, and never rewritten: normal collection is append-only. That is what keeps caches valid and history trustworthy.

The unit of atomicity is one engine’s output at one commit — not the whole collection. Engines are stored one at a time, so a run killed part-way through can leave a commit holding Criterion data but not Callgrind. Nothing is corrupt, but the commit is incomplete, and a later backfill will consider it already covered.

If a collection was interrupted, re-collect that commit with --overwrite rather than assuming backfill will notice.

Taking the minimum of several runs

--best-of N runs the whole suite N times and keeps, per benchmark case and per metric kind, the smallest value.

Two details follow from “per metric kind”:

• A stored result can mix metrics chosen from different repetitions — the instruction count from run 2, the branch count from run 3.
• The winning metric is kept whole, so its confidence interval travels with the value it belongs to. Intervals are never merged or recomputed.

The minimum is a useful heuristic here because shared-runner contention commonly adds delay. Keeping the smallest observed value reduces the influence of transient positive interference, without guaranteeing that the chosen value is disturbance-free.

--best-of is a protocol, not a flag. The stored value depends on N, so changing it shifts every series at once and reads as a suite-wide step. See Insights.

What this stage hands on

A store: immutable objects, keyed by partition and commit, each holding one engine’s reduced output. Next: Collection, which is how they get there.