Comparability and partitioning

The central tenet: partition only by what makes results fundamentally incomparable; record everything else as metadata so the analysis can see its effect over time.

Discriminant sets

Two results are comparable exactly when they were measured in the same project and their discriminant sets match. A discriminant set is { engine, target_triple, machine_key }:

• engine — different units and semantics never mix.
• target_triple — even simulated counts are not comparable across architectures.
• machine_key — always present: a fingerprint of the host hardware the benchmark ran on. Every engine is partitioned by it, because every engine’s numbers vary with the hardware in practice. See Collection.

The project — workspace identity, configured and defaulting to the repository directory name — partitions too, but it sits one level above: it selects which store is being read, so results from two projects never meet in the first place. That is why it is not part of the discriminant set proper.

Deliberately metadata, not partition — so a change shows up as a timeline step, which is the whole point of the tool — are the toolchain versions, OS/libc, commit, branch, the environment provider, and the --best-of repetition count the run’s values were reduced from. A rustc bump therefore appears as a step in the series rather than silently forking history.

Read the repetition count differently from the rest, though. Because the reduction keeps a minimum, and the expected minimum of N samples falls as N rises, changing it moves the recorded level of every benchmark noisy enough to be affected — a broad step across the suite at one commit is the signature of the count changing, not of the code changing. Each run records the count it was reduced from, so which protocol produced a value stays recoverable from the stored data. See collect --best-of.

Ordering the timeline

A series is built per (discriminant set, benchmark identity, metric) and ordered by git first-parent topology, not by wall-clock time. Every run does carry a single wall-clock observation timestamp, but it is provenance only and never orders anything — a benchmarked commit’s position on the timeline is its committer date, read from the git graph at analyze time, so a rebase or amended date can never leave a stale timestamp behind.

flowchart LR
  RR[BenchmarkResult] --> RS[Run]
  CTX[RunContext: git + env + toolchain + machine + best-of] --> RS
  RS -->|collect stores| ST[(Storage)]
  ST -->|analyze reconstructs by topology| SE[Series] --> F[Findings]

Benchmark identity

A benchmark identity is an ordered, non-empty list of string segments; each engine adapter decides the segments, and reports render the full form with segments joined by /. Renaming a benchmark starts a new series. Because identity is per discriminant set, the same benchmark can exist under several triples or machine keys at once.

Blessing scopes

A blessing records acceptance of a benchmark change and sets a new analysis baseline at a chosen commit. Its blessing scope selects which discriminant sets receive that acceptance, using engine, target-triple and machine-key restrictions. An unrestricted field matches every value, including values encountered later. Benchmark-id prefixes select the affected benchmark identities separately from this scope.

A blessing can apply to several partitions without combining their measurements. Each matching series is re-baselined independently, even when its partition has no measurement at the chosen commit. See bless / unbless for the command syntax and defaults.