Reporting

Everything so far produced findings and a census. This chapter is how they reach you, and how to read what you get.

Terms used here

Ranking

Findings are ordered by descending relative move, then by analysis method, then by a stable identity tie-break so two runs over the same data produce the same order.

There is deliberately no severity classification. Whether a 4% regression in a hot path matters more than a 40% regression in a rarely-used one is a judgment about your project, and the tool does not have the information to make it. It sorts by the one quantity it can measure, and leaves the ranking of importance to you.

Note in particular that history findings carry no confidence score to sort by. Every reported history finding already cleared its test, so any such number would be uniformly high and would rank almost nothing — see Detection. Branch findings report an observed range plus report-wide historical context instead of a probability score.

Reading a finding

A finding, as the text report prints it:

http_parse/case
  +29.57% wall_time
    regression via change point · 100.4 → 130.1 · somewhere near commit10
 135 ┤           ╭─╮╭─╮╭╮           
 126 ┤         ╭─╯ ╰╯ ╰╯╰─          
 116 ┤         │                    
 106 ┤ ╭╮ ╭─╮╭─╯                    
  97 ┼─╯╰─╯ ╰╯

The chart is drawn against topology, not against the observations. One column per commit, so a gap is a gap and the trailing stretch after the last observation is visible as one. That matters: a chart that packed the observations together would hide exactly the sparseness that detection cannot see.

Where a series has more commits than the chart has columns, commits are grouped before plotting. Grouping first is deliberate — the alternative attenuates an isolated observation surrounded by gaps into nothing.

Analysis outcomes

The analysis outcome is the result of a successful analysis, combining findings with the series coverage state. Findings take precedence: an analysis with any findings has outcome findings, even when some series could not be judged.

Without findings, a fully judged in-scope suite is clean. An in-scope suite with no judged series is insufficient_baseline; a suite with some judged and some unjudged series is partial. When no series entered analysis, or every accounted series was absent at the analyzed context commit or ignored by configuration, the outcome is nothing_in_scope.

These outcomes describe the series available to analysis, not an inventory of collection jobs. Platform coverage is separate: matrix automation must verify which expected platforms completed collection before presenting a complete all-clear. A missing collection leg can coexist with a clean analysis of the available series. Analysis failure is an execution error, not an outcome.

The formats

The tool emits reports and their derived outputs in these forms, each requested independently:

• Text — the default terminal report.
• Markdown (--markdown <path>) — every finding plus complete coverage counts and reasons, for a pull request or an issue.
• JSON (--json <path>) — the complete machine-readable result.
• Condensed summary (--markdown-summary <path>, analyze only) — a short, capped Markdown digest for a size-limited destination such as a pull request comment.
• Outcome (--outcome <path>, analyze only) — the stable analysis outcome for selecting a message without parsing the full JSON report.

Full Markdown carries every finding and ends with a Coverage section containing the complete counts and unjudged-reason breakdown, including ghosts. JSON always carries the complete census under census, which makes it the machine-readable signal: each finding self-describing, plus every unjudged reason. Text omits that breakdown when findings exist. JSON deliberately omits the per-commit chart series — that is presentation, and examine is the way to get the underlying points. The condensed summary is lossy by design — capped, and flattened so the per-set grouping is dropped — so it is the one output you must not automate against. The outcome file repeats only the JSON report’s top-level outcome field; it carries no report details. The full table is under Where output goes.

Here is the same analysis in text and JSON. The JSON excerpt illustrates the shape automation reads, not a complete field catalog. For matching full Markdown and JSON examples, see Reading a silent report.

Text — the default output, printed to standard output.

Analyzed project textproc (history mode)
  commit: 9f2c4a1d3b5e708aab12cd34ef5678901234567a
  runs: 128 (4d17b0c93ea2 → 9f2c4a1d3b5e)  in-scope series judged: 5 of 8  ignored by configuration: 1 series  regressions: 2

criterion/x86_64-unknown-linux-gnu/a1b2c3d4
  runs: 128  regressions: 2
  filter: --engine criterion --target-triple x86_64-unknown-linux-gnu --machine-key a1b2c3d4

http_parse/case
  +29.57% wall_time
    regression via change point · 100.4 → 130.1 · somewhere near commit10
 135 ┤           ╭─╮╭─╮╭╮           
 126 ┤         ╭─╯ ╰╯ ╰╯╰─          
 116 ┤         │                    
 106 ┤ ╭╮ ╭─╮╭─╯                    
  97 ┼─╯╰─╯ ╰╯                     

index_build/case
  +10.40% wall_time
    regression via drift · 98.5 → 108.7 · accumulated commit0 → commit29
 113 ┤                    ╭╮    ╭── 
 109 ┤          ╭╮╭╮╭─╮   ││ ╭╮╭╯   
 105 ┤ ╭╮     ╭─╯╰╯╰╯ ╰─╮╭╯╰─╯╰╯    
 101 ┼╮│╰─────╯         ╰╯          
  97 ┤╰╯

JSON — the same analysis, written by --json.

{
  "project": "textproc",
  "tip_commit": "9f2c4a1d3b5e708aab12cd34ef5678901234567a",
  "tip_dirty": false,
  "mode": "history",
  "outcome": "findings",
  "notable": true,
  "runs": 128,
  "series": 8,
  "regressions": 2,
  "ghosts_excluded": 2,
  "census": {
    "total": 11,
    "in_scope": 8,
    "judged": 5,
    "unjudged": 6,
    "coverage": "partial",
    "reasons": [
      {
        "reason": "ghost",
        "count": 2
      },
      {
        "reason": "ignored",
        "count": 1
      },
      {
        "reason": "too_few_points",
        "count": 3
      }
    ]
  },
  "findings": [
    {
      "engine": "criterion",
      "target_triple": "x86_64-unknown-linux-gnu",
      "machine_key": "a1b2c3d4",
      "segments": [
        "http_parse",
        "case"
      ],
      "kind": "wall_time",
      "method": "change_point",
      "direction": "regression",
      "baseline": 100.36830218253652,
      "latest": 130.05171405014943,
      "relative_delta": 0.29574488381430086,
      "commit": "commit10"
    },
    {
      "engine": "criterion",
      "target_triple": "x86_64-unknown-linux-gnu",
      "machine_key": "a1b2c3d4",
      "segments": [
        "index_build",
        "case"
      ],
      "kind": "wall_time",
      "method": "drift",
      "direction": "regression",
      "baseline": 98.50313347697937,
      "latest": 108.7433855201543,
      "relative_delta": 0.10395864254986495,
      "commit": "commit29",
      "window_start": "commit0"
    }
  ],
  "sets": [
    {
      "engine": "criterion",
      "target_triple": "x86_64-unknown-linux-gnu",
      "machine_key": "a1b2c3d4",
      "runs": 128,
      "series": 8,
      "regressions": 2
    }
  ]
}

Findings never fail the build

The exit code reflects whether the analysis ran. A report full of regressions exits successfully; a report that could not reach its storage backend does not.

Why: a benchmark tool that breaks builds on a measurement gets disabled, and a disabled tool detects nothing. Findings are advisory by design.

What this means for automation: write the JSON report to a file with --json. Its top-level outcome field carries the analysis outcome; the notable boolean is true exactly for findings. Use --outcome <path> when this value is all the caller needs, and retain separate collection evidence for matrix-wide claims. The full JSON remains necessary for counts, reasons and findings.

The coverage line is not decoration

Text and Markdown headers state how many in-scope series were judged when anything was in scope. JSON records those counts in census. This terminal example shows a silent report:

A report with nothing to say still says how far that silence reaches:

Analyzed project textproc (history mode)
  commit: 9f2c4a1d3b5e708aab12cd34ef5678901234567a
  runs: 128 (4d17b0c93ea2 → 9f2c4a1d3b5e)  in-scope series judged: 3 of 6  regressions: 0
No notable changes detected among the series that were judged.
  Judged 3 of 6 in-scope series; no reportable move survived the gates.
  Not judged: 2 series not measured at the analyzed context commit; 1 series ignored by configuration; 3 series with too few points in the analyzed window.

The judged ratio heads each human-readable report that had anything in scope. Text prints the per-reason breakdown only when there are no findings, as here. Full Markdown always includes it in the final Coverage section, and JSON always carries it under census.reasons.

This is given the same prominence as the findings on purpose. “No notable changes” means nothing changed among the series this run was able to judge, and without the denominator you cannot tell that statement apart from “the benchmarks did not run”.

The final Coverage section in full Markdown and the JSON census always contain the reasons series went unjudged, including when findings exist. Text and condensed summaries list the full reason breakdown only when there are no findings. Configured exclusions also receive a compact disclosure beside findings, so an intentional reduction in analysis scope stays visible. When in-scope series remain unjudged, the condensed summary’s coverage bullet points to the full report for details, without a separate GitHub warning. See Reading a silent report for matching Markdown and JSON examples.

Comparison-base lag

Branch mode compares the analyzed context commit against the base ref’s recent commits within the same discriminant set. On a rotating CI pool, the newest base-ref commits may only carry data under a different machine key — so your runner compares against base data from several commits back.

The empty stretch after the comparison base is the lag: commits that exist on the base ref, but not for the context run’s machine key.

The report says so, naming how far behind and which of two reasons applies:

The lag is advisory. It never changes which findings are reported and never affects the exit code; what it changes is how much weight a marginal branch finding deserves, because a comparison against a base state several commits old is exactly that.

It is advisory. It never changes which findings are reported, and never affects the exit code. What it changes is how much weight you should give a marginal branch finding: a comparison against a base state ten commits old is a comparison against a base state ten commits old.

Branch range and historical context

A branch finding does not borrow history-mode change-point wording. It states the measured context value, the lowest and highest values observed in the selected current-base regime, and the excess beyond the nearest edge. “Slower than all 20 current-base observations” is a claim about those recorded observations, not a probability estimate.

The discriminant-set section then gives the report-wide historical comparison, for example:

None of 10 comparable base commits showed as much out-of-range movement as this branch
(12 series compared).

When too little shared history exists, the report says the comparison could not be formed and still prints every factual range excursion. JSON exposes the same distinction: finding-level branch range metadata and set-level branch_comparison counts.

Where output goes

The text report goes to stdout. Diagnostics, the effective-selection summary and verbose notes go to stderr.

Warnings are the exception, and it is deliberate: a comparison-base lag notice, or the note that a dirty run was admitted on the base tip, is rendered into the report body rather than onto stderr. Those qualify the findings printed beside them, so a reader who captured only the report would otherwise lose the caveat that changes how to read it.

The other formats are written to files rather than to a stream — --markdown, --json, --markdown-summary and --outcome each take a path, and --no-text suppresses the stdout report when you want only those.

JSON is the complete machine-readable result: it always carries every finding and the full per-reason census, including the ghost count. It is not an observation archive — the per-commit series behind the charts is obtained with examine. Full Markdown always includes the same coverage counts and reasons in its final Coverage section. Text omits the full reason breakdown when findings exist. Configured exclusions remain visible beside findings on every surface. The condensed summary is lossy by design, so it is the one output that must not be automated against: a check reading it cannot distinguish findings that were capped away from findings that were never made. The outcome file repeats only JSON’s top-level outcome field; it carries no report details.

What comes next

Nothing — this is the end of the pipeline. If a report left you with a question rather than an answer, Insights is the triage chapter.