BenchX comparison document¶

urn:benchx:schema:comparison-document:0.1.0 · version 0.1.0 · schema.json · source

The comparator's output: verdicts for target results against their references, in history mode or run mode, with the evidence behind each. See docs/design/comparator.md section 5. Derived and recomputed on demand; the store keeps its inputs, not the document.

Implementer note

Rules the schema cannot express and the comparator enforces: inputs lists every result the document refers to; local_only is true when any input is not clean in subject_dirty or benchmark_dirty (benchmark-result-schema.md section 5.5); a unit affected by a failed invariant has no verdict and appears in no units entry; in history mode every reference result precedes the target under the recorded ordering.

Properties¶

Property	Required	Type	Description
`schema_version`	yes	constant `"benchx/comparison-document/0.1.0"`	Message contract version this object conforms to.
`mode`	yes	one of `"history"`, `"run"`	History mode compares a target series point against preceding points of its series; run mode compares two sides measured in one run, as a profile defines them.
`method`	yes	method
`estimator`	yes	estimator	Estimator every compared value was computed with.
`identity_schema`	yes	token	Versioned name of the projection under which series membership and comparability were decided: for a profile, its own name and its identity policy's (benchmark-result-schema.md section 4.3).
`local_only`	yes	boolean	True when any input is not clean; such a comparison never promotes into tracked history.
`run_key`		token	Run mode: the run both sides were measured in.
`profile`		one of `"single-run"`, `"variants"`, `"revisions"`, `"environments"`, `"cross-machine"`	Run mode: the comparison profile the run was checked against (benchmark-result-schema.md section 5.5).
`varying`		token	Run mode: path of the coordinate whose value distinguishes the sides, such as provenance.subject_tree or coordinates.workload.parameters.role.
`baseline`		side	Run mode: the reference side, as its value of the varying coordinate.
`contender`		side	Run mode: the target side. Absent when a failed invariant prevented identifying it.
`sides`		object	Run mode: what each side declared about how it was built and entered, recorded as given and never judged, beside the environment the runner observed for it. See `sides`.
`pairing`		constant `"procedure.round"`	Run mode: how the two sides' attempts are paired. See `pairing`.
`failed_invariants`		array of failedInvariant	Run mode: profile invariants the run did not satisfy.
`units`	yes	array of any	One evaluation each: a series point in history mode, a workload variant and quantity in run mode.
`missing`	yes	array of missing	Expected targets with no result, such as a side absent from a run or a series with no result from a batch request's target.
`exclusions`	yes	array of exclusion	Results left out of the evidence, with the reason.
`inputs`	yes	array of resultRef (unique items)	Every result the comparison consumed.

No other properties are allowed.

Rules:

If mode is "run", then requires run_key, profile, varying, baseline, pairing, failed_invariants; units must be array of runUnit; else forbids run_key, profile, varying, baseline, contender, sides, pairing, failed_invariants; units must be array of historyUnit.

`sides`¶

Run mode: what each side declared about how it was built and entered, recorded as given and never judged, beside the environment the runner observed for it.

Property	Required	Type	Description
`baseline`		declared
`contender`		declared

No other properties are allowed.

`pairing`¶

Run mode: how the two sides' attempts are paired.

Implementer note

Only paired comparison is defined; comparator.md open question 2 asks whether an unpaired method should be allowed.

Type: constant "procedure.round"

Definitions¶

Reusable subschemas referenced from the tables above and from other schemas.

`method`¶

Comparison method: a stable name and version, the effect measure it reports, and its complete configuration, such as k and minimum evidence.

Property	Required	Type	Description
`name`	yes	token
`version`	yes	token
`effect_measure`	yes	token	How effect is measured from the reference estimate, such as relative, absolute, ratio, or log-ratio.
`configuration`	yes	object

No other properties are allowed.

`outcome`¶

What one evaluation found. Shared by both unit kinds.

Property	Required	Type	Description
`workload`	yes	workload
`quantity`	yes	token
`reference_estimate`		number	Expected value from the reference points or the baseline side.
`effect`		number	Change from the reference estimate, in the method's effect measure.
`noise`		number (>= 0)	Variation on the effect's scale. Absent for a deterministic quantity.
`noise_source`		token	Where the noise came from, such as reference-points or paired-rounds.
`provisional`		boolean	True when the noise came from a provisional rule rather than evidence.
`threshold`		number (>= 0)	The effect magnitude the method required to indicate a change.
`verdict`	yes	one of `"improved"`, `"regressed"`, `"changed"`, `"no-change-detected"`, `"indeterminate"`
`reason`		token	Why the verdict is indeterminate, such as insufficient-history or duplicate-points.
`unmatched_differences`		array of token (unique items)	Coordinates the projection does not match, by JSON Pointer, whose values differ between target and reference.
`caveats`		array of nonEmptyString	Quality caveats the reader should see.

Rules:

If verdict is "indeterminate", then requires reason; else requires reference_estimate, effect.

`runUnit`¶

One workload variant and quantity, its two sides paired by round.

Property	Required	Type	Description
`rounds`	yes	integer (>= 0)	Paired rounds used as evidence.
`series`		object	The series each side's results belong to, when the store assigns them one. Run mode reads no history from them. See `runUnit.series`.

Rules:

Must also satisfy outcome.

`runUnit.series`¶

The series each side's results belong to, when the store assigns them one. Run mode reads no history from them.

Property	Required	Type	Description
`baseline`	yes	sha256
`contender`	yes	sha256

No other properties are allowed.

`historyUnit`¶

One target series point against preceding points of its series.

Property	Required	Type	Description
`series`	yes	sha256	Series fingerprint of the target and every reference point.
`target`	yes	resultRef
`reference`	yes	array of resultRef	The reference points the reference estimate was formed from, oldest first.
`ordering`		one of `"ancestry"`, `"native-order"`, `"native-time"`	How the reference points were ordered before the target. Absent when no order was available.
`boundaries`		array of boundary	Resets, change points, and continuity joins that truncated or joined the reference.

Rules:

Must also satisfy outcome.

`boundary`¶

Property	Required	Type	Description
`kind`	yes	one of `"distribution-reset"`, `"change-point"`, `"continuity"`
`at`		resultRef	First result after the boundary.
`detail`		nonEmptyString

No other properties are allowed.

`side`¶

A value of the varying coordinate, such as a tree id, a label, or a parameter value.

Type: not null

Must not be null.

`declared`¶

One side's declared text, copied from its results' provenance.info, and the allowlisted environment variables observed for it.

Property	Type	Description
`how_built`	nonEmptyString
`activation`	nonEmptyString
`shell`	token
`target_provider`	object	See `declared.target_provider`.
`observed_env`	map of string

No other properties are allowed.

`declared.target_provider`¶

Property	Required	Type	Description
`name`	yes	nonEmptyString
`version`		nonEmptyString

No other properties are allowed.

`failedInvariant`¶

Property	Required	Type	Description
`invariant`	yes	token	Name of the invariant, such as alternating-slots or equal-attempts.
`detail`	yes	nonEmptyString
`workload`		workload	The unit it affects, when it affects only one.
`quantity`		token

No other properties are allowed.

`missing`¶

Property	Required	Type	Description
`workload`	yes	workload
`quantity`	yes	token
`side`		one of `"baseline"`, `"contender"`	Run mode: the side with no result.
`series`		sha256	History mode: the expected series.

No other properties are allowed.

`exclusion`¶

Property	Required	Type	Description
`result`	yes	resultRef
`reason`	yes	nonEmptyString

No other properties are allowed.

`workload`¶

Workload variant: a named operation plus one parameter assignment.

Property	Required	Type	Description
`name`	yes	token
`parameters`		map of parameterValue

No other properties are allowed.

`estimator`¶

Estimator declaration, as in the measurement-result schema.

Property	Required	Type
`name`	yes	token
`method_version`	yes	token
`input_level`	yes	one of `"observations"`, `"estimates"`
`parameters`		object
`inputs`		array of object (min items 1)

No other properties are allowed.

`resultRef`¶

Stable reference to a stored result.

Property	Required	Type	Description
`producer`	yes	token
`ingest_key`	yes	token

No other properties are allowed.

`parameterValue`¶

Type: exactly one of: primitive; array of primitive

`primitive`¶

Type: string or number or boolean or null

`sha256`¶

Type: string (pattern ^[0-9a-f]{64}$)

`token`¶

Non-empty string without whitespace.

Type: string (pattern ^\S+$; min length 1)

`nonEmptyString`¶

Type: string (min length 1)

Examples¶

historyrun

{
  "schema_version": "benchx/comparison-document/0.1.0",
  "mode": "history",
  "method": {
    "name": "benchx/dispersion-scaled",
    "version": "v1",
    "effect_measure": "relative",
    "configuration": {
      "k": 3.0,
      "window": 20,
      "min_points": 4
    }
  },
  "estimator": {
    "name": "min",
    "method_version": "benchx/min/v1",
    "input_level": "observations"
  },
  "identity_schema": "arrow/identity/v2",
  "local_only": false,
  "units": [
    {
      "workload": {
        "name": "parquet-read",
        "parameters": {
          "compression": "snappy"
        }
      },
      "quantity": "wall-time",
      "series": "5b7d9f1a3c5e7b9d1f3a5c7e9b1d3f5a7c9e1b3d5f7a9c1e3b5d7f9a1c3e5b7d",
      "target": {
        "producer": "example.org/arrow-adapter",
        "ingest_key": "ci-1240:parquet-read/snappy:wall-time"
      },
      "reference": [
        {
          "producer": "example.org/arrow-adapter",
          "ingest_key": "ci-1236:parquet-read/snappy:wall-time"
        },
        {
          "producer": "example.org/arrow-adapter",
          "ingest_key": "ci-1237:parquet-read/snappy:wall-time"
        },
        {
          "producer": "example.org/arrow-adapter",
          "ingest_key": "ci-1238:parquet-read/snappy:wall-time"
        },
        {
          "producer": "example.org/arrow-adapter",
          "ingest_key": "ci-1239:parquet-read/snappy:wall-time"
        }
      ],
      "ordering": "ancestry",
      "boundaries": [
        {
          "kind": "distribution-reset",
          "at": {
            "producer": "example.org/arrow-adapter",
            "ingest_key": "ci-1236:parquet-read/snappy:wall-time"
          },
          "detail": "runner kernel upgraded"
        }
      ],
      "reference_estimate": 0.0102,
      "effect": -0.008,
      "noise": 0.004,
      "noise_source": "reference-points",
      "threshold": 0.012,
      "verdict": "no-change-detected",
      "unmatched_differences": [
        "/coordinates/environment/system/os/kernel"
      ]
    }
  ],
  "missing": [],
  "exclusions": [
    {
      "result": {
        "producer": "example.org/arrow-adapter",
        "ingest_key": "ci-1235:parquet-read/snappy:wall-time"
      },
      "reason": "status error"
    }
  ],
  "inputs": [
    {
      "producer": "example.org/arrow-adapter",
      "ingest_key": "ci-1235:parquet-read/snappy:wall-time"
    },
    {
      "producer": "example.org/arrow-adapter",
      "ingest_key": "ci-1236:parquet-read/snappy:wall-time"
    },
    {
      "producer": "example.org/arrow-adapter",
      "ingest_key": "ci-1237:parquet-read/snappy:wall-time"
    },
    {
      "producer": "example.org/arrow-adapter",
      "ingest_key": "ci-1238:parquet-read/snappy:wall-time"
    },
    {
      "producer": "example.org/arrow-adapter",
      "ingest_key": "ci-1239:parquet-read/snappy:wall-time"
    },
    {
      "producer": "example.org/arrow-adapter",
      "ingest_key": "ci-1240:parquet-read/snappy:wall-time"
    }
  ]
}

{
  "schema_version": "benchx/comparison-document/0.1.0",
  "mode": "run",
  "method": {
    "name": "benchx/paired-relative",
    "version": "v1",
    "effect_measure": "relative",
    "configuration": {
      "k": 3.0,
      "min_rounds": 3
    }
  },
  "estimator": {
    "name": "median",
    "method_version": "benchx/median/v1",
    "input_level": "observations"
  },
  "identity_schema": "benchx/profile/environments/v1+benchx/benchx-identity/v1",
  "local_only": true,
  "run_key": "urn:uuid:9b2e4f1a-6c3d-4e8b-a1f0-2d7c5b9e3a41",
  "profile": "environments",
  "varying": "provenance.labels.build",
  "baseline": "plain",
  "contender": "hardened",
  "pairing": "procedure.round",
  "failed_invariants": [],
  "units": [
    {
      "workload": {
        "name": "TakeChunked/4194304/2"
      },
      "quantity": "wall-time",
      "series": {
        "baseline": "3f1c9a0b7d2e4f6a8b1c3d5e7f9a0b2c4d6e8f0a1b3c5d7e9f0a2b4c6d8e0f1a",
        "contender": "8e2d4c6b0a9f1e3d5c7b9a0f2e4d6c8b0a1f3e5d7c9b0a2f4e6d8c0b1a3f5e7d"
      },
      "rounds": 3,
      "reference_estimate": 0.0455,
      "effect": 0.024,
      "noise": 0.0021,
      "noise_source": "paired-rounds",
      "threshold": 0.0063,
      "verdict": "regressed",
      "unmatched_differences": []
    },
    {
      "workload": {
        "name": "TakeChunked/65536/2"
      },
      "quantity": "wall-time",
      "rounds": 2,
      "noise_source": "paired-rounds",
      "verdict": "indeterminate",
      "reason": "insufficient-rounds"
    }
  ],
  "missing": [
    {
      "workload": {
        "name": "TakeChunked/1024/2"
      },
      "quantity": "wall-time",
      "side": "contender"
    }
  ],
  "exclusions": [],
  "inputs": [
    {
      "producer": "benchx/gbench-adapter",
      "ingest_key": "run-9b2e:TakeChunked/4194304/2:slot-1:wall-time"
    },
    {
      "producer": "benchx/gbench-adapter",
      "ingest_key": "run-9b2e:TakeChunked/4194304/2:slot-2:wall-time"
    },
    {
      "producer": "benchx/gbench-adapter",
      "ingest_key": "run-9b2e:TakeChunked/4194304/2:slot-3:wall-time"
    },
    {
      "producer": "benchx/gbench-adapter",
      "ingest_key": "run-9b2e:TakeChunked/4194304/2:slot-4:wall-time"
    },
    {
      "producer": "benchx/gbench-adapter",
      "ingest_key": "run-9b2e:TakeChunked/4194304/2:slot-5:wall-time"
    },
    {
      "producer": "benchx/gbench-adapter",
      "ingest_key": "run-9b2e:TakeChunked/4194304/2:slot-6:wall-time"
    },
    {
      "producer": "benchx/gbench-adapter",
      "ingest_key": "run-9b2e:TakeChunked/65536/2:slot-7:wall-time"
    },
    {
      "producer": "benchx/gbench-adapter",
      "ingest_key": "run-9b2e:TakeChunked/65536/2:slot-8:wall-time"
    },
    {
      "producer": "benchx/gbench-adapter",
      "ingest_key": "run-9b2e:TakeChunked/65536/2:slot-9:wall-time"
    },
    {
      "producer": "benchx/gbench-adapter",
      "ingest_key": "run-9b2e:TakeChunked/65536/2:slot-10:wall-time"
    },
    {
      "producer": "benchx/gbench-adapter",
      "ingest_key": "run-9b2e:TakeChunked/1024/2:slot-11:wall-time"
    },
    {
      "producer": "benchx/gbench-adapter",
      "ingest_key": "run-9b2e:TakeChunked/1024/2:slot-12:wall-time"
    }
  ]
}

BenchX comparison document¶

Properties¶

sides¶

pairing¶

Definitions¶

method¶

outcome¶

runUnit¶

runUnit.series¶

historyUnit¶

boundary¶

side¶

declared¶

declared.target_provider¶

failedInvariant¶

missing¶

exclusion¶

workload¶

estimator¶

resultRef¶

parameterValue¶

primitive¶

sha256¶

token¶

nonEmptyString¶

Examples¶

`sides`¶

`pairing`¶

`method`¶

`outcome`¶

`runUnit`¶

`runUnit.series`¶

`historyUnit`¶

`boundary`¶

`side`¶

`declared`¶

`declared.target_provider`¶

`failedInvariant`¶

`missing`¶

`exclusion`¶

`workload`¶

`estimator`¶

`resultRef`¶

`parameterValue`¶

`primitive`¶

`sha256`¶

`token`¶

`nonEmptyString`¶