BenchX measurement result ingest object¶

urn:benchx:schema:measurement-result:0.1.0 · version 0.1.0 · schema.json · source

One measurement attempt for one quantity and one workload variant, as reported by a producer. See docs/design/benchmark-result-schema.md section 5. Producers report coordinates and evidence; comparison identity, series, and series points are derived by the server and are never part of this message. Conditions such as temperature or load are sibling results sharing attempt_key, not coordinates; whether a coordinate splits history is decided by a projection, by default the project's identity policy (schemas/identity-policy/0.1.0).

Implementer note

Rules the schema cannot express and the server enforces at ingest: the primary subject component's source, when present, equals the top-level source URI; a quantity's unit never changes once used in a project; at most one estimate summary per estimator declaration; results sharing an attempt_key agree on the fields listed in section 5.3; identical (producer, ingest_key) payloads are idempotent and differing ones conflict; UCUM unit syntax; interval lower bounds not exceeding upper bounds; completed_repetitions never exceeding attempted_repetitions, and likewise completed never exceeding attempted at each repetition level; exactly one provenance.references entry per compound input role; comparison profiles (section 5.5) are checked by the comparator, not here. A result without project or benchmark is thin (section 4.1): valid, comparable within its run, and joinable to tracked series only through an approved mapping.

Properties¶

Property	Required	Type	Description
`schema_version`	yes	constant `"benchx/measurement-result/0.1.0"`	Message contract version this object conforms to.
`producer`	yes	object	Stable, namespaced identity of the adapter or harness that produced this message, plus its software and mapping versions. Only the name is part of the idempotency key. See `producer`.
`ingest_key`	yes	token	Producer-scoped idempotency key. Unique per result under one producer name; must distinguish workload variant, quantity, and attempt.
`attempt_key`	yes	token	Globally namespace-qualified key, such as a URI-like key or UUID, shared by every result from one concrete execution of one workload variant. A retry gets a new key.
`project`		token	Namespace for project-defined names and policy. Optional only for thin local results.
`source`	yes	sourceRef	Designated subject source whose revisions form the history axis.
`revision`	yes	revisionRef	Subject revision measured, keyed by the clean VCS identifier. Dirty state is a provenance flag.
`benchmark`		object	Exact code that defined and executed the workload. A reported coordinate that policy normally maps to a declared benchmark version. Optional only for thin local results with no benchmark repository. The revision may be omitted when the benchmark code has no version control; provenance.benchmark_tree then identifies it and benchmark_dirty is dirty. See `benchmark`.
`coordinates`	yes	object	Reported coordinates: what was measured and under which conditions. Fingerprinted as reported; identity policy projects them into comparison identity. See `coordinates`.
`measurement`	yes	object	The measured quantity, its outcome, and producer evidence. Like the estimator, the quantity says what the number is, not what produced it, so it is never a coordinate. See `measurement`.
`procedure`		object	Realized batch-wide acquisition details. Per-observation metadata belongs on structured observations instead. See `procedure`.
`provenance`	yes	object	Audit metadata. Nothing here enters comparison identity. See `provenance`.
`quality`		object	Immutable producer-side validation and warnings. Store-side quarantine and exclusion are separate state. See `quality`.

No other properties are allowed.

Rules:

If measurement.derivation is present or measurement.summaries contains an item where estimator.inputs is present, then requires provenance.references.
If benchmark.revision is absent, then requires provenance.benchmark_tree; provenance.benchmark_dirty must be constant "dirty".

`producer`¶

Stable, namespaced identity of the adapter or harness that produced this message, plus its software and mapping versions. Only the name is part of the idempotency key.

Property	Required	Type	Description
`name`	yes	token
`version`	yes	nonEmptyString
`mapping_version`	yes	nonEmptyString	Version of the native-to-BenchX mapping applied by the producer.

No other properties are allowed.

`benchmark`¶

Exact code that defined and executed the workload. A reported coordinate that policy normally maps to a declared benchmark version. Optional only for thin local results with no benchmark repository. The revision may be omitted when the benchmark code has no version control; provenance.benchmark_tree then identifies it and benchmark_dirty is dirty.

Property	Required	Type	Description
`source`	yes	sourceRef
`revision`		revisionRef

No other properties are allowed.

`coordinates`¶

Reported coordinates: what was measured and under which conditions. Fingerprinted as reported; identity policy projects them into comparison identity.

Property	Required	Type	Description
`workload`	yes	object	Workload variant: a named operation plus one resolved parameter assignment. See `coordinates.workload`.
`subject`	yes	object	Implementation or system under test. Exactly one component has role 'primary' and is the designated source; other components are pinned coordinates. See `coordinates.subject`.
`protocol`	yes	object	How the subject is exercised and measured: protocol version, warmup and calibration policy, timer.
`environment`	yes	object	Host or allocation shared by every result of the attempt. See `coordinates.environment`.
`resource_selection`		object (min properties 1)	Optional resource this result applies to within the environment: CPU/core set, GPU UUID or device index, accelerator partition.

No other properties are allowed.

`coordinates.workload`¶

Workload variant: a named operation plus one resolved parameter assignment.

Property	Required	Type	Description
`name`	yes	token
`parameters`		map of parameterValue	One concrete parameter assignment. Values are primitives or arrays of primitives. A baseline-versus-variant study uses a `role` parameter.
`dataset`		object	Identity of the input data. Part of the workload variant, so runs on different data never share a variant. See `coordinates.workload.dataset`.

No other properties are allowed.

`coordinates.workload.dataset`¶

Identity of the input data. Part of the workload variant, so runs on different data never share a variant.

Property	Required	Type	Description
`name`	yes	token
`version`		nonEmptyString
`sha256`		sha256
`parameters`		map of parameterValue	Generation parameters such as row count or seed.

No other properties are allowed.

`coordinates.subject`¶

Implementation or system under test. Exactly one component has role 'primary' and is the designated source; other components are pinned coordinates.

Property	Required	Type	Description
`name`	yes	token
`components`	yes	array of subjectComponent (min items 1; contains an item that is object)
`configuration`		object	How the artifact under test was built or configured: build type, compiler, flags, JIT/AOT mode, backend.

No other properties are allowed.

`coordinates.environment`¶

Host or allocation shared by every result of the attempt.

Property	Required	Type	Description
`schema_version`	yes	token	Identity schema that defines which identity fields exist and how they are hashed.
`identity`	yes	object (min properties 1)	Fields that define environment identity under the named schema.
`system`		object	Facts about the system the attempt ran on that do not name the host: kernel, C library, drivers, image digest, runtime, the allowlisted process environment. null means read but undeterminable.

No other properties are allowed.

`measurement`¶

The measured quantity, its outcome, and producer evidence. Like the estimator, the quantity says what the number is, not what produced it, so it is never a coordinate.

Property	Required	Type	Description
`quantity`	yes	object	Measured quantity and its canonical UCUM unit. Distinct measurement semantics require distinct names. See `measurement.quantity`.
`kind`		one of `"measurand"`, `"condition"`	measurand: what the attempt set out to measure. condition: a state the system passed through during the attempt, recorded beside its measurands (an influence quantity, in metrology terms; section 4.4). A property of this result, not of the quantity: load average is a condition beside a timing benchmark and a measurand in a load study.
`status`	yes	one of `"success"`, `"partial"`, `"censored"`, `"error"`, `"skipped"`
`reason`		string (pattern `^[a-z0-9]+([.-][a-z0-9]+)*$`)	Machine-readable lowercase token such as 'timeout' or 'resource-exhausted.memory'. Required for every status except success.
`constraint`		constraint
`observations`		array of exactly one of: number; structuredObservation (min items 1)	Observation batch: finite values in the quantity's canonical unit after declared normalization. Numbers, or structured observations carrying per-observation metadata.
`derivation`		all of: estimator; object (requires `inputs`)	For a compound result with a pairwise batch: the per-pair rule that produced the observations, declared as a compound estimator at input_level observations (section 4.5).
`summaries`		array of summary (min items 1)	Typed producer summaries. Server-derived values never appear here.

No other properties are allowed.

Rules:

If status is "success", then forbids constraint; at least one of: requires observations; requires summaries; summaries contains an item that is estimateSummary.
If status is "partial", then requires reason; forbids constraint; at least one of: requires observations; requires summaries; summaries contains an item that is estimateSummary.
If status is "censored", then requires reason, constraint; forbids observations, summaries.
If status is one of "error", "skipped", then requires reason; forbids observations, summaries, constraint.

`measurement.quantity`¶

Measured quantity and its canonical UCUM unit. Distinct measurement semantics require distinct names.

Property	Required	Type
`name`	yes	token
`unit`	yes	nonEmptyString
`direction`		one of `"lower-is-better"`, `"higher-is-better"`
`deterministic`		boolean

No other properties are allowed.

`procedure`¶

Realized batch-wide acquisition details. Per-observation metadata belongs on structured observations instead.

Property	Type	Description
`inner_iterations`	integer (>= 1)
`attempted_repetitions`	integer (>= 1)
`completed_repetitions`	integer (>= 0)
`repetition_levels`	array of object (min items 1)	Realized counterpart of a multi-level protocol.repetitions, outermost level first. Counts are totals over the attempt, so 20 processes of 3 values each attempt 20 processes and 60 values. attempted_repetitions and completed_repetitions remain the totals at the innermost level.
`warmups_performed`	integer (>= 0)
`duration_seconds`	number (>= 0)
`caches_cleared`	array of token	Caches actually dropped during the attempt, in order.
`slot`	integer (>= 0)	Position of this attempt in the run's realized order across all sides, zero-based. The intended schedule is protocol.schedule.
`timeout_seconds`	number (> 0)	Limit the run author imposed on the attempt, recorded whether or not it fired. An attempt stopped by it is error with reason timeout, not censored.
`round`	integer (>= 0)	Position of this attempt in an interleaved run, monotone per run and kept by a retry. Attempts with the same round number across sides of a comparison are paired.

`provenance`¶

Audit metadata. Nothing here enters comparison identity.

Implementer note

A tree id is required whenever the matching dirty flag is clean or dirty, and absent when it is unknown.

Property	Required	Type	Description
`run_key`		token	Grouping key assigned by the run author: scheduler, CLI, or CI system.
`labels`		map of nonEmptyString	Caller-supplied labels for selecting sides of a comparison, such as {"env": "numpy-2.0"}. Never enter identity.
`native_id`		nonEmptyString	Identifier of this result in the producer's native system.
`started_at`	yes	timestamp	Measurement start on the producer's clock. The only fact that orders attempts across runs; ingest time is never used.
`ended_at`		timestamp
`subject_dirty`	yes	dirtyState
`subject_tree`		treeId	Content hash of the measured subject checkout: the git tree object id of the working tree, or a native equivalent. Equal ids mean identical code regardless of commit.
`benchmark_dirty`	yes	dirtyState	unknown when no benchmark repository exists or its state could not be determined.
`benchmark_tree`		treeId	Content hash of the measured benchmark checkout, as for subject_tree.
`runner`		object	Audit-only identity of the runner software. See `provenance.runner`.
`original_unit`		nonEmptyString	Unit used by the producer before normalization to the canonical unit.
`source_payload_uri`		uri
`source_payload_sha256`		sha256
`ci_uri`		uri
`references`		array of object (min items 1)	Stable references to other results in the same run: compound-estimator inputs by role, or an anchor measurement. Never enter identity.
`artifacts`		array of artifact	External evidence such as large observation sets, profiles, or histograms.
`info`		object	Open descriptive metadata: logs, notes, corrections.

No other properties are allowed.

Rules:

If subject_dirty is one of "clean", "dirty", then requires subject_tree; else forbids subject_tree.
If benchmark_dirty is one of "clean", "dirty", then requires benchmark_tree; else forbids benchmark_tree.

`provenance.runner`¶

Audit-only identity of the runner software.

Property	Required	Type	Description
`name`	yes	nonEmptyString
`version`		nonEmptyString

No other properties are allowed.

`quality`¶

Immutable producer-side validation and warnings. Store-side quarantine and exclusion are separate state.

Property	Required	Type	Description
`warnings`		array of token
`validation`		object	Producer-observed validation facts. The verdict that acts on them is decision policy. See `quality.validation`.

No other properties are allowed.

`quality.validation`¶

Producer-observed validation facts. The verdict that acts on them is decision policy.

Property	Required	Type	Description
`correctness`		one of `"pass"`, `"fail"`	Outcome of a harness-run correctness check against expected output.

Definitions¶

Reusable subschemas referenced from the tables above and from other schemas.

`nonEmptyString`¶

Type: string (min length 1)

`token`¶

Non-empty string without whitespace.

Type: string (pattern ^\S+$; min length 1)

`uri`¶

Type: string (format uri; min length 1)

`sha256`¶

Type: string (pattern ^[0-9a-f]{64}$)

`treeId`¶

Lowercase hex content hash of a working tree: 40 characters for git SHA-1 repositories, 64 for SHA-256 ones or other hash schemes.

Type: string (pattern ^([0-9a-f]{40}|[0-9a-f]{64})$)

`dirtyState`¶

Working-tree state of the measured checkout. unknown is distinct from clean so comparators can reject anything not known to be clean.

Type: one of "clean", "dirty", "unknown"

`timestamp`¶

RFC 3339 UTC timestamp.

Implementer note

format is an annotation in draft 2020-12; the pattern is the assertion.

Type: string (format date-time; pattern ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]+)?Z$)

`finiteNumber`¶

Type: number

`primitive`¶

Type: string or number or boolean or null

`parameterValue`¶

Type: exactly one of: primitive; array of primitive

`sourceRef`¶

Property	Required	Type	Description
`uri`	yes	uri
`type`		token	Source kind, such as git, hg, or dataset.

No other properties are allowed.

`revisionRef`¶

Property	Required	Type	Description
`key`	yes	token	Clean VCS identifier or other native revision key.
`native_order`		integer	Optional order metadata from the native system when the source cannot supply topology.
`native_time`		timestamp

No other properties are allowed.

`resultRef`¶

Stable reference to a stored result.

Property	Required	Type	Description
`producer`	yes	token
`ingest_key`	yes	token

No other properties are allowed.

`subjectComponent`¶

Property	Required	Type
`role`	yes	token
`source`		uri
`revision`		token

No other properties are allowed.

Rules:

If role is "primary", then forbids revision; else requires source, revision.

`estimator`¶

Estimator declaration. Name, method version, parameters, input level, and input roles enter the estimator fingerprint.

Property	Required	Type	Description
`name`	yes	token
`method_version`	yes	token
`input_level`	yes	one of `"observations"`, `"estimates"`
`parameters`		object	Details that can alter the value, such as percentile interpolation.
`inputs`		array of object (min items 1)	Semantic input roles of a compound estimator.

No other properties are allowed.

Rules:

If input_level is "estimates", then requires inputs.

`structuredObservation`¶

Property	Required	Type	Description
`value`	yes	finiteNumber
`ordinal`		integer (>= 0)
`slot`		integer (>= 0)	Position of this observation in the run's realized order across all sides, for observation-level interleaving.
`time`		timestamp
`group`		token
`pair`		token
`included`		boolean
`exclusion_reason`		token

No other properties are allowed.

`constraint`¶

Quantitative constraint for censored results: kind, finite bound(s), inclusivity of each bound present, and cause. Fields present must match the kind.

Property	Required	Type
`kind`	yes	one of `"lower_bound"`, `"upper_bound"`, `"interval"`
`lower`		finiteNumber
`upper`		finiteNumber
`lower_inclusive`		boolean
`upper_inclusive`		boolean
`cause`	yes	token

No other properties are allowed.

Rules:

If kind is "lower_bound", then requires lower, lower_inclusive; forbids upper, upper_inclusive.
If kind is "upper_bound", then requires upper, upper_inclusive; forbids lower, lower_inclusive.
If kind is "interval", then requires lower, upper, lower_inclusive, upper_inclusive.

`summarySource`¶

Every summary in an ingest object is producer evidence.

Type: constant "producer"

`estimateSummary`¶

Property	Required	Type
`type`	yes	constant `"estimate"`
`estimator`	yes	estimator
`value`	yes	finiteNumber
`source`	yes	summarySource

No other properties are allowed.

`statisticSummary`¶

Property	Required	Type
`type`	yes	constant `"statistic"`
`name`	yes	token
`value`	yes	finiteNumber
`method`		nonEmptyString
`source`	yes	summarySource

No other properties are allowed.

`intervalSummary`¶

Confidence or coverage interval for a named statistic at a stated level.

Property	Required	Type
`type`	yes	one of `"confidence_interval"`, `"coverage_interval"`
`statistic`	yes	token
`level`	yes	number (> 0; < 1)
`lower`	yes	finiteNumber
`upper`	yes	finiteNumber
`method`		nonEmptyString
`source`	yes	summarySource

No other properties are allowed.

`sourceBoundsSummary`¶

Opaque bounds whose exact semantics are unknown. The label preserves source terminology such as stat, sys, range, or error.

Property	Required	Type
`type`	yes	constant `"source_bounds"`
`statistic`		token
`label`	yes	nonEmptyString
`lower`		finiteNumber
`upper`		finiteNumber
`source`	yes	summarySource

No other properties are allowed.

Rules:

At least one of: requires lower; requires upper.

`standardUncertaintySummary`¶

Property	Required	Type
`type`	yes	constant `"standard_uncertainty"`
`statistic`	yes	token
`value`	yes	number (>= 0)
`method`		nonEmptyString
`source`	yes	summarySource

No other properties are allowed.

`expandedUncertaintySummary`¶

Property	Required	Type
`type`	yes	constant `"expanded_uncertainty"`
`statistic`	yes	token
`value`	yes	number (>= 0)
`coverage_factor`		number (> 0)
`level`		number (> 0; < 1)
`method`		nonEmptyString
`source`	yes	summarySource

No other properties are allowed.

Rules:

At least one of: requires coverage_factor; requires level.

`summary`¶

Type: exactly one of: estimateSummary; statisticSummary; intervalSummary; sourceBoundsSummary; standardUncertaintySummary; expandedUncertaintySummary

`artifact`¶

Property	Required	Type
`kind`	yes	token
`media_type`	yes	nonEmptyString
`schema`		nonEmptyString
`uri`	yes	uri
`sha256`	yes	sha256

No other properties are allowed.

Examples¶

adhocarrow-temperaturearrowcupy-matmul-cpu-timecupy-matmul-gpu-timenarwhals-overhead-ratio

{
  "schema_version": "benchx/measurement-result/0.1.0",
  "producer": {
    "name": "example.org/benchx-cli",
    "version": "0.1.0",
    "mapping_version": "timeit-to-benchx/v1"
  },
  "ingest_key": "session-7f3a:truncate/role=variant:round-1:wall-time",
  "attempt_key": "urn:uuid:7c9e6679-7425-40de-944b-e07fc1f90ae7",
  "source": {
    "uri": "https://github.com/pandas-dev/pandas",
    "type": "git"
  },
  "revision": {
    "key": "a1b2c3d4e5f60718293a4b5c6d7e8f9012345678"
  },
  "coordinates": {
    "workload": {
      "name": "truncate",
      "parameters": {
        "role": "variant",
        "rows": 1000000
      }
    },
    "subject": {
      "name": "pandas",
      "components": [
        {
          "role": "primary"
        },
        {
          "role": "numpy",
          "source": "https://github.com/numpy/numpy",
          "revision": "v2.0.0"
        }
      ]
    },
    "protocol": {
      "name": "timeit",
      "version": "v1",
      "timer": "perf_counter",
      "warmup": {
        "mode": "repetitions",
        "n_warmup": 1
      },
      "calibration": {
        "mode": "adaptive",
        "minimum_sample_seconds": 0.2
      },
      "repetitions": {
        "mode": "fixed",
        "n_repeat": 5
      },
      "schedule": {
        "unit": "attempt",
        "rule": "alternating",
        "rounds": 5
      }
    },
    "environment": {
      "schema_version": "local/v1",
      "identity": {
        "hostname": "rok-laptop"
      },
      "system": {
        "runtime": {
          "python": "3.12"
        }
      }
    }
  },
  "measurement": {
    "quantity": {
      "name": "wall-time",
      "unit": "s"
    },
    "status": "success",
    "observations": [
      0.000131,
      0.000129,
      0.000133,
      0.00013,
      0.000128
    ]
  },
  "procedure": {
    "inner_iterations": 2000,
    "attempted_repetitions": 5,
    "completed_repetitions": 5,
    "warmups_performed": 1,
    "round": 1,
    "slot": 3
  },
  "provenance": {
    "run_key": "urn:uuid:3f1d2c4b-8a9e-4f60-b1c2-d3e4f5a6b7c8",
    "started_at": "2026-09-14T10:02:31Z",
    "subject_dirty": "dirty",
    "subject_tree": "e7a1c3d5b7f9a1c3e5b7d9f1a3c5e7b9d1f3a5c7",
    "benchmark_dirty": "unknown",
    "labels": {
      "env": "numpy-2.0"
    }
  }
}

{
  "schema_version": "benchx/measurement-result/0.1.0",
  "producer": {
    "name": "example.org/benchmark-adapter",
    "version": "4.0",
    "mapping_version": "native-to-benchx/v5"
  },
  "ingest_key": "ci-1234:parquet-read/snappy:cpu-temperature",
  "attempt_key": "ci.example.org/runs/ci-1234/attempts/parquet-read-snappy-1",
  "project": "arrow",
  "source": {
    "uri": "https://github.com/apache/arrow",
    "type": "git"
  },
  "revision": {
    "key": "02addad7a1f3ce7d6de6a6230ec4c17828cb8f52"
  },
  "benchmark": {
    "source": {
      "uri": "https://github.com/example/arrow-benchmarks",
      "type": "git"
    },
    "revision": {
      "key": "92ef3aa0c1d5e8b7f4a2936d5c1e7b8a9f0d3c2e"
    }
  },
  "coordinates": {
    "workload": {
      "name": "parquet-read",
      "parameters": {
        "compression": "snappy"
      }
    },
    "subject": {
      "name": "apache-arrow",
      "components": [
        {
          "role": "primary"
        }
      ],
      "configuration": {
        "build_type": "release",
        "compiler": "clang-18"
      }
    },
    "protocol": {
      "probe": {
        "name": "hwmon",
        "sensor": "coretemp/package",
        "sampling": {
          "mode": "interval",
          "seconds": 1
        }
      }
    },
    "environment": {
      "schema_version": "machine/v1",
      "identity": {
        "runner": "bench-01",
        "cpu": "AMD EPYC 7R13",
        "cores": 8
      },
      "system": {
        "os": {
          "kernel": "6.8.0-31-generic"
        },
        "libc": {
          "version": "2.39"
        },
        "runtime": {
          "python": "3.12"
        }
      }
    }
  },
  "measurement": {
    "quantity": {
      "name": "cpu-temperature",
      "unit": "Cel"
    },
    "kind": "condition",
    "status": "success",
    "observations": [
      { "value": 61.0, "ordinal": 0, "time": "2026-07-18T09:12:44Z" },
      { "value": 63.5, "ordinal": 1, "time": "2026-07-18T09:12:45Z" },
      { "value": 64.0, "ordinal": 2, "time": "2026-07-18T09:12:46Z" },
      { "value": 64.5, "ordinal": 3, "time": "2026-07-18T09:12:47Z" }
    ]
  },
  "procedure": {
    "duration_seconds": 3.4
  },
  "provenance": {
    "run_key": "ci.example.org/runs/ci-1234",
    "started_at": "2026-07-18T09:12:44Z",
    "subject_dirty": "clean",
    "subject_tree": "4b825dc642cb6eb9a060e54bf8d69288fbee4904",
    "benchmark_dirty": "dirty",
    "benchmark_tree": "9fceb02d0ae598e95dc970b74767f19372d61af8",
    "runner": {
      "name": "example-runner",
      "version": "2.1"
    }
  },
  "quality": {
    "warnings": [
      "dirty-benchmark-tree"
    ]
  }
}

{
  "schema_version": "benchx/measurement-result/0.1.0",
  "producer": {
    "name": "example.org/benchmark-adapter",
    "version": "4.0",
    "mapping_version": "native-to-benchx/v5"
  },
  "ingest_key": "ci-1234:parquet-read/snappy:wall-time",
  "attempt_key": "ci.example.org/runs/ci-1234/attempts/parquet-read-snappy-1",
  "project": "arrow",
  "source": {
    "uri": "https://github.com/apache/arrow",
    "type": "git"
  },
  "revision": {
    "key": "02addad7a1f3ce7d6de6a6230ec4c17828cb8f52"
  },
  "benchmark": {
    "source": {
      "uri": "https://github.com/example/arrow-benchmarks",
      "type": "git"
    },
    "revision": {
      "key": "92ef3aa0c1d5e8b7f4a2936d5c1e7b8a9f0d3c2e"
    }
  },
  "coordinates": {
    "workload": {
      "name": "parquet-read",
      "parameters": {
        "compression": "snappy"
      }
    },
    "subject": {
      "name": "apache-arrow",
      "components": [
        {
          "role": "primary"
        }
      ],
      "configuration": {
        "build_type": "release",
        "compiler": "clang-18"
      }
    },
    "protocol": {
      "name": "benchmark-time",
      "version": "v1",
      "timer": "perf_counter",
      "warmup": {
        "mode": "time",
        "seconds": 1
      },
      "calibration": {
        "mode": "adaptive",
        "minimum_sample_seconds": 0.01
      },
      "filesystem_cache": "drop-before-attempt"
    },
    "environment": {
      "schema_version": "machine/v1",
      "identity": {
        "runner": "bench-01",
        "cpu": "AMD EPYC 7R13",
        "cores": 8
      },
      "system": {
        "os": {
          "kernel": "6.8.0-31-generic"
        },
        "libc": {
          "version": "2.39"
        },
        "runtime": {
          "python": "3.12"
        }
      }
    }
  },
  "measurement": {
    "quantity": {
      "name": "wall-time",
      "unit": "s"
    },
    "status": "success",
    "observations": [
      0.0371,
      0.0364,
      0.0363,
      0.037,
      0.0368
    ],
    "summaries": [
      {
        "type": "estimate",
        "estimator": {
          "name": "mean",
          "method_version": "benchx/mean/v1",
          "input_level": "observations"
        },
        "value": 0.03672,
        "source": "producer"
      },
      {
        "type": "statistic",
        "name": "sample_standard_deviation",
        "value": 0.0003564,
        "method": "n-1",
        "source": "producer"
      },
      {
        "type": "confidence_interval",
        "statistic": "mean",
        "level": 0.95,
        "lower": 0.03628,
        "upper": 0.03716,
        "method": "student-t",
        "source": "producer"
      }
    ]
  },
  "procedure": {
    "inner_iterations": 100,
    "attempted_repetitions": 5,
    "completed_repetitions": 5,
    "warmups_performed": 1,
    "caches_cleared": [
      "filesystem"
    ]
  },
  "provenance": {
    "run_key": "ci.example.org/runs/ci-1234",
    "started_at": "2026-07-18T09:12:44Z",
    "subject_dirty": "clean",
    "subject_tree": "4b825dc642cb6eb9a060e54bf8d69288fbee4904",
    "benchmark_dirty": "dirty",
    "benchmark_tree": "9fceb02d0ae598e95dc970b74767f19372d61af8",
    "runner": {
      "name": "example-runner",
      "version": "2.1"
    },
    "source_payload_sha256": "9a8b7c6d5e4f3a2b1c0d9e8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e0f9a8b",
    "artifacts": [
      {
        "kind": "patch",
        "media_type": "text/x-diff",
        "uri": "file:///tmp/bench/benchmark.patch",
        "sha256": "5f3c1e8a9b2d4c6e7f0a1b3c5d7e9f2a4b6c8d0e1f3a5b7c9d2e4f6a8b0c1d3e"
      }
    ]
  },
  "quality": {
    "warnings": [
      "dirty-benchmark-tree"
    ]
  }
}

{
  "schema_version": "benchx/measurement-result/0.1.0",
  "producer": {
    "name": "example.org/cupyx-adapter",
    "version": "1.3.0",
    "mapping_version": "cupyx-benchmark-to-benchx/v1"
  },
  "ingest_key": "nightly-778:matmul/n=4096,dtype=float32:gpu0:cpu-time",
  "attempt_key": "ci.example.org/runs/nightly-778/attempts/matmul-n4096-float32-gpu0-1",
  "project": "cupy",
  "source": {
    "uri": "https://github.com/cupy/cupy",
    "type": "git"
  },
  "revision": {
    "key": "6c1f2a9d8e7b4c3f0a5d6e7f8091a2b3c4d5e6f7"
  },
  "benchmark": {
    "source": {
      "uri": "https://github.com/example/cupy-benchmarks",
      "type": "git"
    },
    "revision": {
      "key": "b7e2d4c6a8f0e1d3c5b7a9f2e4d6c8b0a2f4e6d8"
    }
  },
  "coordinates": {
    "workload": {
      "name": "matmul",
      "parameters": {
        "n": 4096,
        "dtype": "float32"
      }
    },
    "subject": {
      "name": "cupy",
      "components": [
        {
          "role": "primary"
        }
      ],
      "configuration": {
        "cuda_build": "12.x",
        "distribution": "wheel"
      }
    },
    "protocol": {
      "name": "cupyx-profiler-benchmark",
      "version": "v1",
      "timer": "perf_counter",
      "warmup": {
        "mode": "repetitions",
        "n_warmup": 3
      },
      "repetitions": {
        "mode": "fixed",
        "n_repeat": 10
      }
    },
    "environment": {
      "schema_version": "gpu-host/v1",
      "identity": {
        "runner": "gpu-01",
        "cpu": "AMD EPYC 9354",
        "gpus": [
          "NVIDIA H100 80GB HBM3",
          "NVIDIA H100 80GB HBM3"
        ]
      },
      "system": {
        "os": {
          "kernel": "6.8.0-45-generic"
        },
        "gpu": {
          "driver": "550.90.07"
        },
        "runtime": {
          "python": "3.12"
        }
      }
    },
    "resource_selection": {
      "gpu_index": 0,
      "gpu_uuid": "GPU-3f2a1b4c-5d6e-7f80-91a2-b3c4d5e6f708"
    }
  },
  "measurement": {
    "quantity": {
      "name": "cpu-time",
      "unit": "s"
    },
    "status": "success",
    "observations": [
      {
        "ordinal": 0,
        "value": 0.000221
      },
      {
        "ordinal": 1,
        "value": 0.000214
      },
      {
        "ordinal": 2,
        "value": 0.000218
      },
      {
        "ordinal": 3,
        "value": 0.000213
      },
      {
        "ordinal": 4,
        "value": 0.000226
      },
      {
        "ordinal": 5,
        "value": 0.000215
      },
      {
        "ordinal": 6,
        "value": 0.000212
      },
      {
        "ordinal": 7,
        "value": 0.000219
      },
      {
        "ordinal": 8,
        "value": 0.000217
      },
      {
        "ordinal": 9,
        "value": 0.000214
      }
    ],
    "summaries": [
      {
        "type": "estimate",
        "estimator": {
          "name": "mean",
          "method_version": "benchx/mean/v1",
          "input_level": "observations"
        },
        "value": 0.0002169,
        "source": "producer"
      },
      {
        "type": "statistic",
        "name": "sample_standard_deviation",
        "value": 4.3e-06,
        "method": "n-1",
        "source": "producer"
      }
    ]
  },
  "procedure": {
    "attempted_repetitions": 10,
    "completed_repetitions": 10,
    "warmups_performed": 3
  },
  "provenance": {
    "run_key": "ci.example.org/runs/nightly-778",
    "started_at": "2026-09-13T02:14:07Z",
    "subject_dirty": "clean",
    "subject_tree": "3e0f8a7b6c5d4e3f2a1b0c9d8e7f6a5b4c3d2e1f",
    "benchmark_dirty": "clean",
    "benchmark_tree": "7d1c2b3a4f5e6d7c8b9a0f1e2d3c4b5a6f7e8d9c",
    "runner": {
      "name": "example-gpu-runner",
      "version": "3.0"
    },
    "source_payload_sha256": "1c9e4f7a2b5d8e0f3a6c9b2d5e8f1a4c7b0d3e6f9a2c5b8d1e4f7a0c3b6d9e2f"
  }
}

{
  "schema_version": "benchx/measurement-result/0.1.0",
  "producer": {
    "name": "example.org/cupyx-adapter",
    "version": "1.3.0",
    "mapping_version": "cupyx-benchmark-to-benchx/v1"
  },
  "ingest_key": "nightly-778:matmul/n=4096,dtype=float32:gpu0:gpu-time",
  "attempt_key": "ci.example.org/runs/nightly-778/attempts/matmul-n4096-float32-gpu0-1",
  "project": "cupy",
  "source": {
    "uri": "https://github.com/cupy/cupy",
    "type": "git"
  },
  "revision": {
    "key": "6c1f2a9d8e7b4c3f0a5d6e7f8091a2b3c4d5e6f7"
  },
  "benchmark": {
    "source": {
      "uri": "https://github.com/example/cupy-benchmarks",
      "type": "git"
    },
    "revision": {
      "key": "b7e2d4c6a8f0e1d3c5b7a9f2e4d6c8b0a2f4e6d8"
    }
  },
  "coordinates": {
    "workload": {
      "name": "matmul",
      "parameters": {
        "n": 4096,
        "dtype": "float32"
      }
    },
    "subject": {
      "name": "cupy",
      "components": [
        {
          "role": "primary"
        }
      ],
      "configuration": {
        "cuda_build": "12.x",
        "distribution": "wheel"
      }
    },
    "protocol": {
      "name": "cupyx-profiler-benchmark",
      "version": "v1",
      "timer": "cuda-events",
      "warmup": {
        "mode": "repetitions",
        "n_warmup": 3
      },
      "repetitions": {
        "mode": "fixed",
        "n_repeat": 10
      }
    },
    "environment": {
      "schema_version": "gpu-host/v1",
      "identity": {
        "runner": "gpu-01",
        "cpu": "AMD EPYC 9354",
        "gpus": [
          "NVIDIA H100 80GB HBM3",
          "NVIDIA H100 80GB HBM3"
        ]
      },
      "system": {
        "os": {
          "kernel": "6.8.0-45-generic"
        },
        "gpu": {
          "driver": "550.90.07"
        },
        "runtime": {
          "python": "3.12"
        }
      }
    },
    "resource_selection": {
      "gpu_index": 0,
      "gpu_uuid": "GPU-3f2a1b4c-5d6e-7f80-91a2-b3c4d5e6f708"
    }
  },
  "measurement": {
    "quantity": {
      "name": "gpu-time",
      "unit": "s"
    },
    "status": "success",
    "observations": [
      {
        "ordinal": 0,
        "value": 0.018412
      },
      {
        "ordinal": 1,
        "value": 0.018377
      },
      {
        "ordinal": 2,
        "value": 0.018395
      },
      {
        "ordinal": 3,
        "value": 0.01836
      },
      {
        "ordinal": 4,
        "value": 0.018431
      },
      {
        "ordinal": 5,
        "value": 0.018388
      },
      {
        "ordinal": 6,
        "value": 0.018371
      },
      {
        "ordinal": 7,
        "value": 0.018402
      },
      {
        "ordinal": 8,
        "value": 0.01839
      },
      {
        "ordinal": 9,
        "value": 0.018366
      }
    ],
    "summaries": [
      {
        "type": "estimate",
        "estimator": {
          "name": "mean",
          "method_version": "benchx/mean/v1",
          "input_level": "observations"
        },
        "value": 0.0183892,
        "source": "producer"
      },
      {
        "type": "statistic",
        "name": "sample_standard_deviation",
        "value": 2.2e-05,
        "method": "n-1",
        "source": "producer"
      }
    ]
  },
  "procedure": {
    "attempted_repetitions": 10,
    "completed_repetitions": 10,
    "warmups_performed": 3
  },
  "provenance": {
    "run_key": "ci.example.org/runs/nightly-778",
    "started_at": "2026-09-13T02:14:07Z",
    "subject_dirty": "clean",
    "subject_tree": "3e0f8a7b6c5d4e3f2a1b0c9d8e7f6a5b4c3d2e1f",
    "benchmark_dirty": "clean",
    "benchmark_tree": "7d1c2b3a4f5e6d7c8b9a0f1e2d3c4b5a6f7e8d9c",
    "runner": {
      "name": "example-gpu-runner",
      "version": "3.0"
    },
    "source_payload_sha256": "1c9e4f7a2b5d8e0f3a6c9b2d5e8f1a4c7b0d3e6f9a2c5b8d1e4f7a0c3b6d9e2f"
  }
}

{
  "schema_version": "benchx/measurement-result/0.1.0",
  "producer": {
    "name": "example.org/narwhals-bench",
    "version": "0.4.0",
    "mapping_version": "narwhals-bench-to-benchx/v1"
  },
  "ingest_key": "nightly-912:truncate/rows=1000000:overhead-ratio",
  "attempt_key": "ci.example.org/runs/nightly-912/attempts/truncate-ratio",
  "project": "narwhals",
  "source": {
    "uri": "https://github.com/narwhals-dev/narwhals",
    "type": "git"
  },
  "revision": {
    "key": "c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3"
  },
  "benchmark": {
    "source": {
      "uri": "https://github.com/narwhals-dev/narwhals",
      "type": "git"
    },
    "revision": {
      "key": "c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3"
    }
  },
  "coordinates": {
    "workload": {
      "name": "truncate",
      "parameters": {
        "rows": 1000000
      }
    },
    "subject": {
      "name": "narwhals",
      "components": [
        {"role": "primary"},
        {"role": "pandas", "source": "https://github.com/pandas-dev/pandas", "revision": "v2.2.3"}
      ]
    },
    "protocol": {
      "name": "timeit",
      "version": "v1",
      "timer": "perf_counter",
      "warmup": {"mode": "repetitions", "n_warmup": 1},
      "calibration": {"mode": "adaptive", "minimum_sample_seconds": 0.2},
      "repetitions": {"mode": "fixed", "n_repeat": 5}
    },
    "environment": {
      "schema_version": "machine/v1",
      "identity": {
        "runner": "ci-01",
        "cpu": "AMD EPYC 7R13",
        "cores": 4
      },
      "system": {
        "runtime": {"python": "3.12"}
      }
    }
  },
  "measurement": {
    "quantity": {
      "name": "overhead-ratio",
      "unit": "1",
      "direction": "lower-is-better"
    },
    "status": "success",
    "summaries": [
      {
        "type": "estimate",
        "estimator": {
          "name": "ratio",
          "method_version": "benchx/ratio/v1",
          "input_level": "estimates",
          "parameters": {
            "input_estimator": "min"
          },
          "inputs": [
            {"role": "numerator", "quantity": "wall-time"},
            {"role": "denominator", "quantity": "wall-time"}
          ]
        },
        "value": 1.032,
        "source": "producer"
      }
    ]
  },
  "provenance": {
    "run_key": "ci.example.org/runs/nightly-912",
    "started_at": "2026-09-16T02:40:19Z",
    "subject_dirty": "clean",
    "subject_tree": "1a2b3c4d5e6f7081920a1b2c3d4e5f60718293a4",
    "benchmark_dirty": "clean",
    "benchmark_tree": "1a2b3c4d5e6f7081920a1b2c3d4e5f60718293a4",
    "references": [
      {
        "role": "numerator",
        "result": {"producer": "example.org/narwhals-bench", "ingest_key": "nightly-912:truncate/rows=1000000,role=narwhals:wall-time"}
      },
      {
        "role": "denominator",
        "result": {"producer": "example.org/narwhals-bench", "ingest_key": "nightly-912:truncate/rows=1000000,role=native:wall-time"}
      }
    ],
    "runner": {"name": "example-runner", "version": "2.1"}
  }
}

BenchX measurement result ingest object¶

Properties¶

producer¶

benchmark¶

coordinates¶

coordinates.workload¶

coordinates.workload.dataset¶

coordinates.subject¶

coordinates.environment¶

measurement¶

measurement.quantity¶

procedure¶

provenance¶

provenance.runner¶

quality¶

quality.validation¶

Definitions¶

nonEmptyString¶

token¶

uri¶

sha256¶

treeId¶

dirtyState¶

timestamp¶

finiteNumber¶

primitive¶

parameterValue¶

sourceRef¶

revisionRef¶

resultRef¶

subjectComponent¶

estimator¶

structuredObservation¶

constraint¶

summarySource¶

estimateSummary¶

statisticSummary¶

intervalSummary¶

sourceBoundsSummary¶

standardUncertaintySummary¶

expandedUncertaintySummary¶

summary¶

artifact¶

Examples¶

`producer`¶

`benchmark`¶

`coordinates`¶

`coordinates.workload`¶

`coordinates.workload.dataset`¶

`coordinates.subject`¶

`coordinates.environment`¶

`measurement`¶

`measurement.quantity`¶

`procedure`¶

`provenance`¶

`provenance.runner`¶

`quality`¶

`quality.validation`¶

`nonEmptyString`¶

`token`¶

`uri`¶

`sha256`¶

`treeId`¶

`dirtyState`¶

`timestamp`¶

`finiteNumber`¶

`primitive`¶

`parameterValue`¶

`sourceRef`¶

`revisionRef`¶

`resultRef`¶

`subjectComponent`¶

`estimator`¶

`structuredObservation`¶

`constraint`¶

`summarySource`¶

`estimateSummary`¶

`statisticSummary`¶

`intervalSummary`¶

`sourceBoundsSummary`¶

`standardUncertaintySummary`¶

`expandedUncertaintySummary`¶

`summary`¶

`artifact`¶