BenchX work order¶

urn:benchx:schema:work-order:0.1.0 · version 0.1.0 · schema.json · source

The runner's single input: what to run, at what, how, and for whom (docs/design/runner-schema.md §3). A work order is plain JSON, written by hand, a scheduler, a CI job, or the workbench. The runner reads nothing else.

Implementer note

Shared primitives ($defs/token, $defs/nonEmptyString, $defs/uri, $defs/sha256, $defs/sourceRef, $defs/parameterValue) are $refs into urn:benchx:schema:measurement-result:0.1.0#/$defs; a validator must register both schemas together (e.g. a referencing.Registry in Python, addSchema in ajv). The benchmark field is not in the original runner-schema.md draft: it closes a gap found while planning this schema (see runner-schema.md §3.3) — every non-thin result requires benchmark.source/revision, and nothing in §3's original field table supplied them. plan is required exactly when state is resolved (see the allOf below); this schema cannot express the rest of §3.2's resolution rules (a resolved order never differs from what its draft would resolve to; the runner refuses a draft with an unpinned branch name or a versionless policy). A name may not appear in both environment_variables and workload_parameters; the runner enforces this, not the schema. suites[].filter is mutually exclusive with include/exclude; which an adapter supports is its own capability, not a schema promise. The runner never builds: the former revision target kind and build.cache were removed as unimplemented draft fields, and build now only declares the prebuilt target's configuration. The three kinds of target also take the optional provenance text how_built, activation, shell and provider ($defs/provenanceText): declared by the user, recorded as given, never executed or checked. An earlier revision loosened environment_policy from required to optional and widens precision, adding no breaking change, so it stays at 0.1.0.

Properties¶

Property	Required	Type	Description
`schema_version`	yes	constant `"benchx/work-order/0.1.0"`
`work_order_id`	yes	nonEmptyString	Stable name for this order: a URI or UUID.
`state`	yes	one of `"draft"`, `"resolved"`	Only a resolved order executes (§3.2).
`project`		token	Project in the store. Omit for an ad hoc run; results are then thin (result schema §4.1).
`suites`	yes	array of suite (min items 1)
`quantities`	yes	array of token (min items 1)	Quantities to collect, e.g. wall-time, peak-rss.
`target`	yes	exactly one of: targetWorkingTree; targetBuild; targetArtifact	What code to measure (§3.1): always something already built or present. No kind checks out or builds a revision. See `target`.
`collectors`		array of object	Context collectors (harness-adapter.md §5.1): commands the runner runs once, before the cases, in the harness's working directory and environment. Each prints one JSON object of facts, recorded under coordinates.environment.system.. The runner executes these commands, so `bx run` refuses an order that has them unless told to allow it.
`benchmark`		object	Where the benchmark suite's own code lives, when different from target (§3.3 addendum). Absent means the benchmarks live in the same checkout as target. The revision is absent when the suite is not under version control; the runner then identifies the suite file by its content hash. See `benchmark`.
`components`		array of object	Pinned non-primary components.
`build`		object	The declared configuration of the prebuilt target, for facts the runner cannot read from it. Never an instruction to build. See `build`.
`precision`	yes	object	Harness-neutral precision settings, spelled per the result schema's Appendix B (protocol.{repetitions,calibration,warmup}). See `precision`.
`schedule`		object	Interleaving across sides of a comparison. See `schedule`.
`timeouts`	yes	object	See `timeouts`.
`environment_policy`		object	Named policy (schemas/environment-policy/0.1.0). Optional: absent means the runner applies no policy and only records the environment system and condition measurements, the default `benchmark-environments.md`'s laptop scenario describes. See `environment_policy`.
`environment_variables`		map of string (min properties 1; keys string (pattern `^[A-Za-z_][A-Za-z0-9_]*$`))	Environment variables set for the measured process, on top of the environment the runner inherited. Workload coordinates, not machine facts: each name and value is copied into coordinates.workload.parameters, so a run at OMP_NUM_THREADS=1 and one at 10 are different, non-pooling workload variants. Values are strings; one order carries one assignment.
`workload_parameters`		map of parameterValue (min properties 1)	Parameters passed to the runner or the harness, never as environment variables. Copied into coordinates.workload.parameters alongside environment_variables; a name may not appear in both.
`run_key`	yes	token	Groups this order with sibling orders in one comparison.
`round`		integer (>= 0)	Position in an interleaved run; set by the run author.
`labels`		map of nonEmptyString	Caller labels, e.g. {"env": "numpy-2.0"}.
`requester`	yes	object	See `requester`.
`reason`		nonEmptyString	Free text: why this run exists.
`plan`		array of object (min items 1)	Every case x quantity the run will attempt. Required exactly when state is resolved (§3.2, §6). For a harness whose cases are only enumerable from the prebuilt target (e.g. by filter against a compiled binary), the runner fills this in as part of resolving a draft order: it runs the harness's own case-listing step, writes the filled plan into a new resolved order, hashes and records that as the provenance artifact, and only then executes. A hand-authored resolved order (e.g. from a scheduler that already knows the case list) skips that step.

No other properties are allowed.

Rules:

If state is "resolved", then requires plan.
If state is "draft", then forbids plan.

`target`¶

What code to measure (§3.1): always something already built or present. No kind checks out or builds a revision.

Type: exactly one of: targetWorkingTree; targetBuild; targetArtifact

`benchmark`¶

Where the benchmark suite's own code lives, when different from target (§3.3 addendum). Absent means the benchmarks live in the same checkout as target. The revision is absent when the suite is not under version control; the runner then identifies the suite file by its content hash.

Property	Required	Type	Description
`source`	yes	sourceRef
`revision`		nonEmptyString

No other properties are allowed.

`build`¶

The declared configuration of the prebuilt target, for facts the runner cannot read from it. Never an instruction to build.

Property	Required	Type	Description
`profile`		nonEmptyString
`type`		nonEmptyString
`compiler`		nonEmptyString
`flags`		array of string
`options`		object

No other properties are allowed.

Rules:

At least one property must be present.

`precision`¶

Harness-neutral precision settings, spelled per the result schema's Appendix B (protocol.{repetitions,calibration,warmup}).

Property	Required	Type	Description
`repetitions`	yes	exactly one of: integer (>= 1); repetitionLevels	How many observations to take. A bare integer is shorthand for one level at unit 'repetition'; the object form supports harnesses that repeat at more than one level (e.g. a process count outside a per-process repetition count). See `precision.repetitions`.
`calibration`		object	How the inner iteration count inside one repetition is chosen. See `precision.calibration`.
`warmup`		object	Discarded measurements before the recorded ones. mode distinguishes a count-based warmup (a number of repetitions) from a duration-based one, since harnesses split on this (e.g. Google Benchmark warms up by time only). See `precision.warmup`.

No other properties are allowed.

`precision.repetitions`¶

How many observations to take. A bare integer is shorthand for one level at unit 'repetition'; the object form supports harnesses that repeat at more than one level (e.g. a process count outside a per-process repetition count).

Type: exactly one of: integer (>= 1); repetitionLevels

`precision.calibration`¶

How the inner iteration count inside one repetition is chosen.

Property	Required	Type
`mode`	yes	one of `"adaptive"`, `"fixed"`
`minimum_sample_seconds`		number (> 0)
`n_iterations`		integer (>= 1)

No other properties are allowed.

Rules:

If mode is "adaptive", then requires minimum_sample_seconds; forbids n_iterations.
If mode is "fixed", then requires n_iterations; forbids minimum_sample_seconds.

`precision.warmup`¶

Discarded measurements before the recorded ones. mode distinguishes a count-based warmup (a number of repetitions) from a duration-based one, since harnesses split on this (e.g. Google Benchmark warms up by time only).

Property	Required	Type
`mode`	yes	one of `"none"`, `"count"`, `"time"`
`n_warmup`		integer (>= 0)
`seconds`		number (> 0)

No other properties are allowed.

Rules:

If mode is "count", then requires n_warmup; forbids seconds.
If mode is "time", then requires seconds; forbids n_warmup.
If mode is "none", then forbids n_warmup, seconds.

`schedule`¶

Interleaving across sides of a comparison.

Property	Required	Type	Description
`kind`	yes	one of `"sequential"`, `"alternating"`, `"random"`
`seed`		integer

No other properties are allowed.

`timeouts`¶

Property	Required	Type	Description
`case_s`	yes	number (> 0)
`order_s`	yes	number (> 0)

No other properties are allowed.

`environment_policy`¶

Named policy (schemas/environment-policy/0.1.0). Optional: absent means the runner applies no policy and only records the environment system and condition measurements, the default benchmark-environments.md's laptop scenario describes.

Property	Required	Type	Description
`name`	yes	token
`version`	yes	nonEmptyString

No other properties are allowed.

`requester`¶

Property	Required	Type	Description
`kind`	yes	one of `"scheduler"`, `"ci"`, `"workbench"`, `"person"`
`name`	yes	nonEmptyString

No other properties are allowed.

Definitions¶

Reusable subschemas referenced from the tables above and from other schemas.

`token`¶

Type: token

`nonEmptyString`¶

Type: nonEmptyString

`uri`¶

Type: uri

`repetitionLevels`¶

Property	Required	Type	Description
`mode`	yes	constant `"fixed"`
`levels`	yes	array of object (min items 1)

No other properties are allowed.

`suite`¶

Property	Required	Type	Description
`adapter`	yes	token
`suite`	yes	nonEmptyString
`filter`		nonEmptyString	Harness-native filter string, passed through unchanged (e.g. a Google Benchmark --benchmark_filter regular expression). An alternative to include/exclude for adapters with only single-pattern, no-exclude selection; mutually exclusive with them.
`include`		array of string
`exclude`		array of string
`parameters`		object

No other properties are allowed.

`targetWorkingTree`¶

Property	Required	Type
`kind`	yes	constant `"working_tree"`
`path`	yes	nonEmptyString
`source`	yes	sourceRef
`source_dir`		sourceDir
`how_built`		provenanceText.how_built
`activation`		provenanceText.activation
`shell`		provenanceText.shell
`provider`		provenanceText.provider

No other properties are allowed.

`targetBuild`¶

Property	Required	Type
`kind`	yes	constant `"build"`
`path`	yes	nonEmptyString
`source`	yes	sourceRef
`source_dir`		sourceDir
`how_built`		provenanceText.how_built
`activation`		provenanceText.activation
`shell`		provenanceText.shell
`provider`		provenanceText.provider

No other properties are allowed.

`targetArtifact`¶

Property	Required	Type
`kind`	yes	constant `"artifact"`
`uri`	yes	uri
`sha256`	yes	sha256
`source`	yes	sourceRef
`revision`	yes	nonEmptyString
`how_built`		provenanceText.how_built
`activation`		provenanceText.activation
`shell`		provenanceText.shell
`provider`		provenanceText.provider

No other properties are allowed.

`provenanceText`¶

User-owned text about how a target was prepared, declared and recorded as given (benchmark-environments.md §3.2). Never executed and never checked; the runner copies it into provenance.info of every result. Absent, null, empty, or whitespace-only means not declared. None of it enters identity. It is copied verbatim into results that may reach a shared store, so none of it may hold secrets.

Property	Type	Description
`how_built`	exactly one of: string or null (max length 8192); object	What the user did to build the target, such as the cmake commands or the name of the build script; or, as an object, structured facts about the build that a collector with verify: how_built must observe exactly (an object is never parsed from text). A string is never checked. See `provenanceText.how_built`.
`activation`	string or null (max length 8192)	What the user ran to enter the target's environment, such as sourcing an activate script, loading modules, or exporting library paths. The user activates before invoking benchx; the runner records the environment it inherited beside this declaration.
`shell`	string or null (pattern `^$\|^[A-Za-z0-9_.+-]+$\|^/\S+$`)	The shell the activation text is written for, such as bash: a bare name or an absolute path. Never invoked.
`provider`	object	Audit-only identity of the target provider that produced the description. See `provenanceText.provider`.

`provenanceText.how_built`¶

What the user did to build the target, such as the cmake commands or the name of the build script; or, as an object, structured facts about the build that a collector with verify: how_built must observe exactly (an object is never parsed from text). A string is never checked.

Type: exactly one of: string or null (max length 8192); object

`provenanceText.provider`¶

Audit-only identity of the target provider that produced the description.

Property	Required	Type	Description
`name`	yes	string (min length 1)
`version`		string (min length 1)

No other properties are allowed.

`sourceDir`¶

Local checkout path to inspect for revision, dirty state, and tree id, when it differs from what the target itself records (e.g. a build directory's own CMAKE_HOME_DIRECTORY). Optional; falls back to the target's own record when absent.

Type: nonEmptyString

Examples¶

adhoc-minimalarrow-laptopdraft

{
  "schema_version": "benchx/work-order/0.1.0",
  "work_order_id": "urn:uuid:7c9e6679-7425-40de-944b-e07fc1f90ae7",
  "state": "resolved",
  "suites": [
    { "adapter": "google-benchmark", "suite": "demo-bench", "filter": "BM_Quiet.*" }
  ],
  "quantities": ["wall-time"],
  "target": {
    "kind": "build",
    "path": "/tmp/bench-plain",
    "source": { "uri": "https://example.org/demo.git", "type": "git" }
  },
  "precision": { "repetitions": 5 },
  "timeouts": { "case_s": 30, "order_s": 600 },
  "run_key": "local:demo-2026-09-29",
  "requester": { "kind": "workbench", "name": "local" },
  "plan": [
    { "id": "p0", "case": "BM_Quiet/1024", "quantity": "wall-time", "from": "BM_Quiet.*" }
  ]
}

{
  "schema_version": "benchx/work-order/0.1.0",
  "work_order_id": "urn:uuid:6f1c2b7e-3d4a-4e8b-9a51-0c2d7e9f4b13",
  "state": "resolved",
  "project": "arrow",
  "suites": [
    {
      "adapter": "google-benchmark",
      "suite": "parquet-read",
      "include": ["parquet-read/snappy"],
      "parameters": {}
    }
  ],
  "quantities": ["wall-time"],
  "target": {
    "kind": "working_tree",
    "path": "~/arrow",
    "source": { "uri": "https://github.com/apache/arrow", "type": "git" },
    "source_dir": "~/arrow"
  },
  "build": {
    "profile": "ninja-release",
    "type": "release",
    "compiler": "clang-18"
  },
  "precision": {
    "repetitions": 5,
    "calibration": { "mode": "adaptive", "minimum_sample_seconds": 0.01 },
    "warmup": { "mode": "count", "n_warmup": 1 }
  },
  "timeouts": { "case_s": 60, "order_s": 1800 },
  "environment_policy": { "name": "laptop-default", "version": "1.0.0" },
  "environment_variables": { "OMP_NUM_THREADS": "1" },
  "run_key": "local:parquet-snappy-2026-09-23",
  "round": 1,
  "labels": { "side": "contender" },
  "requester": { "kind": "workbench", "name": "local" },
  "reason": "check snappy decode change before opening PR",
  "plan": [{ "id": "p0", "case": "parquet-read/snappy", "quantity": "wall-time" }]
}

{
  "schema_version": "benchx/work-order/0.1.0",
  "work_order_id": "urn:uuid:9b2e4f1a-6c3d-4e8f-a7b0-1c2d3e4f5a6b",
  "state": "draft",
  "suites": [
    { "adapter": "google-benchmark", "suite": "demo-bench", "filter": "BM_.*" }
  ],
  "quantities": ["wall-time"],
  "target": {
    "kind": "build",
    "path": "/tmp/bench-plain",
    "source": { "uri": "https://example.org/demo.git", "type": "git" }
  },
  "precision": { "repetitions": 5 },
  "timeouts": { "case_s": 30, "order_s": 600 },
  "run_key": "local:demo-2026-09-29",
  "requester": { "kind": "workbench", "name": "local" }
}

BenchX work order¶

Properties¶

target¶

benchmark¶

build¶

precision¶

precision.repetitions¶

precision.calibration¶

precision.warmup¶

schedule¶

timeouts¶

environment_policy¶

requester¶

Definitions¶

token¶

nonEmptyString¶

uri¶

sha256¶

sourceRef¶

parameterValue¶

repetitionLevels¶

suite¶

targetWorkingTree¶

targetBuild¶

targetArtifact¶

provenanceText¶

provenanceText.how_built¶

provenanceText.provider¶

sourceDir¶

Examples¶

`target`¶

`benchmark`¶

`build`¶

`precision`¶

`precision.repetitions`¶

`precision.calibration`¶

`precision.warmup`¶

`schedule`¶

`timeouts`¶

`environment_policy`¶

`requester`¶

`token`¶

`nonEmptyString`¶

`uri`¶

`sha256`¶

`sourceRef`¶

`parameterValue`¶

`repetitionLevels`¶

`suite`¶

`targetWorkingTree`¶

`targetBuild`¶

`targetArtifact`¶

`provenanceText`¶

`provenanceText.how_built`¶

`provenanceText.provider`¶

`sourceDir`¶