BenchX project configuration¶
urn:benchx:schema:config:0.1.0 · version 0.1.0 · schema.json · source
benchx.conf.json, in the project root: how this project is measured. Every key is optional here; the workbench takes a value from the command line first, then from this file, then from a built-in default, and refuses a command that is left without a value it requires (adapter.name, adapter.suite, a target). Relative paths are relative to the directory holding the file.
Implementer note
adapter.precision is checked against the work-order precision schema by the loader, after it is merged over the harness defaults, because it may be partial here (a repetition level, or calibration or warmup, may be left to the harness). adapter.name must name a harness; the loader checks that, so a new harness needs no schema change.
Properties¶
| Property | Required | Type | Description |
|---|---|---|---|
schema_version |
yes | constant "benchx/config/0.1.0" |
|
project |
string (min length 1) | Names the project, so results join its series. Without it a run is ad hoc, and the store files it as local/ |
|
source |
object | See source. |
|
provider |
object | The target provider, called for an @CONFIG target. It runs with the project root as its working directory. See provider. |
|
adapter |
object | The harness, and what is specific to it. See adapter. |
|
collectors |
map of object | Context collectors, by name (harness-adapter.md §5.1), copied into every work order. The commands run in the harness's working directory, so a command argument may contain {root}, replaced by the project root. verify: how_built requires the target's how_built object to be observed exactly; expect requires facts by exact value; on_fail says what a failure does. | |
environment_variables |
map of string | Set for the benchmark's processes; each becomes a workload parameter. | |
timeouts |
object | See timeouts. |
|
paths |
object | See paths. |
|
compare |
object | See compare. |
No other properties are allowed.
source¶
| Property | Required | Type | Description |
|---|---|---|---|
uri |
string (min length 1) | The source's URI, when the checkouts' origin remote does not give it. |
No other properties are allowed.
provider¶
The target provider, called for an @CONFIG target. It runs with the project root as its working directory.
| Property | Required | Type | Description |
|---|---|---|---|
command |
yes | array of string (min length 1) (min items 1) |
No other properties are allowed.
adapter¶
The harness, and what is specific to it.
| Property | Required | Type | Description |
|---|---|---|---|
name |
yes | string (min length 1) | pyperf or google-benchmark. |
suite |
string (min length 1) | The benchmark program in the target: pyperf, the module's path in the working tree; google-benchmark, the binary in the build directory. | |
quantities |
array of string (min length 1) (min items 1; unique items) | What one run collects, from those the harness reports. | |
precision |
object | A work-order precision, possibly partial. |
No other properties are allowed.
timeouts¶
| Property | Required | Type | Description |
|---|---|---|---|
case_s |
number (> 0) | ||
order_s |
number (> 0) |
No other properties are allowed.
paths¶
| Property | Required | Type | Description |
|---|---|---|---|
store |
string (min length 1) | The local store (default store.parquet). | |
results |
string (min length 1) | Where result files are written (default results). |
No other properties are allowed.
compare¶
| Property | Required | Type | Description |
|---|---|---|---|
profile |
one of "revisions", "environments", "variants" |
||
rounds |
integer (>= 1) | ||
k |
number (> 0) | ||
min_rounds |
integer (>= 1) |
No other properties are allowed.
Examples¶
{
"schema_version": "benchx/config/0.1.0",
"adapter": {
"name": "google-benchmark",
"suite": "bench/example-bench",
"quantities": ["wall-time", "cpu-time"],
"precision": {
"repetitions": 10,
"calibration": { "mode": "adaptive", "minimum_sample_seconds": 0.5 },
"warmup": { "mode": "time", "seconds": 0.1 }
}
},
"timeouts": { "case_s": 120, "order_s": 1800 }
}
{
"schema_version": "benchx/config/0.1.0",
"adapter": {
"name": "pyperf",
"suite": "benchmarks/bench_example.py",
"quantities": ["wall-time"],
"precision": {
"repetitions": { "mode": "fixed", "levels": [{ "unit": "process", "n": 20 }, { "unit": "value", "n": 3 }] },
"calibration": { "mode": "adaptive", "minimum_sample_seconds": 0.1 },
"warmup": { "mode": "count", "n_warmup": 1 }
}
},
"environment_variables": {
"OMP_NUM_THREADS": "1",
"OPENBLAS_NUM_THREADS": "1",
"MKL_NUM_THREADS": "1"
},
"timeouts": { "case_s": 120, "order_s": 1800 }
}