Runner¶
The runner executes scenarios through a device pool and writes the run's report artifacts.
bajutsu.common.runner.pipeline
¶
Run every scenario through a device pool and write the run's report artifacts.
with_lifecycle_phases(eff, scenarios, target_effs=None, *, skip=())
¶
Fold the target config's hooks.setup / hooks.cleanup into each scenario's own (BE-0392).
The two merge in opposite orders. setup is config-then-scenario, like interrupts: the
app-wide prelude seeds the state this scenario's own setup then builds on. cleanup is
scenario-then-config, so this scenario releases the record it created before the app-wide
teardown closes around it — the last-acquired-first-released order a fixture-based teardown pair
gives.
Applied to the scenario itself rather than passed beside it, so the scenario the runner executes
and the scenario whose setup / cleanup blocks the report renders are one object. Handing the
merged lists to the runner separately would leave the report pairing an app-wide step's outcome
with the scenario's own step definition, and drop an app-wide cleanup rule's outcomes entirely.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
eff
|
Effective
|
The run's primary target config — the only one a scenario declaring no |
required |
scenarios
|
list[Scenario]
|
The scenarios to fold into; returned unchanged when nothing folds in. |
required |
target_effs
|
Mapping[str, Effective] | None
|
Every declared target's own config, keyed by name (BE-0428). A self-declaring
scenario folds each of these in instead of |
None
|
skip
|
Collection[int]
|
Indices of scenarios that never run — the ones the worker capability check failed (BE-0450). Their targets were never resolved, so folding would read the primary's hooks under another target's name; they keep their own phases only. |
()
|
run_all(eff, scenarios, lease, targets=None, clock=None, alert_guard=None, alert_guard_for=None, run_dir=None, workers=1, bindings=None, secret_values=None, progress=None, baselines_dir=None, schemas_dir=None, actuator=None, resolve_actuator=None, worker_failures=None, golden_context=None, lease_udid_spec='booted', on_score=None, crash_retries=None, crash_recovery_budget=None, run_crash_recovery_budget=None, force_erase_on_retry=True, cancelled=not_cancelled, trace_driver=False, step_gate=None)
¶
Run every scenario, each on a freshly leased device, and return one result per scenario.
lease(eff, scenario) blocks until a device is free, launches the app, and returns a Lease
bundling the live driver with that device's evidence sink / relaunch / control / network
collector; lease.release() afterwards terminates the app and returns the device to the pool.
A lease's collector, when present, has its exchanges cleared per scenario, exposed to request
assertions, and written to <sid>/network.json (redacted with secret_values).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
eff
|
Effective
|
The resolved target config (drives redaction, backend, launch). |
required |
scenarios
|
list[Scenario]
|
The scenarios to run; results come back in this declaration order. |
required |
lease
|
LeaseFn
|
Leases a device and launches the app for one scenario (a single-device run is a pool of one). |
required |
targets
|
Mapping[str, TargetPool] | None
|
Every target a scenario in this run declares, already resolved and pooled by the
caller, keyed by name (BE-0428) — the primary's own entry included. Empty keeps the
single-target path, where |
None
|
clock
|
Clock | None
|
Injectable time source for condition waits, so tests need no real sleeps. None uses the real clock. |
None
|
alert_guard
|
AlertGuardConfig | None
|
A single alert-guard handler, used by tests. |
None
|
alert_guard_for
|
AlertGuardFor | None
|
Picks each scenario's alert-guard handler (honoring its |
None
|
run_dir
|
Path | None
|
Where per-scenario artifacts (network.json, visual diffs) are written. None skips writing them. |
None
|
workers
|
int
|
Concurrent scenarios; >1 hands each worker its own device + per-device resources, so the loop keeps no shared mutable state. |
1
|
bindings
|
Mapping[str, str] | None
|
|
None
|
secret_values
|
list[str] | None
|
The raw secret values to redact from evidence. |
None
|
progress
|
ProgressFn | None
|
Receives one-line progress messages (the web UI streams these). None is silent. |
None
|
baselines_dir
|
Path | None
|
Baseline images for |
None
|
schemas_dir
|
Path | None
|
Directory the |
None
|
actuator
|
str | None
|
The single selected actuator (e.g. |
None
|
resolve_actuator
|
Callable[[Scenario], str] | None
|
Per-scenario actuator resolver (BE-0240); when given, each scenario's
actuator — and thus the capability set it is preflighted against — is resolved from the
scenario's own steps (cheapest sufficient), instead of the one fixed |
None
|
worker_failures
|
Mapping[int, str] | None
|
Scenarios the worker capability check (BE-0450) already failed, keyed by index into scenarios, each with its prefixed reason; they are reported, never leased. |
None
|
golden_context
|
GoldenContext | None
|
Goldens directory for |
None
|
lease_udid_spec
|
str
|
The run's resolved udid spec (the provider's |
'booted'
|
on_score
|
Callable[[Score], None] | None
|
Sink for the app's entry-screen convention score, emitted once from the first
scenario's freshly launched driver (the |
None
|
crash_retries
|
int | None
|
How many times to re-run a scenario whose backend crashed mid-run
( |
None
|
crash_recovery_budget
|
float | None
|
A wall-clock ceiling (seconds) on the total time one scenario may spend
respawning after a crash, on top of |
None
|
run_crash_recovery_budget
|
float | None
|
A wall-clock ceiling (seconds) on the total time crash recovery may
spend across this one |
None
|
force_erase_on_retry
|
bool
|
Whether a crash-triggered retry (attempt > 1) may force
|
True
|
cancelled
|
CancelSource
|
Reports whether this run has been asked to stop (BE-0370). A scenario the request
reaches — at a step boundary, inside a condition wait's poll, or before it was leased at
all — comes back as |
not_cancelled
|
trace_driver
|
bool
|
|
False
|
step_gate
|
StepGate | None
|
|
None
|
Returns:
| Type | Description |
|---|---|
list[RunResult]
|
One result per scenario, in the same order as |
run_and_report(eff, scenarios, lease, runs_dir, run_id, targets=None, clock=None, alert_guard=None, alert_guard_for=None, workers=1, bindings=None, secret_values=None, source_name=None, description=None, plan_sources=None, progress=None, baselines_dir=None, schemas_dir=None, actuator=None, resolve_actuator=None, worker_failures=None, config_source=None, exec_provenance=None, label=None, golden_context=None, lease_udid_spec='booted', on_score=None, force_erase_on_retry=True, cancelled=not_cancelled, trace_driver=False, step_gate=None)
¶
Run the scenarios, then write the run's artifacts under runs_dir/run_id.
Wraps run_all and persists the report: manifest.json, JUnit XML, and the executed
scenario.yaml (so a run is re-runnable / reviewable).
Beyond run_all's arguments (targets, force_erase_on_retry,
cancelled, and trace_driver pass
straight through — see their docstrings there), runs_dir + run_id locate this run's artifact directory
(runs_dir/run_id),
source_name / description are recorded in the report, plan_sources (keyed by declared
scenario name — run/cli.py's _load_scenarios) feeds the report's verbatim YAML / per-step
line numbers, and config_source — the Git source the config came from (BE-0063), or None for
a local config — is stamped into the manifest's provenance so a branch-based run states the
exact commit it executed.
Returns:
| Type | Description |
|---|---|
tuple[list[RunResult], Path]
|
The per-scenario results and the path to the written |
run_matrix_and_report(eff, scenarios, engines, run_pass, runs_dir, run_id, *, source_name=None, description=None, plan_sources=None, secret_values=None, config_source=None, exec_provenance=None, label=None, cancelled=not_cancelled, worker_failures=None)
¶
Run the scenarios once per engine, then assemble ONE report at runs_dir/run_id (BE-0076).
The cross-browser fan-out: a loop over engines, each a full pass. run_pass(engine, run_dir)
runs the selected scenarios for one engine against its own pool, writing that engine's evidence
under run_dir (the caller hands it runs_dir/run_id/<engine>, prefixing the existing NN-slug
layout so two engines never collide); its results are tagged with engine here. The passes'
tagged results are concatenated into one flat list and written as a single manifest / JUnit /
report — the manifest's matrix block aggregates the per-engine verdicts, and ok is
all-must-pass across every engine x scenario (pure aggregation, no LLM).
cancelled (BE-0370) is read between passes: each one first builds a whole device_pool, so a
cancel during the first engine would otherwise still pay every remaining engine's bring-up and
teardown before the run could finish, overrunning the grace period. Every scenario of an engine
that never ran is failed as cancelled rather than left out, so the matrix still names every
requested engine and ok aggregates a cancelled run to False — dropping those engines instead
would let a first pass that happened to be green aggregate to a PASS for a run that never
finished.
Returns:
| Type | Description |
|---|---|
tuple[list[RunResult], Path]
|
The concatenated per-engine results and the path to the written |
report_run_matrix(eff, scenarios, results, plan_order, runs_dir, run_id, *, target_effs=None, not_applicable=None, source_name=None, description=None, plan_sources=None, secret_values=None, config_source=None, exec_provenance=None, label=None, worker_failures=None)
¶
Write ONE report for the runs runsOn conditions formed across scenarios (BE-0452).
The caller has already run each run against its own eligible devices and tagged every result with its coordinates; this only assembles the report. plan_order names the scenario each result ran, since a scenario contributes one result per run. not_applicable records the runs no available device fell in an open range for, which the manifest lists apart from the results.
Returns:
| Type | Description |
|---|---|
Path
|
The path to the written |
reroot_evidence(r, prefix)
¶
Prefix r's run-dir-relative evidence paths with <prefix>/, as a matrix pass writes them.
A run formed by runsOn conditions writes its evidence under its own subdirectory of the run
dir, as an engine pass of --browsers does (BE-0076, BE-0452).