Skip to content

Runner

The runner executes scenarios through a device pool and writes the run's report artifacts.

bajutsu.common.runner.pipeline

Run every scenario through a device pool and write the run's report artifacts.

with_lifecycle_phases(eff, scenarios, target_effs=None, *, skip=())

Fold the target config's hooks.setup / hooks.cleanup into each scenario's own (BE-0392).

The two merge in opposite orders. setup is config-then-scenario, like interrupts: the app-wide prelude seeds the state this scenario's own setup then builds on. cleanup is scenario-then-config, so this scenario releases the record it created before the app-wide teardown closes around it — the last-acquired-first-released order a fixture-based teardown pair gives.

Applied to the scenario itself rather than passed beside it, so the scenario the runner executes and the scenario whose setup / cleanup blocks the report renders are one object. Handing the merged lists to the runner separately would leave the report pairing an app-wide step's outcome with the scenario's own step definition, and drop an app-wide cleanup rule's outcomes entirely.

Parameters:

Name Type Description Default
eff Effective

The run's primary target config — the only one a scenario declaring no targets reads.

required
scenarios list[Scenario]

The scenarios to fold into; returned unchanged when nothing folds in.

required
target_effs Mapping[str, Effective] | None

Every declared target's own config, keyed by name (BE-0428). A self-declaring scenario folds each of these in instead of eff; None keeps the single-target path.

None
skip Collection[int]

Indices of scenarios that never run — the ones the worker capability check failed (BE-0450). Their targets were never resolved, so folding would read the primary's hooks under another target's name; they keep their own phases only.

()

run_all(eff, scenarios, lease, targets=None, clock=None, alert_guard=None, alert_guard_for=None, run_dir=None, workers=1, bindings=None, secret_values=None, progress=None, baselines_dir=None, schemas_dir=None, actuator=None, resolve_actuator=None, worker_failures=None, golden_context=None, lease_udid_spec='booted', on_score=None, crash_retries=None, crash_recovery_budget=None, run_crash_recovery_budget=None, force_erase_on_retry=True, cancelled=not_cancelled, trace_driver=False, step_gate=None)

Run every scenario, each on a freshly leased device, and return one result per scenario.

lease(eff, scenario) blocks until a device is free, launches the app, and returns a Lease bundling the live driver with that device's evidence sink / relaunch / control / network collector; lease.release() afterwards terminates the app and returns the device to the pool. A lease's collector, when present, has its exchanges cleared per scenario, exposed to request assertions, and written to <sid>/network.json (redacted with secret_values).

Parameters:

Name Type Description Default
eff Effective

The resolved target config (drives redaction, backend, launch).

required
scenarios list[Scenario]

The scenarios to run; results come back in this declaration order.

required
lease LeaseFn

Leases a device and launches the app for one scenario (a single-device run is a pool of one).

required
targets Mapping[str, TargetPool] | None

Every target a scenario in this run declares, already resolved and pooled by the caller, keyed by name (BE-0428) — the primary's own entry included. Empty keeps the single-target path, where eff and lease alone are read, and refuses any scenario declaring two or more targets, since nothing could launch the second one.

None
clock Clock | None

Injectable time source for condition waits, so tests need no real sleeps. None uses the real clock.

None
alert_guard AlertGuardConfig | None

A single alert-guard handler, used by tests.

None
alert_guard_for AlertGuardFor | None

Picks each scenario's alert-guard handler (honoring its systemAlert interrupts entries); takes precedence over alert_guard.

None
run_dir Path | None

Where per-scenario artifacts (network.json, visual diffs) are written. None skips writing them.

None
workers int

Concurrent scenarios; >1 hands each worker its own device + per-device resources, so the loop keeps no shared mutable state.

1
bindings Mapping[str, str] | None

secrets.<name> → value substitutions applied to step inputs.

None
secret_values list[str] | None

The raw secret values to redact from evidence.

None
progress ProgressFn | None

Receives one-line progress messages (the web UI streams these). None is silent.

None
baselines_dir Path | None

Baseline images for visual assertions. None disables visual comparison.

None
schemas_dir Path | None

Directory the responseSchema assertions' schema files resolve against. None disables them.

None
actuator str | None

The single selected actuator (e.g. xcuitest / playwright); when given, each scenario is preflighted against its static capability set and failed up front if it needs a capability the actuator lacks (BE-0082). None skips the fixed preflight (a lease driven directly in tests, or when resolve_actuator chooses per scenario instead).

None
resolve_actuator Callable[[Scenario], str] | None

Per-scenario actuator resolver (BE-0240); when given, each scenario's actuator — and thus the capability set it is preflighted against — is resolved from the scenario's own steps (cheapest sufficient), instead of the one fixed actuator. Mutually exclusive with actuator (passing both raises): the CLI's single-engine path and audit pass this, the cross-browser matrix passes actuator.

None
worker_failures Mapping[int, str] | None

Scenarios the worker capability check (BE-0450) already failed, keyed by index into scenarios, each with its prefixed reason; they are reported, never leased.

None
golden_context GoldenContext | None

Goldens directory for golden assertions (BE-0006). None disables them.

None
lease_udid_spec str

The run's resolved udid spec (the provider's udid_spec). A WebDriver URL routes the run to the live XCUITest environment, so the preflight narrows to that transport's set (BE-0238) — the same is_webdriver_endpoint signal environment_for routes on. "booted" (the default) is never a URL, so the local path is unchanged.

'booted'
on_score Callable[[Score], None] | None

Sink for the app's entry-screen convention score, emitted once from the first scenario's freshly launched driver (the run --score inline of doctor's grade). None (the default) scores nothing; diagnostic only, never on the verdict path.

None
crash_retries int | None

How many times to re-run a scenario whose backend crashed mid-run (base.BackendCrashError) on a fresh device before failing it. A crash is backend infrastructure, not a verdict; the default (None) reads BAJUTSU_CRASH_RETRIES — 1 when unset — so a loaded CI lane can raise the budget without a code change, while a scenario that crashes every attempt still fails loudly once it is spent (BE-0049). 0 disables the recovery. The replay re-runs the whole scenario on a respawned (not erased) app, so it is safe only for scenarios idempotent up to the crash point.

None
crash_recovery_budget float | None

A wall-clock ceiling (seconds) on the total time one scenario may spend respawning after a crash, on top of crash_retries (the count). None reads BAJUTSU_CRASH_RECOVERY_BUDGET — unset is unbounded (count is the only cap, unchanged). It stops recovery once spent so a never-recovering runner can't burn crash_retries x the cold-startup ceiling and blow a job's timeout; the first respawn is never blocked, so a genuine one-off is still ridden out.

None
run_crash_recovery_budget float | None

A wall-clock ceiling (seconds) on the total time crash recovery may spend across this one run_all call, not just one scenario. Note the scoping: the cross-browser matrix (run_matrix_and_report) runs run_all once per engine, so each engine pass gets its own full budget rather than sharing one. None reads BAJUTSU_RUN_CRASH_RECOVERY_BUDGET — unset is unbounded, unchanged from before this parameter existed. crash_recovery_budget resets for every new scenario, so a device that keeps degrading pays it again and again; this bounds the cumulative spend instead, so the run fails loudly once it is exhausted rather than each scenario silently re-spending its own budget until an external timeout cancels the job. The first respawn anywhere in the run is never blocked, the same never-block-the-first-respawn rule crash_recovery_budget already follows per scenario.

None
force_erase_on_retry bool

Whether a crash-triggered retry (attempt > 1) may force preconditions.subject.reinstall: erase, the same recovery a scenario already gets by declaring it (see _ScenarioRunner.run_one). True (the default) preserves every existing caller's behavior. bajutsu run --no-erase passes False here, carrying the operator's explicit opt-out past _filter_scenarios — a scenario that never erased reads the same with --no-erase or without it, so its preconditions.subject.reinstall alone cannot distinguish "the operator asked to keep the device" from "nobody said anything" by the time a retry decides whether to force it.

True
cancelled CancelSource

Reports whether this run has been asked to stop (BE-0370). A scenario the request reaches — at a step boundary, inside a condition wait's poll, or before it was leased at all — comes back as RunResult(ok=False, failure="cancelled"), so the caller still receives one result per scenario and writes an ordinary failed run's report. The default never cancels, leaving every existing caller unchanged.

not_cancelled
trace_driver bool

bajutsu run --trace-driver (BE-0415): write one <sid>/driver_trace.json per scenario, recording every Python<->driver call — the driver method invoked, its host-device round trips, and (on Android) which fell back to a subprocess — attributed to the step it happened during. False (the default) writes nothing; diagnostic only, never on the verdict path.

False
step_gate StepGate | None

bajutsu run --step: the pause hook the step loop calls at each step boundary. It may raise RunCancelled to end the run and never chooses a verdict. None (the default) never pauses.

None

Returns:

Type Description
list[RunResult]

One result per scenario, in the same order as scenarios.

run_and_report(eff, scenarios, lease, runs_dir, run_id, targets=None, clock=None, alert_guard=None, alert_guard_for=None, workers=1, bindings=None, secret_values=None, source_name=None, description=None, plan_sources=None, progress=None, baselines_dir=None, schemas_dir=None, actuator=None, resolve_actuator=None, worker_failures=None, config_source=None, exec_provenance=None, label=None, golden_context=None, lease_udid_spec='booted', on_score=None, force_erase_on_retry=True, cancelled=not_cancelled, trace_driver=False, step_gate=None)

Run the scenarios, then write the run's artifacts under runs_dir/run_id.

Wraps run_all and persists the report: manifest.json, JUnit XML, and the executed scenario.yaml (so a run is re-runnable / reviewable).

Beyond run_all's arguments (targets, force_erase_on_retry, cancelled, and trace_driver pass straight through — see their docstrings there), runs_dir + run_id locate this run's artifact directory (runs_dir/run_id), source_name / description are recorded in the report, plan_sources (keyed by declared scenario name — run/cli.py's _load_scenarios) feeds the report's verbatim YAML / per-step line numbers, and config_source — the Git source the config came from (BE-0063), or None for a local config — is stamped into the manifest's provenance so a branch-based run states the exact commit it executed.

Returns:

Type Description
tuple[list[RunResult], Path]

The per-scenario results and the path to the written manifest.json.

run_matrix_and_report(eff, scenarios, engines, run_pass, runs_dir, run_id, *, source_name=None, description=None, plan_sources=None, secret_values=None, config_source=None, exec_provenance=None, label=None, cancelled=not_cancelled, worker_failures=None)

Run the scenarios once per engine, then assemble ONE report at runs_dir/run_id (BE-0076).

The cross-browser fan-out: a loop over engines, each a full pass. run_pass(engine, run_dir) runs the selected scenarios for one engine against its own pool, writing that engine's evidence under run_dir (the caller hands it runs_dir/run_id/<engine>, prefixing the existing NN-slug layout so two engines never collide); its results are tagged with engine here. The passes' tagged results are concatenated into one flat list and written as a single manifest / JUnit / report — the manifest's matrix block aggregates the per-engine verdicts, and ok is all-must-pass across every engine x scenario (pure aggregation, no LLM).

cancelled (BE-0370) is read between passes: each one first builds a whole device_pool, so a cancel during the first engine would otherwise still pay every remaining engine's bring-up and teardown before the run could finish, overrunning the grace period. Every scenario of an engine that never ran is failed as cancelled rather than left out, so the matrix still names every requested engine and ok aggregates a cancelled run to False — dropping those engines instead would let a first pass that happened to be green aggregate to a PASS for a run that never finished.

Returns:

Type Description
tuple[list[RunResult], Path]

The concatenated per-engine results and the path to the written manifest.json.

report_run_matrix(eff, scenarios, results, plan_order, runs_dir, run_id, *, target_effs=None, not_applicable=None, source_name=None, description=None, plan_sources=None, secret_values=None, config_source=None, exec_provenance=None, label=None, worker_failures=None)

Write ONE report for the runs runsOn conditions formed across scenarios (BE-0452).

The caller has already run each run against its own eligible devices and tagged every result with its coordinates; this only assembles the report. plan_order names the scenario each result ran, since a scenario contributes one result per run. not_applicable records the runs no available device fell in an open range for, which the manifest lists apart from the results.

Returns:

Type Description
Path

The path to the written manifest.json.

reroot_evidence(r, prefix)

Prefix r's run-dir-relative evidence paths with <prefix>/, as a matrix pass writes them.

A run formed by runsOn conditions writes its evidence under its own subdirectory of the run dir, as an engine pass of --browsers does (BE-0076, BE-0452).