English · 日本語
Configuration, onboarding a target, and doctor¶
The tool core is app-agnostic. All app-specific differences belong in config, so the same binary and the same drivers run multiple apps. Adding a target means adding one targets.<name> entry.
Implementation: bajutsu/common/config/resolve.py (resolution) · bajutsu/common/doctor.py (convention score). No config ships in the repo root; pass one with --config (default filename bajutsu.config.yaml) — the demos ship ready-to-run configs, e.g. demos/showcase/showcase.config.yaml (iOS) and demos/web/demo.config.yaml (web).
Related: app-agnostic in concepts · drivers · scenarios
Config layering (defaults × targets)¶
bajutsu.config.yaml has three layers. The resolution order is defaults < templates < target <
scenario (the one closer to the test wins). A template is optional. It joins the chain when a
target names one (below).
A target accepts nine keys. Each key groups the settings for one purpose
(BE-0452).
Every target must name its platform. The loader fails on a key outside these nine and lists the
keys a target accepts. It never reads the flat keys of earlier releases, such as bundleId or
appPath.
BE-0473
removed the run group. A target, defaults, and defaults.platforms.<platform> all refuse it.
Each of its fields moved to the group that already answers the same question. A config that still
writes run fails to load. The error names the new location of each field it carries:
| Old key | New location |
|---|---|
run.erase: true |
subject.reinstall: erase |
run.network, run.visualCompare, run.secrets |
evidence.collectNetwork, evidence.visualCompare, evidence.secrets |
run.systemAlertHandling |
one - systemAlert: <prompt> entry with its choice per rule, in hooks.interrupts (below) |
run.tipKitHandling: true |
- tipKit: dismiss in hooks.interrupts |
run.ignoredSettings |
runsOn.ignoredSettings (below) |
The subject group was named app until
BE-0470. A config that
still writes app, in a target, a template, or defaults.platforms.<platform>, fails to load, and
the error names subject. The fields inside the group keep their names, so renaming the key is
the whole edit.
| Key | Purpose | Shape depends on platform |
|---|---|---|
platform |
Which backend drives the target: ios, android, web, or fake |
— |
subject |
What is under test (an app or a site): its identifier, how to build and launch it, readiness, and the id contract | Yes |
runsOn |
Where the target runs: one block per destination, holding device conditions and device state | Yes |
driver |
How Bajutsu drives the target | Yes |
services |
The services a run depends on: the ones it starts and the ones it reaches | No |
hooks |
Steps that wrap every scenario, and what the run clears on its own when it surfaces | No |
evidence |
What a run keeps, how it compares, and what it masks | No |
paths |
Where files live | No |
ai |
Which provider backs the AI paths | No |
bajutsu config schema prints the JavaScript Object Notation (JSON) Schema of this file
(cli). Each target in the schema is a oneOf keyed by platform. An
editor thus completes the keys of the platform a target names.
defaults: # shared across all targets
evidence:
capture: [screenshot.after, elements, actionLog]
redact: { headers: [Authorization, Cookie], fields: [token, password] }
secrets: [LOGIN_PASSWORD] # env var names usable as ${secrets.X} (values masked in evidence)
ai: { provider: api-key, keyEnv: ANTHROPIC_API_KEY } # the AI paths' provider/model/endpoint/key (below)
platforms: # the platform-shaped groups, per platform
ios:
runsOn: { ios: { locale: en_US } }
reservedNamespaces: [auth, nav] # team-wide: the id contract for shared flows / components (informational)
targets:
showcase-swiftui: # ← selected by --target showcase-swiftui
platform: ios
subject:
id: com.bajutsu.showcase.ios.swiftui
path: ios/swiftui/build/Showcase.app # installed before each scenario when present
build: make swiftui-build # builds `path` on demand
launch: { env: { SHOWCASE_UITEST: "1" } }
startWhen: { exists: { id: [stable.row.1, stable_row_1] } }
idNamespaces: [stable, horse, search, log, notice, perm, sys, net]
driver:
runner: { testRunner: ../../BajutsuKit/Runner/build/dd/Build/Products/BajutsuRunner.xctestrun }
hooks:
interrupts:
- systemAlert: notifications # answer the notification prompt; --no-system-alert-handling turns the guard off
choice: grant
- tipKit: dismiss # iOS only: dismiss a blocking TipKit tip (built-in off)
paths: { scenarios: scenarios } # this target's scenarios dir (run reads it; record writes here)
web: # a web target (Playwright backend) is identified by URL
platform: web
subject: { url: "http://127.0.0.1:8787/index.html" }
runsOn:
browser:
kind: chromium # chromium / chrome / edge / firefox / webkit; --browser overrides per run
emulate: iPhone 13 # omitted = a desktop context; else a Playwright device preset
driver: { headless: true } # false = a visible (headed) browser; --headed overrides per run
services:
server: { cmd: "python -m http.server 8787", readyUrl: "http://127.0.0.1:8787/" }
paths: { scenarios: demos/web/scenarios }
The shape of four groups depends on the platform. Each platform registers its own models for them, so a key another platform takes fails as unknown on this one, with the accepted fields listed:
| Group | iOS | Android | Web |
|---|---|---|---|
subject |
id, path, build, reinstall, launch.env, launch.args, launch.deeplink, startWhen, idNamespaces |
id, path, build, reinstall, launch.env, launch.deeplink, grantPermissions, startWhen, idNamespaces |
url, launch.env, startWhen, idNamespaces |
runsOn |
ios.model, ios.os, ios.kind, ios.family, ios.locale, ios.seedPhotos |
android.avd, android.apiLevel, android.locale, android.seedPhotos |
browser.kind, browser.version, browser.emulate |
driver |
runner.testRunner, runner.build, runner.hostAddress |
nativeZ, residentRecovery.readRetries, residentRecovery.restart |
headless |
runsOn also takes ignoredSettings on every platform. That key names no destination. It lists
the settings the target's environment cannot apply
(below).
The test-only fake platform takes an optional subject.id, subject.startWhen, and subject.idNamespaces,
and nothing in runsOn or driver. A fake target resolves to the iOS-shaped sub-config.
The shared groups hold the same fields on every platform:
| Group | Fields |
|---|---|
services |
server (bring up the app's own host for the run, below), mockServer, mailbox (an external inbox the run polls and never starts, below) |
hooks |
setup, cleanup, interrupts (screens, system alerts, and TipKit, below) |
evidence |
capture, redact, collectNetwork, visualCompare, secrets |
paths |
scenarios, baselines, schemas, goldens |
Defaults¶
defaults holds the shared groups directly. The platform-shaped groups sit under
defaults.platforms.<platform>, and apply to that platform's targets alone. One file can hold
defaults for each platform at once. The loader merges defaults into a target before it validates
the target. A field the platform requires, such as subject.id, may therefore come from the defaults.
The merge follows five rules:
- A dictionary merges key by key, and the target's value wins.
- A target clears an inherited value by writing
null. - A list replaces the inherited one, except
evidence.secretsand eachevidence.redactlist, which union. Replacing them would let a target that adds one secret stop masking every team-wide one.hooks.interruptsmerges form by form, since one list holds three forms: the screen entries replace the inherited ones when the target writes any, asystemAlertentry replaces the inherited entry for the same prompt, and atipKitentry replaces the inherited one. A target that adds one screen handler therefore keeps the promptsdefaultsanswers. An emptyhooks.interrupts: []still clears every inherited entry. - A target may not write
nullon those union lists, nor onevidenceorevidence.redact. The load fails and names the path, so a target never wipes the team's masking. aimerges field by field, as described below.
reservedNamespaces and doctor apply to the whole team, so they sit at the top level of the file.
notify stays at the top level too. A target has no notify of its own, so the top-level list
alone says where results go (below).
Sharing a block between targets (x- keys)¶
defaults covers a value every target, or every target of one platform, shares. Two variants of one
app often share more than that, such as most of their subject block. A YAML anchor covers that case.
The loader ignores any top-level key whose name starts with x-, so such a key can host the anchor,
and a target merges it with YAML's << key:
x-swiftui-app: &swiftui-app
launch: { env: { SHOWCASE_UITEST: "1" } }
idNamespaces: [stable, horse, search, log, notice, perm, sys, net]
targets:
showcase-swiftui-noax:
platform: ios
subject:
<<: *swiftui-app
id: com.bajutsu.showcase.ios.swiftui.noax
idNamespaces: [] # a key written here wins over the anchor's
The << key merges one level deep. A nested block written in the target, such as launch,
replaces the anchor's whole launch block rather than merging into it. Place the anchor at the
level of the block that differs, here subject, not at the level of the whole target. The x- prefix
follows the extension-field convention of Docker Compose files. Every other unknown top-level key
still fails to load.
Templates (templates:, BE-0471)¶
A template is a named, partial target block. A target pulls one in with extends. A setting that
targets would otherwise repeat then sits in one place. defaults applies to every target, and
defaults.platforms.<platform> to every target of one platform. A template fills the gap between
those two layers and a single target. It covers, for example, the targets that build one app in
different ways.
templates:
showcase-ios: # any target key, none required
platform: ios
subject:
launch: { env: { SHOWCASE_UITEST: "1" } }
masked-password:
evidence: { redact: { fields: [password] } }
targets:
showcase-swiftui:
extends: [showcase-ios, masked-password] # later entries win over earlier ones
subject: { id: com.bajutsu.showcase.ios.swiftui }
showcase-uikit:
extends: ./templates/uikit.template.yaml # a template kept in its own file
subject: { id: com.bajutsu.showcase.ios.uikit }
A template lives under its own top-level key templates:, never under targets:. --target and
every server endpoint refuse a template's name. --target and most endpoints say the name is a
template. Four endpoints refuse it earlier in their scenarios check: run, run_set, record,
and triage. Their error reads target '<name>' has no scenarios dir. The CLI also lists the
targets. A server error leaves the list out, because one config can hold the targets of other orgs.
A template and a target may share a name, since extends looks up templates alone.
The layers fold from the weakest to the strongest:
defaultsdefaults.platforms.<platform>- each
extendsentry, in listed order - the target itself
A template may extend other templates. The base folds first, so the extending template wins over it. A template that two entries both reach, such as a shared base, folds once, at its first position. A base thus stays weaker than every template that extends it.
Each layer merges under the five rules of Defaults. A template can clear a value
defaults supplies with null. evidence.secrets and the evidence.redact lists union across every
layer. A template's ai folds into the target's ai first. The result then merges with
defaults.ai field by field.
platform may come from a template, so a target can omit the key. Every layer that declares
platform must agree, the target included. The target cannot override a template's platform.
platform decides the shape of subject, runsOn, and driver. A web target must never receive a
template's iOS-shaped blocks. The load fails and names both layers when two of them differ.
An extends entry that contains / or ends in .yaml or .yml names a file. Any other entry
names a template under templates:. A component's use step tells a name from a path the same way
(Scenarios). A template file holds one template block, and its rules follow those of
a component file:
- The path resolves against the directory of the file that holds the
extends. - The file must stay inside the suite root.
- A relative value inside the file resolves against the file's own directory. The value can be a
nested
extendsentry, a path field, or a component itshookscall withuse. The path fields aresubject.path,runsOn.ios.seedPhotos,driver.runner.testRunner, andpaths.*. - A build command (
subject.build,driver.runner.build) is not rewritten. It runs in the config file's directory, as a target's own does. Keep a build command and the path it produces in the same place. Split across files, the build writes to one directory and the path points at another. - The file's
extendsnames other files alone, since the file has notemplates:of its own.
A config read from a document with no file may name inline templates alone.
The load fails on an unknown template name, and lists the known ones. It fails on a cycle too, and
prints the cycle's path (a → b → a). A key outside the eleven target keys and extends also fails, on any template, extended or not.
The loader checks a template's nested fields through the targets that extend it. A template that no
target extends keeps its nested fields unchecked. bajutsu config schema narrows that gap for an
editor. It states each template as a partial target of the template's own platform. A template
that omits platform gets any platform's partial shape.
Device conditions (runsOn)¶
A target's runsOn says where the target runs. Its fields sit in one block named for the
destination: runsOn.ios, runsOn.android, or runsOn.browser
(BE-0469).
Each field thus names what it limits, so runsOn.ios.os reads as an iOS version.
| Platform | Destination block | Fields |
|---|---|---|
| iOS | runsOn.ios |
model, os, kind, family, locale, seedPhotos |
| Android | runsOn.android |
avd, apiLevel, locale, seedPhotos |
| Web | runsOn.browser |
kind, version, emulate |
| fake | none | none |
platform and runsOn are separate settings. platform names the kind of app under test, which
decides the shape of subject and driver. runsOn limits where that app runs. The loader checks
that the pair is one the table allows, rather than that the two names match. A
platform: ios target with runsOn.browser fails and names the destinations iOS allows. A fake
target has no destination, so any key under its runsOn fails. Team defaults hold the same block
under defaults.platforms.<platform>.runsOn.
targets:
app-sim: # a Simulator
platform: ios
subject: { id: com.example.app }
runsOn:
ios: { model: "iPhone 16", os: "18" }
app-device: # a real device, taken from --udid
platform: ios
subject: { id: com.example.app }
runsOn:
ios: { kind: device }
site: # a browser
platform: web
subject: { url: https://example.com }
runsOn:
browser: { kind: firefox, emulate: "iPhone 15" }
Earlier releases wrote these fields flat under runsOn. The loader reads no flat spelling. A flat
key fails the load and names the path it moved to:
| Old key | New key |
|---|---|
runsOn.model, os, kind, locale, seedPhotos |
runsOn.ios.<same field> |
runsOn.avd, apiLevel |
runsOn.android.<same field> |
runsOn.emulate |
runsOn.browser.emulate |
runsOn.browser.engine gave way to runsOn.browser.kind
(BE-0468). An
engine key fails the load and names kind. Renaming the field keeps every run: a kind list runs
once per listed browser, as an engine list did.
Some runsOn fields are device conditions:
| Platform | Conditions |
|---|---|
| iOS | ios.model, ios.os, ios.family, and a list on ios.kind |
| Android | android.avd, android.apiLevel |
| Web | browser.version, and a list on browser.kind |
A list on model or avd counts as a condition too. A version range uses npm's
node-semver range grammar, such as ">=17 <19".
The loader refuses a malformed range. Quote a decimal version: YAML reads 18.10 as 18.1.
Bajutsu reads each available device before the run and matches it against these conditions. See BE-0452. What it reads depends on the platform:
| Condition | Read from | Comparison |
|---|---|---|
iOS model |
the simctl device-type name of the udid | exact |
iOS os |
the Simulator's runtime version | version range |
iOS family |
the productFamily of a Simulator's device type, or the deviceType that devicectl reports for a real device |
iphone or ipad; the first listed family the run's devices have |
Android avd |
the emulator's Android Virtual Device (AVD) name, from ro.boot.qemu.avd_name, then ro.kernel.qemu.avd_name, then adb emu avd name |
exact; a physical device never matches |
Android apiLevel |
ro.build.version.sdk |
version range on an integer |
Web browser.version |
the version of the browser Playwright launches | version range |
The pool is every udid of --udid. An emulator that reports no AVD name fails the run. An avd
condition thus never misses a device without a message.
A run is one scenario on one set of coordinates. Two kinds of condition multiply runs:
| Field | Runs | When no available device fits |
|---|---|---|
a list on kind, model, avd, or browser.kind |
one per listed value | the run fails, naming the value |
a range bounded on both sides, such as ">=17 <19", "18", or "33 - 35" |
one per major version the range covers | the run fails, naming the major version |
a range open on a side, such as ">=18" |
one per major version the available devices have in the range | one record is not applicable |
a single model or avd |
no extra run | the run fails |
family, a single value or a list |
no extra run | the run fails, naming the families |
On Android each API level counts as one major version. A browser.version range never multiplies
runs: it narrows the launched browser instead. A not-applicable record carries no verdict of its
own. Every output lists it apart from the passes and failures (reporting):
- the console and the manifest
- the HTML report's run matrix
- JUnit and Common Test Report Format (CTRF), as a skipped test
- the webhook payload
An invocation where every record is not applicable exits with code 1 and prints no run executed.
A multi-target scenario runs every combination of its targets' runs.
The target below lists two models over a range bounded on both sides. Each of its scenarios makes four runs, one for each model on iOS 17 and on iOS 18:
targets:
app:
platform: ios
subject: { id: com.example.app }
runsOn:
ios:
model: ["iPhone SE (3rd generation)", "iPad Pro 13-inch (M4)"]
os: ">=17 <19"
A scenario declares no condition of its own. A scenario that needs an iPad names a target that runs
on one, such as targets: [app-ipad] (scenarios).
A range on a target thus multiplies every scenario of the target. Take a pool with iOS 17 and iOS 18:
os: ">=17" then runs every scenario twice. For one run per scenario, a target names one major
version, such as os: "18". It may also leave os out.
Each run takes a device from its eligible set. The set keeps the devices that meet the run's
conditions. When the run fixes no model, the set then keeps the first model in pool order with
enough devices at one release. Within that model, it keeps the newest release that has enough. Every member shares one
model and one release, so --workers spreads runs over identical devices.
A target can name a device family instead of a model. Apple renames device types with each Xcode
release. An exact model then stops matching on a host whose Xcode lacks that type. family takes
iphone, ipad, or a priority list of both, and a list never multiplies runs. Each run takes the
first listed family that its devices have. The order in the list alone decides the winner. The same
pool thus always resolves the same way. The eligible set then narrows to the winning family. A
winner with too few devices fails the run, which never falls back to the next family. A list of two
or more families adds the winner to the run's coordinates (family=ipad). The run label, the
manifest, and the flakiness history then keep the runs on each family apart. family and model
both name the device a run takes, so declaring both fails the load.
A list on kind runs each scenario once per listed kind. kind: [simulator, device] runs it on a
Simulator and on a real device. Each run adds its kind to the coordinates (kind=device). The run
then drives the XCUITest destination of that kind. The pool must hold both kinds. Bajutsu reads each
udid of --udid through the listing that shows it:
simctlfor a Simulatordevicectlfor a real device
A udid that neither listing shows fails the read. A listed kind that the pool lacks fails its run. The message names the kind, so no run goes missing without a word.
targets:
app:
platform: ios
subject: { id: com.example.app }
runsOn:
ios:
kind: [simulator, device] # one run on each
family: [ipad, iphone] # each run takes an iPad when its pool has one
record, crawl, and repl never multiply runs. Each takes the first listed value, the first
listed family a fitting device has, and the newest major version the pool has within the range. Each
exits 2 when no device fits. doctor lists the
target's own runs, and each scenario's runs when --scenario names a file, with the devices that
fit each one (doctor output). Its screen probe
scores the device of the first run. --backend fake matches no condition, and says so once. A
target whose runsOn.ios.kind includes device may not declare model or os. Bajutsu does not
compare a real device's model or release. A real device does report its family, so
family stays allowed there. A target an appium grid serves has no pool to read, and neither does
a Device Farm job. A condition there fails before any device work and names the field.
Three runsOn.ios fields are device state rather than conditions. A single kind (simulator or
device) selects the XCUITest destination. locale pins the language a run applies. iOS keeps the built-in
en_US when locale is unset. seedPhotos lists the images seeded into the photo library. A
scenario may replace locale and seedPhotos under preconditions.device.ios. On Android,
runsOn.android.locale and runsOn.android.seedPhotos are device state in the same way, and a
scenario may replace either under preconditions.device.android. The web block's
emulate likewise selects a device mode rather than matching a device.
Android app locale (runsOn.android.locale, BE-0478)¶
runsOn.android.locale sets the language of the app under test. It takes the spelling of
runsOn.ios.locale, such as ja_JP. The run converts it to a Best Current Practice 47 (BCP 47)
language tag (ja-JP). Before every launch, the run sets the tag with cmd locale set-app-locales
and reads it back. A mismatch fails the start and names the package and the locale.
targets:
shop-android:
platform: android
subject: { id: com.example.shop }
runsOn:
android: { apiLevel: ">=33", locale: ja_JP }
The per-app language differs from the iOS locale in three ways:
- Unset keeps the device's language. iOS falls back to a built-in
en_US; Android sets none. An earlier run may have left a per-app language on the reused emulator. On API level 33 and later, the run clears it, so the app follows the device's own language. - It needs API level 33.
cmd localeexists from API level 33. Two checks enforce that: - When
runsOn.android.apiLeveladmits an earlier level, the run fails before any device work. The message suggests narrowingapiLevel. - When
apiLevelis unset, the run reads the device's level at start and fails there.
A run that sets no locale merely logs a warning when it cannot read the level.
- System dialogs keep the emulator's language. The per-app language reaches the app alone. A
runtime permission prompt stays in the language of the emulator image. iOS, by contrast, also pins
the Simulator's system language. Address a system dialog's buttons with an id selector, because
their text follows the image.
Resolution (resolve → Effective)¶
resolve(config, target) builds the effective values Effective (a frozen dataclass) for one target.
An undefined target raises KeyError (the CLI exits with code 2). The platform-specific knobs below
— each platform's own identifier plus the web-only headless / browser / device_mode — are not
flat Effective attributes: Effective.platform_config narrows to one of three mutually exclusive
sub-configs (IosConfig / WebConfig / AndroidConfig) keyed by the target's platform
(BE-0126).
Read them through bajutsu/common/config/accessors.py's narrowing helpers — require_ios / require_web /
require_android for code already committed to one platform, or the "soft" ios_bundle_id /
web_base_url / web_engine / android_package for code that reads a value defensively across
platforms — rather than effective.bundle_id directly, which does not exist.
Effective field |
Source | Notes |
|---|---|---|
platform |
platform |
the target's platform (ios/android/web); a fake target reads as ios, since it resolves to the iOS-shaped sub-config |
backend |
platform |
the target's platform as a one-entry list, from which the actuator is selected (drivers); --backend may replace it, but only with the same platform, its actuator, or fake |
platform_config.bundle_id (IosConfig) |
subject.id |
iOS target identifier |
platform_config.base_url (WebConfig) |
subject.url |
web target URL (Playwright backend) |
platform_config.package (AndroidConfig) |
subject.id |
Android target identifier |
platform_config.app_path / .build (IosConfig, AndroidConfig) |
subject.path / subject.build |
the built app a run installs before each scenario (if missing), and the shell command that builds it on demand |
platform_config.xcuitest (IosConfig) |
driver.runner + runsOn.ios.kind |
the XCUITest runner: testRunner (a prebuilt .xctestrun), build (the command that builds it), hostAddress (below), and the destination kind |
platform_config.grant_permissions (AndroidConfig) |
subject.grantPermissions |
Android only: runtime permissions granted before launch (pm grant), so a permission prompt never blocks a scenario |
platform_config.native_z (AndroidConfig) |
driver.nativeZ |
Android only: nativeZ: true asks the device for each opted-in view's own View.getZ(), which elements.json reports as the diagnostic nativeZ field (BE-0355). Off by default, because the node walk that answers it runs on every screen read — 20-100 ms — and reports nothing at all unless the app under test links BajutsuAndroid and opts views in. A target that turns it on also gives up the round trip a confirmed gesture's own reply would otherwise have saved: that reply cannot carry a nativeZ reading, so such a target asks the device not to build the tree at all rather than take one whose positions would all read absent (BE-0407 units 18-19). iOS needs no such flag: its responder answers only for an app that linked BajutsuKit |
platform_config.resident_read_retries, platform_config.resident_restart (AndroidConfig) |
driver.residentRecovery.readRetries, driver.residentRecovery.restart |
Android only: what a timed-out read through the resident UI Automator server tries before the lease falls back to uiautomator dump. readRetries (0 to 5, default 1) repeats the read on a fresh connection. restart (default true) then restarts the server, and it also allows a later non-ASCII type step to start a lost server again. A refused or closed connection takes neither route, because it shows that the server cannot answer |
platform_config.headless (WebConfig) |
driver.headless |
web only: true (default) runs headless; false shows a visible (headed) browser, in slow-motion. bajutsu run --headed / --no-headed and the Web UI's "show browser" toggle override per run |
platform_config.browser (WebConfig) |
runsOn.browser.kind |
web only: the browser to drive — chromium (default), chrome, edge, firefox, or webkit. chrome and edge launch the Google Chrome or Microsoft Edge installed on the host, through Playwright's chromium engine on a channel; webkit stands in for Safari, which Playwright cannot drive. bajutsu run/record --browser <kind> overrides per run (flag > config > default), and bajutsu run --browsers <list> runs the cross-browser matrix (below). A missing Playwright engine is installed on demand; a missing Chrome or Edge fails the run, because installing one would change the host's own browser. An unknown value is rejected at config load (BE-0468) (BE-0076) |
platform_config.device_mode (WebConfig) |
runsOn.browser.emulate |
web only: the device mode a browser context is created with — omitted for a desktop context, or a Playwright device preset name (e.g. iPhone 13) that emulates its viewport / touch / device scale / user agent, driving the web target as that mobile device. It is desktop-browser emulation (Chrome DevTools' device toolbar), not a real device (drivers → Playwright). Resolved lazily against playwright.devices in the driver, so config load never imports Playwright; an unknown preset fails loudly at driver start, not at config load (BE-0228) |
conditions |
runsOn |
the device conditions the target declares (above), ranges as written; run forms its runs from them, and doctor lists those runs |
device_provider |
--worker-config |
where this target's devices come from. The config no longer names it: an appium environment in the worker.yaml the command runs under hands the grid's endpoint to the targets of the grid's platform (what a machine can run), and every other target takes the built-in local provider (the --udid path). Bajutsu drives that endpoint end to end over a live W3C WebDriver transport (iOS device cloud, BE-0236, BE-0238) |
launch_server |
services.server |
optional {cmd, readyUrl, readyTimeout, cwd, env} — bring up the app's host for the run, then tear it down: probe readyUrl (default subject.url), reuse it if already serving, else run cmd and wait until ready (a condition wait, never a fixed sleep). The web analogue of subject.build (BE-0059). For an uploaded bundle in serve, the host never runs cmd directly — serve --upload-exec governs it (see self-hosting); a sandbox run needs the extra fields dockerImage (a Docker image reference, e.g. node:20-slim) or dockerfile (a bundle-relative path built with docker build) — exactly one — plus port (the in-container listen port, published to a loopback host port) (BE-0090) |
run_defaults.system_alerts / .ios_tip_kit_handling |
the systemAlert and tipKit entries of hooks.interrupts |
per-app defaults for the system-alert guard and the TipKit guard, which a scenario's own interrupts layers over (BE-0177, BE-0473). A systemAlert entry answers its prompt even inside a scenario that declares entries for other prompts; the run prints a notice when it does, and the scenario overrides it with its own entry for that prompt. The layering rules are below and in scenarios. tipKit: dismiss defaults the TipKit guard (iOS only, built-in off); --ios-tipkit-handling/--no-ios-tipkit-handling override it for every scenario |
run_defaults.setup / .cleanup / .interrupts |
hooks.setup / hooks.cleanup / hooks.interrupts |
steps that wrap every scenario of the target: setup runs ahead of a scenario's own setup, cleanup behind its own cleanup (a rule takes on: always, success, or failure, the words a scenario's cleanup rule uses), and interrupts are checked before the scenario's. A hook step may call a component with use: { component: <file>, with: … }: the loader resolves it when the config loads, against the config file's directory, and refuses a reference that leaves the suite root (BE-0174). group: and setMocks stay rejected there. A scenario skips every declared target's hooks.setup with preconditions.inheritSetup: false; the cleanup still runs. The systemAlert and tipKit entries of hooks.interrupts resolve into the two fields of the row above, so .interrupts holds the screen entries alone |
locale |
runsOn.ios.locale |
iOS: applied at launch (simctl launch args), built-in en_US. On the XCUITest backend it also pins the Simulator's own system language on every cold spawn — written to the device's global preference domain, then rebooted so SpringBoard renders it — so an out-of-process permission prompt's button text is the one this locale predicts rather than whatever language the Simulator happened to carry; a scenario whose preconditions.device.ios.locale differs from the pinned one forces a cold respawn rather than reusing a warm runner (BE-0320). A handleSystemAlert step under a language outside the label table (English and Japanese) answers by the button's position instead; a systemAlert entry in interrupts under such a language still fails the scenario before any device work (BE-0445) |
launch_env / launch_args |
subject.launch.env / subject.launch.args |
a scenario's preconditions.subject.launch.env merges over the environment key by key, and its .args append after the arguments; launch.args is iOS only |
reinstall |
subject.reinstall |
iOS and Android: how much state is reset before each scenario, as one ladder from weakest to strongest. overwrite installs over the app and keeps its data; clean uninstalls, then installs; erase wipes the whole device, then installs (on Android, which has no device-wide wipe, the same as clean). The install itself needs subject.path. A scenario's preconditions.subject.reinstall wins; unset on both, the built-in clean applies (BE-0452). bajutsu run --erase raises every scenario to erase, and --no-erase lowers an erase to clean. erase replaces the removed run.erase: true, which now fails at load naming this key; in defaults, which has no subject group, write it as defaults.platforms.<platform>.subject.reinstall: erase (BE-0473) |
deeplink |
subject.launch.deeplink |
iOS and Android: a URL the app opens once it has launched; a scenario's preconditions.subject.launch.deeplink replaces it |
seed_photos |
runsOn.ios.seedPhotos / runsOn.android.seedPhotos |
Images seeded before launch, in order, so a photo picker shows a known set. Paths resolve against the config file, and a scenario's preconditions.device.<platform>.seedPhotos replaces the list. iOS seeds the Simulator's photo library on the erase path, so selectPhotos addresses a known library; after merging, a non-empty list needs an effective subject.reinstall: erase (the scenario's, else the target's, then moved by --erase / --no-erase), or the run exits 2 before any device work. Android has no device-wide wipe, so it seeds a directory of its own instead (/storage/emulated/0/Pictures/Bajutsu): each scenario empties that directory, pushes the images with ascending modification times from 2037-01-01 (so the set sorts ahead of the device's own images), and asks the media scanner to rescan it. Android therefore needs no erase, and leaves every other image on the device alone. An empty list still empties a directory an earlier seed created. The media store prefers an image's EXIF capture date to its modification time, so the order holds for images without one. The scan finishes after launch, so a scenario waits for a seeded image before it selects one |
start_when |
subject.startWhen |
optional startWhen: { exists: { id: … } } — a selector the launch waits for before the run starts, instead of the default "the app rendered any 2+ elements". It takes the form an interrupts entry uses for its condition, without negate. Use it for an app whose first interactive screen is a modal over always-present chrome (the element-count heuristic can return before the modal presents). Its id / idMatches accept an OR candidate list like a scenario selector (startWhen: { exists: { id: [stable.row.1, stable_row_1] } }, BE-0221), so one startWhen covers a target whose native id syntax differs. A condition wait, never a fixed sleep. Set it only when every scenario for the target starts on that same screen; when first screens vary per scenario, lead each scenario with a wait step instead. startWhen stays the strongest readiness signal: on iOS, a target linking BajutsuKit's screen-transition observer (BE-0310) gets a reported-screen-change rung above the namespace/count heuristics for free, but an explicit startWhen still outranks it. The observer signal governs only when no startWhen is set |
id_namespaces |
subject.idNamespaces |
referenced by doctor |
reserved_namespaces |
top-level reservedNamespaces |
informational (doctor scores against the app's idNamespaces only) |
mock_server |
services.mockServer |
⚠️ schema only · not wired |
evidence_dirs.scenarios |
paths.scenarios |
this target's scenarios dir — run --target loads every *.yaml here; record writes new ones here. Relative to the config file's own directory (like subject.path and the sibling paths.baselines / .schemas / .goldens), so the config behaves the same wherever bajutsu runs from (BE-0242). run --scenario / record --out override it |
capture |
evidence.capture |
the default evidence (the note in evidence) |
redact |
evidence.redact (defaults ∪ target) |
merged (below) |
run_defaults.network |
evidence.collectNetwork |
whether the run collects the app's network exchanges for request assertions; resolves --network/--no-network > this > built-in on |
visual_compare |
evidence.visualCompare |
how a visual assertion compares when it names no compare of its own, exact or pixelmatch; built-in exact |
secrets |
evidence.secrets (defaults ∪ target) |
env var names declaring ${secrets.X}; values are masked in evidence (evidence) |
ai |
ai (defaults < target, field by field) |
the AI paths' provider/model/endpoint/key (below); None (omitted) = the environment alone decides |
doctor_thresholds |
top-level doctor.idCoverageOk / doctor.idCoverageFail |
id-coverage thresholds for doctor grading (below); default 0.9 / 0.7 |
Merging redact¶
defaults.evidence.redact and targets.<name>.evidence.redact are unioned list by list
(labels, headers, fields, and unmaskHeaders each). The scenario's redact
(evidence) layers on top.
Secrets (evidence.secrets)¶
evidence.secrets (a list of environment-variable names, declared in defaults.evidence and/or
targets.<name>.evidence, unioned when the loader merges them) is the declaration site for the
${secrets.X} variables a scenario can input.
At run time bajutsu run resolves each declared name from the environment, interpolates its value
into the action (${secrets.X}), and masks the literal value everywhere it would appear in
evidence (evidence). The scenario source keeps the ${secrets.X}
token, never the value.
System alerts and TipKit in hooks.interrupts¶
hooks.interrupts lists what a run clears on its own when it surfaces mid-scenario. One list holds
three forms of entry
(BE-0473).
The key an entry writes decides its form, so a reader never infers the form from a value's type:
| Form | Required keys | Optional keys | What it clears |
|---|---|---|---|
| Screen | condition |
steps |
An interstitial screen in the app's own tree (scenarios) |
| System alert | systemAlert, choice |
pollInterval, frozenScreenTimeout |
An operating-system prompt, answered by the system-alert guard (scenarios) |
| TipKit | tipKit |
none | Apple's TipKit popover, on an iOS target alone (scenarios) |
targets:
showcase-swiftui:
platform: ios
subject: { id: com.bajutsu.showcase.ios.swiftui }
hooks:
interrupts:
- condition: { exists: { id: [aerr_wait] } } # an in-app screen
steps:
- tap: { id: [aerr_close] }
- systemAlert: notifications # an OS prompt
choice: grant
- systemAlert: savePassword
choice: deny
frozenScreenTimeout: 20 # tuning that belongs to this prompt alone
- tipKit: dismiss # Apple's TipKit popover
The values each form takes:
systemAlertnames one of five prompts:notifications,tracking,paste,savePassword, orlocalNetwork.choiceisgrant,deny, orleave.leaveanswers nothing for that prompt.pollIntervalsets how often, in seconds, the guard checks SpringBoard while a wait is pending. The built-in value is one second.frozenScreenTimeoutsets how long, in seconds, a guarded wait tolerates a frozen screen before it stops.0turns the stop off, and the built-in value is 10 seconds. The loader accepts the key on a prompt raised into the app's own tree alone, which today meanssavePassword. On any other prompt the load fails, because nothing would read the value.tipKitisdismissorleave. AtipKitentry on a target whose platform is not iOS fails at load.
The loader refuses three mistakes, each with an error that names it. An entry that mixes keys of
two forms fails and names both forms. Two entries for the same prompt fail, and so do two tipKit
entries, in one list. A target-level entry may not set target, and its steps may not use group
or setMocks.
The guard itself has no on/off key in the config. The guard runs unless bajutsu run gets
--no-system-alert-handling. A target that used to write systemAlertHandling: false now writes
choice: leave on each prompt it leaves. The config no longer takes visionInstruction either.
run never reads a model, and record and crawl take --alert-vision-instruction instead.
A scenario's own interrupts takes the same three forms and layers over the target's entries:
| Form | How the target's and the scenario's entries combine |
|---|---|
| Screen | Concatenated, the target's entries first |
| System alert | Per prompt: the scenario's entry replaces the target's choice and inherits an omitted frozenScreenTimeout from the target's entry for that prompt; the other prompts from both apply |
| TipKit | The scenario's entry, else the target's; absent from both means off |
pollInterval resolves by layer instead: the scenario, then --alert-poll-interval, then the
target. Within one layer, the smallest value wins. A shorter interval satisfies every entry that
asked for a longer one. The guard has one frozen-screen bound. That bound is the most tolerant
value among the entries that answer a prompt: the largest, with 0 (the stop off) over any value.
Scenarios walks through each rule.
Environment-specific settings (runsOn.ignoredSettings, BE-0467)¶
Some settings reach the device before the first step, and not every environment can apply them. The
iOS Simulator wipes a device with simctl erase, which a real iPhone or a WebDriver endpoint cannot
run; a browser has no app launch to pass subject.launch.env to. One table per environment route (the
iOS Simulator, an iOS real device, a live WebDriver endpoint, Android, and the web) decides how each
setting is handled:
| Support | Meaning | Run behavior |
|---|---|---|
supported |
The route applies the setting | Apply it |
satisfied |
The route already guarantees what the setting asks for (a web run's fresh browser context is already erased) | Do nothing |
unsupported |
The route cannot apply the setting | Fail before any device work, unless the target declares it ignored |
The settings, each with the field it comes from:
erase:subject.reinstall: eraseinstall: thesubject.pathinstall (subject.reinstallis its mode)permissions.<service>: each service in the scenario'spermissionsseedPhotos:runsOn.ios.seedPhotosorrunsOn.android.seedPhotoslocale:runsOn.ios.localeorrunsOn.android.localelaunchEnv,launchArgs,deeplink:subject.launch.env,subject.launch.args,subject.launch.deeplink
A setting the route cannot apply fails the scenario in the
run preflight, which names the setting, the target, and the declaration that would ignore it. Each
environment refuses an unsupported setting again as it launches, so a lease driven directly, as
record or crawl takes one, cannot apply it either.
The one opt-out is the target's runsOn.ignoredSettings: the setting key, and the reason the run goes
without it. The map sits in runsOn beside the destination blocks, because it describes the same
environment. The key was run.ignoredSettings until
BE-0473
removed the run group.
targets:
iphone-device:
platform: ios
subject: { id: com.example.app }
runsOn:
ios: { kind: device }
ignoredSettings:
erase: "A real device keeps app data; this lane accepts state from earlier runs"
An ignored setting is not applied. The run's reports list it with its target and reason, beside the
verdict and never part of it: ignored_settings in each scenario of manifest.json, an "ignored
settings" table in the HTML report's Environment panel, a bajutsu.ignoredSettings property on the
JUnit testcase, and ignoredSettings in the CTRF test's extra (reporting). The
launch also logs a warning naming it — the only trace for a lease that writes no report, such as
record or crawl. The map is checked when the config loads: an unknown key or a
blank reason fails, and defaults.platforms.<platform>.runsOn may not carry it, because an ignored
setting belongs to one environment. A declaration for a setting the target's route supports or satisfies fails the run
preflight, so a stale declaration cannot hide a regression once the route gains the setting. A
scenario never declares a setting ignored; it stays one definition of the test for every environment.
The same table answers in two more places, both before any device work. With --config and
--target, bajutsu lint runs the settings check beside the step check, with no device
(cli). serve routes each job by the settings it puts in effect. A job thus reaches a
worker whose route can apply them, and no other
(self-hosting).
Device runner signing (outside the config, BE-0456)¶
A real iOS device needs an XCUITest runner signed by the user's own Apple Developer team. The target config carries none of that signing by design. The config travels with the repository, so a team ID written there would reach every other user. A per-user signing file holds those settings instead, outside the repository.
bajutsu runner build --device reads that file. A target with runsOn.ios.kind: device and no
driver.runner.testRunner then resolves to the build.
iOS on a real device gives the file format and its lookup
order.
What a machine can run (outside the config, BE-0450)¶
The config describes targets, and an application team writes it. What one machine can run is a
different fact, which the machine's operator knows. A separate file, worker.yaml, states that
fact. The file names the worker's environment and the drivers it offers. It also caps the targets
one scenario may hold. The config never carries those keys. A server worker reads the config after
it leases a job. That point comes too late to decide what the worker advertises. bajutsu run and
bajutsu worker both name the file with --worker-config <path>. Without the flag, the shipped
default applies.
What this machine can run gives the file format and
the check.
Where a run happens is that same kind of fact, so it moved out of the target too
(BE-0452). An
Appium grid that serves one platform's devices is an appium environment in worker.yaml:
version: 1
environment: appium
appium:
endpoint: https://grid.example.com/wd/hub # the grid's Appium / WebDriver address
platform: ios # the one value today
drivers: [xcuitest, adb, playwright, fake]
Each command that drives a device takes --worker-config with such a file: run, record,
crawl, repl, audit, doctor, and bajutsu mcp. The targets of the grid's platform take the
endpoint as their device. The other targets of the same scenario stay on this machine. The endpoint
counts as one device, so the run refuses a scenario with two grid targets. --udid no longer
accepts a URL. A Device Farm fan-out names no batch provider, and no target carries the former
cloudBatch. The server routes each fan-out job by its target's platform (Device Farm).
Real-device host address (driver.runner.hostAddress)¶
A real iOS device does not share the host's loopback. An app on one reaches the network collector on
a host address instead. By default, each run offers the app every routable address of the host's
active interfaces. The app keeps the first one that answers. Some host interfaces can be out of the
device's reach. driver.runner.hostAddress narrows the list to the addresses it names,
comma-separated:
targets:
my-app:
platform: ios
subject: { id: com.example.app }
runsOn: { ios: { kind: device } }
driver:
runner: { hostAddress: 192.168.1.20 }
The BAJUTSU_HOST_ADDRESS environment variable overrides the key. A host whose address changes with
each job can set it per run, without editing the config. The key has no effect on the Simulator.
iOS on a real device describes the
exchange.
AI provider (ai:, BE-0047)¶
The AI paths — record, crawl, and triage --ai — reach the model through
one provider configured by an optional ai block, declared in defaults and/or targets.<name>
and merged field by field (the target's value wins per field). The block resolves into
Effective.ai, so the CLI and serve agree on one source of truth. This is the enforcement behind
"your AI, your key, your data": every AI path runs under the key and endpoint you configure, and the
deterministic run gate still calls no model at all
(BE-0047).
defaults:
ai:
provider: api-key # a registered provider name; api-key (default), bedrock, ant, claude-code, or none (disables every AI path)
model: claude-opus-4-8 # optional: override the path's default model (or the BAJUTSU_AI_MODEL env)
effort: high # optional: reasoning effort — low/medium/high/xhigh/max (or BAJUTSU_AI_EFFORT); claude-code
language: auto # optional: AI output language for the generated prose — ja/en/auto (or BAJUTSU_AI_LANGUAGE)
baseUrl: https://ai-gateway.internal/v1 # optional: a self-hosted gateway / enterprise proxy (anthropic provider)
keyEnv: ANTHROPIC_API_KEY # the NAME of the env var holding the key — never the key itself
- Model and effort are config-first with an env fallback.
model(orBAJUTSU_AI_MODEL) overrides the default model on any provider.effort(orBAJUTSU_AI_EFFORT) sets the reasoning effort, one oflow/medium/high/xhigh/max. Theclaude-codeprovider honors it and passes it to the CLI as--effort. An unrecognizedeffortfrom config or the env var falls back to the model's default. TheserveSettings panel instead validates its input and rejects an unknown value (HTTP 400) rather than falling back. The panel exposes both fields.
On local serve, the saved provider, model, and effort persist to a serve-owned file, and the
next start restores them
(BE-0184).
A restart no longer resets them to the launch environment. A config ai: block still wins over
a restored value.
On a hosted, multi-tenant serve, the selection resolves and persists per organization
(BE-0229).
Each org's record / triage / draft paths therefore use that org's own saved choice. The selection
reaches a spawned job as a per-job environment overlay rather than the shared process environment,
so one org's save never changes another org's AI runs.
record prints the resolved choice up front, as 🤖 AI: <provider> · model <model> · effort <effort>.
-
Output language is a separate, config-first knob (BE-0188).
language(orBAJUTSU_AI_LANGUAGE) fixes the language the AI writes its own generated prose in:record'sfrom:provenance andcrawl's streamed reasoning. It is one ofja/en/auto, andauto(the default) keeps today's behavior —recordfollows the goal's language andcrawlstays English. Set it per invocation with--languageonrecord/crawl(flag > config >auto), or from theserveSettings panel's Output language dropdown; an unrecognized value falls back toauto(the panel instead rejects it with HTTP 400). This governs authoring and investigation prose only — never the deterministicrunverdict — and is distinct from a target's devicelocalebelow, which sets the app/UI language rather than the AI's. -
A provider is a backend behind one interface (BE-0104). The AI paths reach a model only through a vendor-neutral seam (
bajutsu/ai), mirroring how a platform is a backend behind theDriverinterface.provideris therefore an open, registry-validated value, not a fixed set:api-key,bedrock,ant, andclaude-codeare the adapters that ship today. The first three share one Anthropic adapter — the name states the auth method: a direct API key, AWS credentials for Bedrock, or theantCLI's OAuth token, BE-0163; the legacy nameanthropicstill resolves toapi-key.claude-code(BE-0176) is a separate adapter that shells out to the localclaudeCLI. An unknown name fails closed with a clear error the first time an AI path resolves the provider. (The check lives in the AI layer, not at config load: the deterministic core must not import the AI provider stack (BE-0112), so config accepts the name and the registry that owns the valid names rejects an unregistered one.) Adding a model family (e.g. an OpenAI-compatible endpoint) is registering an adapter, and it inherits the redaction and fail-closed guarantees below by construction. One further name,none, is a switch rather than a vendor. The two bullets below cover it. nonestates that no AI path may run (BE-0394). Thenoneprovider's adapter does nothing at all, so writingprovider: nonedisables every AI path at once.record/crawl/triage --aiexit 2 naming the setting, and no code path can construct an AI backend, because the provider's factory raises instead of returning one.runis outside its reach in both directions: since BE-0402 it has no AI path to disable.
Leaving the provider's key out of the environment already produces most of the same behavior. The
setting adds a policy a reviewer can read. An absent key records the intent nowhere in the
repository, and a key exported to author one scenario with record is enough to put the authoring
paths on a model in every later shell. Because config wins over the environment, a committed
provider: none survives BAJUTSU_AI_PROVIDER on a continuous-integration runner whose
environment nobody controls. BAJUTSU_AI_PROVIDER=none still works for a one-off run on a machine
whose config names no provider.
This provider changes nothing about the defaults / targets.<name> merge rule. A repository can
disable AI in defaults and re-enable it for one target, on a line a reviewer reads in the same
file, rather than an environment variable nobody sees.
- serve never offers none in its Settings dropdown. The registry knows the provider but
never lists it as selectable, so /api/provider answers HTTP 400 for it. An organization's
Settings selection reaches a spawned job as BAJUTSU_AI_PROVIDER, which a project's own
ai.provider outranks. So a none picked there would have no effect at all on a repository whose
config names a provider: the jobs would keep calling the model behind a switch labeled off. One
limitation follows, and this design accepts it rather than closing it. The settings endpoint
resolves Claude reachability from the organization's saved selection rather than the target's ai
block, so a repository that sets provider: none still reports claudeAvailable: true and the
web UI leaves the record and crawl tabs enabled. The switch itself holds — a job started from one
of those tabs resolves the repository configuration itself and exits 2 before reaching a model —
but the pre-flight signal that would grey the tab out first is missing.
- Keys never live in config. keyEnv names an environment variable; the value is read from the
environment at call time, so a secret never lands in the repo or an uploaded bundle. baseUrl
points the Anthropic SDK at a self-hosted gateway / proxy (Anthropic(base_url=…,
api_key=os.environ[keyEnv])), so your screenshots and element trees only ever reach the endpoint
you set, never a vendor default. Bedrock keeps the standard AWS credential chain (AWS_REGION +
env / shared profile / instance or task role) and needs a provider-prefixed model.
- ant bills a subscription/SSO seat, no API key (BE-0163). The ant provider reaches the model
through the official Anthropic CLI: install it and
run ant auth login (a browser-based OAuth/SSO sign-in against the Claude Console). Bajutsu reads a
bearer token from the CLI at call time and passes it to the SDK as auth_token (rather than an API
key), so a Claude Pro/Max/Console seat is billed. ANTHROPIC_PROFILE selects a named CLI profile;
no API key is needed, and every AI path (authoring, the alert guard, triage --ai) keeps full
vision. ant is an external binary you install yourself — Bajutsu neither vendors nor installs it.
- claude-code bills a Claude Code subscription via the local CLI (BE-0176). The claude-code
provider shells out to the claude CLI (claude -p,
print mode) instead of the Anthropic SDK, so authoring / investigation draw on the Claude Code
Pro / Max / Console seat you already have signed in (claude setup-token, or an interactive
login). Every AI path keeps full vision: each screenshot is written to a per-call scratch file
whose path the prompt names, and the CLI is allowed only Read (scoped to that directory) to view
it — every other tool is denied and permission prompts fail closed, since on-screen text is
untrusted input
(BE-0125).
Any ANTHROPIC_API_KEY is stripped from the CLI call so billing stays on the subscription rather
than the API. claude is an external binary you install yourself — Bajutsu neither vendors nor
installs it. On a headless host — a CI runner, a container, a remote serve — that can't run
claude setup-token's interactive browser flow, mint the long-lived token once on a machine that
can and set CLAUDE_CODE_OAUTH_TOKEN (in your shell, a .env, or serve's Settings panel, which
holds it write-once alongside the API key)
(BE-0215);
the claude CLI reads it from the environment, so no interactive login is needed there.
- Config first, environment fallback. Any field you omit falls back to today's environment
variables — BAJUTSU_AI_PROVIDER, ANTHROPIC_API_KEY, BAJUTSU_BEDROCK_MODEL (the ant provider
reads its credential from the CLI, honoring ANTHROPIC_PROFILE) — so a config with no ai block
behaves exactly as before.
- Fail closed. record, crawl, and triage --ai exit with a clear, provider-specific error
when the selected provider has no usable credential — they never construct a client that quietly
falls back to a hosted default. run asks about no credential at all (BE-0402).
- The textual inputs are redacted; screenshots cannot be. The element trees, failure text, and
the (possibly user-supplied) alert instruction sent to the model are scrubbed by the same run-scoped
redaction as written evidence (the target's redact keys + resolved secret values). Screenshots
are images and redaction masks text, not pixels — so the second guarantee carries them: every
input, screenshots included, goes only to the provider/endpoint you configured.
- On-screen secrets stay in the pixels (BE-0151). Because images cannot be masked, a secret the
app displays — a typed password, an OTP, PII on screen — stays verbatim in the raw pixels of the
screenshot the AI sees: the live screen every turn during record, and the captured failure
screenshot (if any) during triage --ai, read from the run's runs/ evidence. That image goes to
the AI provider you configured. Redaction covers the ${secrets.X}
value wherever it appears in text (network, element tree, logs), not what the app renders on
screen. So that the exposure is never a surprise, record and triage --ai print a one-time warning when
the target binds secrets:. This warning is a disclosure, not a mitigation (visual evidence is the point):
to avoid the exposure entirely, skip AI-driven authoring for a secret-bearing flow, or keep the
secret off-screen in the app under test.
- Usage and cost are recorded to an attributed ledger
(BE-0196). Every AI call
appends one line to a JSON Lines (JSONL) ledger tagged with what its tokens were spent on (command,
provider, model, scenario) and priced in dollars where the provider has per-token pricing. It is
reporting only — recording is best-effort and never touches the deterministic run verdict. Two
optional fields under ai tune it:
defaults:
ai:
usageLedger: runs/usage.jsonl # optional: ledger path (default runs/usage.jsonl; "" disables)
pricing: # optional: override the shipped per-token rates (USD per million tokens)
api-key/sonnet: { input: 3.0, output: 15.0, cacheWrite: 3.75, cacheRead: 0.3 }
usageLedger sets the JSONL path — the default is runs/usage.jsonl (under the gitignored runs/
tree), and an explicit empty string turns persistence off. pricing overrides the shipped default
rate table, keyed by "provider/model" (the model part matches a model id by family, e.g.
api-key/sonnet prices any claude-sonnet-*); a subscription provider with no per-token price
(ant, claude-code) records the token counts with a null cost rather than a fabricated dollar
figure. Like the textual inputs above, the ledger stores counts, prices, and labels only — never
prompt or response content.
Mailbox (the email step)¶
targets.<name>.services.mailbox configures the generic HTTP mailbox the email
step polls for a 2FA / verification code, so the endpoint and credentials live in config (not the
scenario):
targets:
myapp:
platform: ios
subject: { id: com.example.myapp }
services:
mailbox:
kind: http # transport adapter; defaults to http when omitted
url: "${secrets.MAILBOX_URL}" # inbox endpoint (GET); ${secrets.*} resolved at run time
headers: { Authorization: "Bearer ${secrets.MAILBOX_TOKEN}" }
# Optional response mapping, to read any provider's JSON without per-provider code:
messages: "items" # dotted path to the message array (default: the response is the array)
fields: { to: to, subject: subject, body: text, receivedAt: receivedAt, id: id }
The defaults match the common shape (an array of messages with to / subject / body /
receivedAt / id), so a conforming API needs no messages / fields mapping. The email step
reads the inbox over HTTP, keeps only messages newer than the step's start (keyed on id), waits
for one that matches, and extracts the code — deterministic and LLM-free
(BE-0046).
kind selects the transport adapter behind the mailbox — a mailbox is a backend behind one
interface, keyed by transport (http, later imap) rather than by vendor, so adding a transport
registers an adapter instead of branching the runner
(BE-0186). It
is optional and defaults to http, so an existing mailbox: block is unchanged; an unknown kind
fails the run with a clean config error rather than falling back. Only http ships today — it keys
on transport, not on the mail vendor, because vendors differ only in JSON field names, which
fields already absorbs.
Webhook notifications (notify:, BE-0099)¶
notify: is a top-level list of webhook endpoints bajutsu run posts to as a post-verdict side
effect — a Slack-first delivery path with no LLM and no way to affect the deterministic verdict
(BE-0099):
notify:
- format: slack # renderer; slack is the only one shipped today
url: "${secrets.SLACK_WEBHOOK_URL}" # webhook URL; ${secrets.*} resolved at run time
on: [failure] # failure (default) / change / recovery / always / start
targets: [] # optional: only these scenario names; empty = every scenario
onselects which events fire this endpoint:failure(any scenario failed, the default),always(every run),change/recovery(the run's overall verdict flipped since the previous run of the same config source, read from that run'smanifest.jsonunder the runs dir), andstart(fired once, before the run starts, with its own message — an endpoint whoseonis[start]alone fires only there, never post-verdict).targets(unrelated to the top-level configtargets.<name>map) narrows the notification to scenarios whose name is in the list; empty (the default) covers every scenario in the run.- A target accepts no
notifyof its own (BE-0473). Every target's run posts to the top-level list; atargets.<name>.notifyfails to load and names that list. - Delivery is best-effort: a bounded timeout, a couple of retries, and a failure only logs a warning
— it can never flip the verdict or exit code, the same after-the-verdict discipline
--zipand--evidence-storefollow. Onlyformat: slackrenders today (a Block Kit message, listing up to five failing scenarios with a "…and N more" tail); an unrecognizedformat, or aurlwith an unresolved${secrets.*}token, is skipped with a logged warning rather than posting a broken payload.
Orgs (orgs:, the multi-tenant server backend)¶
orgs: declares tenants for the hosted server backend (BE-0015).
Each org lists its members — explicit GitHub logins (members), whole GitHub orgs (githubOrgs),
and/or single GitHub Teams (githubTeams) — the GitHub Teams whose members may write
(editorTeams),
and the targets it owns:
orgs:
acme:
members: [alice, bob] # explicit GitHub logins
githubOrgs: [acme-gh] # everyone in this GitHub org (needs the read:org OAuth scope)
githubTeams: [acme-gh/qa] # direct members of these Teams, without the whole GitHub org
editorTeams: [acme-gh/scenario-maintainers] # direct members of these Teams become editors — and may sign in
allowedRepositories: [acme/app] # CI jobs in these repositories may act as this org
targets: [demo, checkout]
allowedRepositories is the org's machine roster, beside the human ones
(BE-0414): the
repositories whose continuous-integration (CI) jobs may exchange an OpenID Connect (OIDC) token for
a short-lived session acting as this org. It grants no role — a pipeline is governed by its own
endpoint allowlist, never by viewer / editor / admin. An entry is the bare "<owner>/<repo>" name,
or an object carrying that name plus an environment, ref, or workflowRef bound that narrows it
further. Setting it up — the audience an operator must configure, the exchange endpoint, and the
name-recycling and fork hazards — is covered in
Self-hosting.
At OAuth login users are assigned their org — an explicit members entry first, then a githubOrgs
match from their GitHub org memberships, then a githubTeams or editorTeams match from their direct
Team memberships. Teams rank last, so adding one to an org never moves a login that a members or
githubOrgs entry already placed. Afterward they see only that org's targets, and a run's
artifacts/scenarios/baselines live under the org's own object-store prefix. A target named in no org
falls into the single default org, so a config without an orgs: block is single-tenant — the
CLI and local serve ignore orgs: entirely.
Once GitHub OAuth is configured, org membership also decides access
(BE-0313). Signing in
requires membership in a configured org, through members, githubOrgs, githubTeams, or
editorTeams. That membership grants the viewer role. A member of a configured admin Team signs
in regardless (below). The roles above viewer come from Team membership:
- A direct member of any of the org's
editorTeamsis promoted to editor. - A member of one of the server-wide admin Teams (
BAJUTSU_OAUTH_ADMIN_TEAMS) is admin. Self-hosting describes the setting.
editorTeams admits as well as promotes, so a Team that may write never has to be repeated under
githubTeams to be able to sign in. editorTeams is a list because one org may span more than one
GitHub organization, and a single slot could not then name the writing Team of each. A configuration
still on the older singular editorTeam keeps working: serve folds that key into editorTeams,
and folds both in when a partial rename leaves the singular name behind. Rename it anyway; the plural
name is the documented one.
Each githubTeams entry, each editorTeams entry, and each BAJUTSU_OAUTH_ADMIN_TEAMS entry is one
flat Team, written as "<github-org>/<team-slug>". A nested Team beneath any of them does not match.
All three are compared case-insensitively, as GitHub itself resolves an org login and a Team slug. A
Team-declared org depends on GitHub's Teams API answering. That API fails closed and never invents a
Team, so while it errors, a login whose only membership is a Team is turned away rather than
admitted.
An OAuth deployment must declare an orgs: block, or every login other than an admin Team
member is turned away. A member of a configured admin Team can still sign in unless GitHub's Teams
API is itself erroring, so a broken or missing orgs: block never locks every admin out on its own.
An admin admitted only by their Team is placed in the default org, since no orgs: entry claims
them. A deployment relying on that recovery should avoid declaring a real org named default. If it
does, the recovering admin's user row, audit entries, and object-storage prefix land inside that
tenant instead of a neutral catch-all.
A deployment with a database reads five of these six fields only once
(BE-0375).
On the one boot that finds the orgs table still empty, serve copies each org's members,
githubOrgs, githubTeams, editorTeams, and allowedRepositories into it from the
configuration this server was launched with;
every sign-in after that resolves against the database alone. That copy happens once for the life of
the deployment: a boot that finds any org already there — a retired one included — copies nothing,
so no later configuration edit, and no restart carrying one, can add or reshape a tenant behind an
admin's back. A configuration bound afterwards through the web UI or POST /api/config never
copies at all, whatever its orgs: block says.
An admin edits the membership from the Orgs page from then on, and an edit to those five fields
here has no effect: serve records a warning naming the org whose entry still declares them, so an
operator learns the file stopped deciding rather than watching an edit vanish.
allowedRepositories is the one of the five the Orgs page does not show — it is edited through
POST /api/orgs/<slug>/membership instead, and a save from that page leaves it untouched rather
than clearing it. targets is the
field that keeps working, so an entry pared down to targets: alone is the expected end state on
such a deployment. Paring an entry down before that first boot is safe too, since the order is not
yours to get wrong: an entry declaring only targets is skipped rather than copied, so it never
locks an org at "admits nobody".
Two orgs may each claim a target of the same name, and each is authorized for it; under a single
bound configuration they share the one targets: definition that name resolves to.
A configuration bound through the API — an uploaded bundle, a composed triple, or a Git source —
has its orgs: block ignored for target ownership entirely. It was bound as an org, so every
target it declares belongs to that org and to no other, whatever the block says. Reading ownership
out of a file the deployment does not control is the same trust problem that keeps such a file from
seeding membership, and it failed quietly: a bundle whose orgs: claimed its only target for an org
you are not in left you with an empty target list and nothing explaining why. Leave orgs: out of
an uploaded bundle — it decides nothing there. A deployment
with no database keeps reading every field from this file, none of the above applying to it.
Selecting from the CLI¶
Every command in the CLI (command-line interface) selects one app with --target <name> and points at
config with --config (default bajutsu.config.yaml). --backend checks rather than overrides
the target's platform: it accepts that platform, its actuator (such as xcuitest), or fake, and
any other value, or a comma list, exits with code 2 (cli).
BE-0452
removed the ordered fallback list, because a target now names one platform.
Cross-browser matrix (--browsers, BE-0076)¶
bajutsu run --browsers chromium,firefox,webkit runs the selected scenarios once per engine and
emits a single engine × scenario pass/fail matrix — the multi-engine spelling of the --browser
axis (web backend only; --browsers chromium is exactly --browser chromium, and a single engine
takes the ordinary single-engine path). The run is green only if every requested engine passes
every scenario (all-must-pass); a scenario green on Chromium and Firefox but red on WebKit is a
machine-detected rendering-engine incompatibility — the kind of "works in Chrome, broken in Safari"
bug a single-engine test can never see. The verdict is purely the existing deterministic per-engine
run outcomes aggregated; no AI enters it.
Each engine is a full pass against its own browser pool, so its evidence lands under
runs/<id>/<engine>/<NN-scenario>/ (no collisions between engines). The run then assembles one
manifest.json, junit.xml, and report.html at the run root: the manifest carries a matrix
block aggregating the per-engine verdicts, the report renders the engine × scenario grid, and JUnit
keys the engine into each case (classname="bajutsu.<engine>") so CI sees chromium.login and
webkit.login as distinct cases (reporting). An unknown engine in the
list exits 2 before any browser launches, the same as --browser. All three engines run headless on
Linux, so the matrix runs inside the ordinary gate with no Mac or device farm; the firefox/webkit
binaries are installed on demand.
Config from a Git repository (BE-0063)¶
--config also accepts a Git source, so a command can run a test repository's suite without a
local checkout — bajutsu run --config github:acme/mobile-tests@v1.4.0:e2e/bajutsu.config.yaml --target checkout:
github:<owner>/<repo>[@<ref>][:<path>] # GitHub shorthand
git+https://<host>/<owner>/<repo>.git[@<ref>][#<path>] # general form (host reserved)
- GitHub is the only host implemented today. The general
git+https://<host>/…form is parsed (the door is open for GitHub Enterprise / GitLab later), but a non-github.comhost currently fails with a clear error rather than silently hitting github.com. - A run from a Git source records the resolved commit in its
manifest.jsonprovenance (configSource: { host, owner, repo, ref, sha }), so a branch-based run states the exact commit it executed and is reproducible after the fact (reporting). <ref>is a branch, tag, or commit SHA (default: the repo's default branch);<path>is the config within the repo (default:bajutsu.config.yamlat the root). A value with no recognized scheme is a local path, exactly as before.- Bajutsu resolves the ref to an immutable commit SHA, materializes that subtree into a
content-addressed cache (
~/.cache/bajutsu/gitsrc/<host>/<owner>/<repo>/<sha>/), and loads the config from it. The config's relativepaths.scenarios/paths.baselines/paths.schemas/subject.pathresolve against the checkout root — the same "relative to where the config lives" rule a local config follows against its own directory, except the anchor is the fetched tree's root — so the whole tree comes along, not just the YAML. A fetched config is untrusted, so its paths are also confined to the checkout: an absolute or../-escaping value is refused. A local file, being operator-trusted, resolves against the config file's own directory and is not confined (it may point at a sibling). - A fresh checkout holds no built binary, and there is no local "first" in which to build one, so a
Git-sourced
runbuilds the app on demand: whensubject.pathis set but missing, it runs the config'ssubject.buildcommand from the checkout root (where the command's relative parts, e.g.make -C demos/showcase swiftui-build, are rooted), then proceeds. A failed build exits cleanly. A local-pathrunis unchanged (it never builds; a missing binary still errors). - A pinned commit SHA (
@<sha>) is reproducible and runs offline after the first fetch; a branch (or tag) is resolved fresh each load. - A private repository needs a credential
(BE-0224).
The token is resolved per fetch (so a rotated secret needs no restart), in this order: a
configured GitHub App installation (
BAJUTSU_GITHUB_APP_IDplus a private key), then a serve-entered credential (BAJUTSU_GIT_CONFIG_TOKEN), thenGITHUB_TOKEN/GH_TOKEN, thengh auth token, else anonymous. It is never logged. Grant least privilege: prefer a fine-grained personal access token (PAT) — or an App installation — scoped to just the target repositories with the Contents: read permission, over a classic broad-repoPAT that grants read/write to every private repo. An unattended, self-hostedserveshould authenticate as a GitHub App (a short-lived, per-installation token tied to the service, not a person) — see self-hosting → private-repository access. When access is missing, the fetch fails with a message that names the real cause — a rate limit, an organization single sign-on (SSO) authorization gap, a rejected token, or "provide a credential with Contents: read for<owner>/<repo>" — rather than a bare 404. bajutsu runtakes two gate switches:--config-offlineuses the cache and never touches the network (it needs a pinned@<sha>, since a branch can't be resolved offline), and--require-pinned-configfails unless the Git config pins a commit SHA — a branch or even a tag can move under a gate, so only a SHA is accepted.- The serve UI also binds a Git source —
serve --config github:…at startup, or the "From a Git repository" field in the "Open config" dialog — materializing the checkout and serving from its root (cli → serve). For a private repository the dialog has a credential field (BE-0224): enter a fine-grained PAT or App token and it is stored write-once through serve's secret store — masked, never echoed back (held in the process environment on a local serve; encrypted per organization on the hosted backend). A missing-access diagnostic is shown inline in the dialog. - Remaining follow-ups: read-only Git input for
record/crawl(an authored artifact goes to a local--out, never into the SHA-keyed cache).
Onboarding a new target¶
To add a new app, add app-side preparation and one config entry. The tool itself needs no changes.
- Apply the implementation convention —
accessibilityIdentifieron key elements (in the app's namespace), expose state in label / traits / value, launch hooks, disable animations. - Add
targets.<name>—platformandsubject.id(both required;subject.urlfor a web target), plussubject.launch.env,subject.idNamespaces, and the rest as needed. - (Optional) a shared setup — factor login etc. into a component file and call it from the
target's
hooks.setupwithuse: { component: <file> }, so it runs before every scenario's own steps; a scenario opts out withpreconditions.inheritSetup: false. - Verify with
bajutsu doctor --target <name>— look at the convention score (below). - Place scenarios — write identifiers in the app's namespace.
Identifier naming convention¶
accessibilityIdentifier is dot-separated <namespace>.<element>. All lowercase, each segment
[a-z0-9-]. The first segment is the namespace, one of the set declared in idNamespaces.
settings.reindex # <namespace=settings>.<element=reindex>
home.search
list.row.<id> # dynamic rows: the suffix is a "data-derived stable key" (index-based is forbidden)
Three invariants:
- Unique within a screen — never put the same id twice on one screen
(ambiguity detection in selectors). Repeated elements are
disambiguated by a data-derived key (
list.row.3). Set operations useidMatches+count. - Non-localized, data-derived — do not use display text in an id (it breaks under translation).
- Namespace-prefixed — every id starts with a declared namespace.
The showcase's id catalog is in showcase (and, in full, demos/showcase/SPEC.md).
doctor (the convention score)¶
Implementation: bajutsu/common/doctor.py. AI-independent and deterministic. It analyzes one screen's
query() (the CLI uses the screen obtained via the actuator) and produces a score.
doctorruns a runnability gate first (preflight.py), then the score. The gate checks what the chosen backend needs: the iOS (XCUITest) backend needsxcodebuild/xcrunplus a booted Simulator; the web (Playwright) backend needs the Playwright package and its Chromium browser (uv sync --extra web+playwright install chromium). It then scores the current screen: for a web target it navigates a fresh browser to the target'ssubject.urland scores that page; for iOS it scores the screen on the booted Simulator. The score still covers only the currently displayed screen (entry / current screen, not all screens).
Metrics (Score)¶
Measured over actionable elements (trait ∈ ACTIONABLE_TRAITS = button / link / textField /
searchField / textView / switch / slider / tab / cell).
On the web backend, doctor skips elements read from a cross-origin iframe. The app's developer
cannot give ids to an embedded video player or chat. Those controls neither lower idCoverage nor
count as duplicate ids. Same-origin iframes belong to the app, and doctor scores them.
| Metric | Definition | Threshold (default) |
|---|---|---|
idCoverage |
fraction of actionable elements with an id | ✓ ≥ 0.9 / warn 0.7–0.9 / fail < 0.7 |
namespaceConformance |
fraction of ids whose first segment is in idNamespaces |
off-convention ids listed in off_namespace |
duplicateIds |
number of duplicate ids on one screen | Blocked if any |
Grading¶
- Blocked: no actionable elements on the screen (most likely blank, not yet loaded, or the
wrong screen —
rendersays so), any duplicate id, oridCoverage<idCoverageFail(default 0.7). - Ready:
idCoverage≥idCoverageOk(default 0.9) andnamespaceConformance== 1.0. - Partial: otherwise (runnable, but a forecast of coordinate fallback / flakiness).
Configurable thresholds (doctor, BE-0024)¶
The id-coverage thresholds that determine the grade are configurable in the top-level doctor
block. Teams
with many decorative elements that legitimately lack test IDs can tune the thresholds for leniency
(typically lowering idCoverageOk and/or idCoverageFail) without changing the tool:
doctor:
idCoverageOk: 0.85 # default 0.9 — coverage >= this is eligible for "Ready"
idCoverageFail: 0.6 # default 0.7 — coverage < this drops to "Blocked"
Both values must be in [0, 1] and idCoverageOk must be >= idCoverageFail; an invalid value is
rejected at config load. When omitted, the hardcoded defaults (0.9 / 0.7) apply — existing configs
are unchanged.
Output¶
render(score) returns a human-readable summary. Missing elements are listed concretely so you
can see exactly where to add an id:
grade: Partial
idCoverage: 0.83 (5/6)
namespaceConformance: 1.00
duplicateIds: 0
missing id: label='Close' traits=['button'] frame=(...)
The CLI's doctor exits with code 1 when the grade is Blocked (cli).
A device condition adds a runs: section to the CLI's output. The
section reads the --udid pool the way run does, and forms the same runs. Without --scenario
it lists the target's own runs; with a scenario file, each scenario's. It lists the devices that fit
each run. A target with no pool to read lists its declared runs alone. A physical device
and a grid target both lack a pool. The section is informational and never changes the exit code.
The score itself covers the device of the target's first run, the one record takes.