Skip to content

English · 日本語

The evidence subsystem

Evidence capture for a recurring action is expressed as a repeatedly-firing rule rather than a one-shot instruction. The rule collects the same evidence on every later run, without AI.

Implementation: bajutsu/common/evidence/core.py (instant captures and Sinks) · bajutsu/common/evidence/intervals.py (interval captures: video / deviceLog / appTrace). The orchestrator decides when a rule fires (run-loop).

Related: the capture tokens in scenarios · reporting


Three ways to request evidence

Way Use Example
A. Rules (capturePolicy) ★ central automatic capture every time a particular action happens network exchanges on every tap of settings.*
B. Per-step (capture:) this one step only video + deviceLog around a specific wait
C. Default policy a baseline guarantee config's evidence: { capture: [screenshot.after, elements, actionLog] }

The config default (C), evidence.capture, resolves to Effective.capture (configuration). It applies on top of every step, alongside the scenario's capturePolicy and the per-step capture. Unlike those two, it fires unconditionally rather than on a trigger, so it acts as a baseline guarantee, not a rule.

All three ways request evidence on top of what every step already records on both sides of its action. Naming screenshot.after or elements in any of them therefore changes nothing, and leaving either out costs nothing. Whatever the three ask for, the run captures before.png, after.png, and the post-action elements.json (below).

Evidence kinds and acquisition timing

A capture: token is <kind>[.<modifier>] (scenarios).

Kind Source Interval / instant Status
screenshot the driver (XCUITest's own /screenshot endpoint, adb's screencap, Playwright natively) instant ✅ captured
elements (a11y / accessibility tree) driver.query() as JSON instant ✅ captured
actionLog orchestrator internals (action · duration) plus each driver's own actuation records — ✅ inherent in the manifest
video simctl io recordVideo interval ✅ captured (needs udid)
deviceLog simctl spawn log stream interval ✅ captured (needs udid)
network the in-app collector (BajutsuKit → network.json) interval ✅ captured (the --network run flag)
appTrace simctl spawn log stream over the app's os_log subsystem (iOS); adb logcat filtered to the app's tag (Android) interval ✅ captured (needs a device + subsystem)
rawTree the device's own reply behind elements, untouched (base.RawSourceProvider; adb and XCUITest today) instant ✅ captured (opt-in, no-op elsewhere)

appTrace pairs the app's <name> started / <name> finished log markers into timed intervals. On iOS the markers come from os_signpost / os_log (intervals.parse_app_trace). On Android the app logs them with android.util.Log, as in Log.i(tag, "checkout started"). intervals.parse_logcat_app_trace pairs those (BE-0478). network is produced by the request collector rather than the interval system — its exchanges are written to <sid>/network.json (network observation, the --network flag).

Every entry in elements.json carries the element's identifier, label, traits, value, and frame. One further field is diagnostic: nativeZ, the element's real front-to-back position as the app under test measured it (BE-0355). Reading it changes nothing a run decides — no selector matches on nativeZ, and the occlusion checks (is_tappable, topmost_at_point, and XCUITest's own isHittable) behave exactly as they did before the field existed. On the web backend, an element read from a cross-origin iframe also carries crossOriginFrame: true (drivers); every other element omits the field.

What the number means is the backend's own, and only its direction carries across. On both backends a larger nativeZ is closer to the viewer. Nothing else about two values is comparable unless they came from the same backend.

  • iOS reports an ordinal over the app's real compositing order, counted back to front. CALayer.zPosition reads zero across an ordinary flat layout, and Apple documents it as the wrong tool for sibling order.
  • Android reports View.getZ() (elevation plus any translation on the z axis) in device pixels. That value orders siblings within one parent and nothing wider: a child at 0 under a parent at 8 still composites in front of that parent's sibling at 4.

So on Android, a number tells a reader which of two elements is in front only when they are siblings. On iOS, any two values from the same read compare, since the ordinal spans the whole screen. No comparison holds across backends.

A null is the common case, and it is deliberate rather than a gap to be filled in by inference. Reporting a real value needs an app that opted in:

  • iOS needs two things. The app links BajutsuKit, and bajutsu drives the app through its own XCUITest runner. The WebDriverAgent-backed live/record path injects no port for the responder to answer on, so it reads null even for an opted-in app.
  • Android needs the app to call BajutsuZOrder.report(view) in a debug build.

Two toolkits report nothing even then, because each generates its own accessibility elements and does not expose the underlying one the position would be measured from. SwiftUI materializes its accessibility elements only for an assistive technology attached to the process, so an app looking at its own view tree finds no identifiers on it. Jetpack Compose forwards no app-declared extra-data key through its own node generation. UIKit and Android View screens report; SwiftUI and Compose screens read null.

The element list's own order is the paint-order proxy that topmost_at_point falls back to. Deriving a position from it would read as authoritative while being wrong on exactly the layouts an investigator opens the evidence for. One such layout is an Android view whose elevation lifts it above a sibling declared after it.

Inside an iOS app: block, every element reads null. The responder runs in the test target. While another app owns the screen, the test target sits suspended in the background, so its positions would describe a tree the block is not reading. Requests queued to a suspended app were measured to kill it with SIGPIPE once it resumed (the Home Screen widget spike). The driver skips the round trip until the block ends.

On Android the target must also ask for it, with driver: { nativeZ: true } under its targets.<name> entry. The reading comes from a walk the resident server performs over every node on every screen read. That walk costs 20-100 milliseconds whether or not the app opted a single view in (BE-0407 unit 18). An app that calls BajutsuZOrder.report(view) and does not set the flag reads null, the same honest absence every other unreported case gives.

iOS needs no such flag. Its responder answers only when the app links BajutsuKit, so a target that did not opt in pays nothing to ask.

rawTree writes hierarchy.raw<suffix>, the device's/runner's own reply, untouched by any of bajutsu's processing. For adb that is the uiautomator dump/resident XML (.xml); for XCUITest it is the undecoded GET /elements body (.json).

On adb's resident channel, when narrowing changed something, rawTree also writes hierarchy.parsed-input.xml: what parse_hierarchy actually consumed, after SystemUI decor windows were stripped. XCUITest applies no such transform, so it never writes a second file.

The capture exists to diagnose a mismatch between a resolved coordinate and the real screen. It shows whether the device's/runner's own reply already looked wrong, or whether bajutsu's own parsing changed it. It is never in the default capture list; a scenario opts in with capture: [rawTree, ...].

One redaction rule refuses it outright: when redact.labels is configured, rawTree writes nothing for the whole run and logs why. redact.labels masks a labeled element's value structurally. elements.json is written from the parsed tree, so the writer knows which value to blank. The raw dump is free text with no such structure, so it would ship an unmasked superset of what elements.json just masked.

Every other redaction rule (headers, fields, resolved secret values) applies to the dump as free text and leaves rawTree enabled. One caveat applies to redact.headers and redact.fields. Their key-pattern masking is written for multi-line logs, where a matched value ends at the next newline, but a UI Automator dump is emitted as a single line. A configured key that happens to match text inside the dump itself (an on-screen label or content-desc reading like Token: ...) therefore masks everything after that match to the end of the file, not just the matched value. The dump still ships, just truncated. A resolved secret value (bound via ${secrets.*}) is unaffected, since it is masked by matching a known literal rather than a key pattern.

actionLog — what each step actually did to the screen

actionLog needs no capture request and writes no file of its own: every step's outcome carries an actuations list in manifest.json, one entry per primitive the driver performed, and the report and the bajutsu trace timeline read it from there. It answers the question a screenshot and an element tree cannot: where did this tap land, and how far did this swipe travel.

Field Meaning
gesture the driver primitive — tap, doubleTap, longPress, swipe, scroll, pinch, rotate, the text primitives, selectOption, setPickerValue, systemAlert, back, navigate, openUrl
via how the gesture reached its target: coordinate (the driver computed a point and sent it), handle (XCUITest actuated a snapshot handle), identity (the Android device resolved the element and chose the point), bridge (a WebView call addressed by element id), focused (a text primitive on whatever field holds focus), key, history, url (a page load by navigate, or a URL handed to the operating system by openUrl; the URL itself is not recorded)
unit the coordinate space: point (iOS), pixel (Android), cssPixel (a browser page, or a WebView's own space)
points the coordinates the driver sent, in order — one for a tap, two for a drag's start and end. A two-finger gesture records the single anchor its two contacts were derived from, not the contacts
frame · target the resolved element's bounds and its accessibility identifier
accepted whether the platform accepted this attempt, on the two channels that answer (XCUITest's handle actuation, Android's device-side endpoint). A refused attempt is shown struck through, so a stale-retried tap does not read as several taps; None means the channel gave no separate answer
substitution why the element actuated is not the one the driver's default rule would have named — soleHittableDescendant when a refused tap was redirected to the one reachable named descendant inside its frame. Both readers show it: the report as a badge beside via, the trace timeline as ↷<token>. Absent on the ordinary path, and on every run recorded before schemaVersion 7, which reads the same way: no substitution happened
duration_s · scale · radians the gesture's non-positional parameters, where it has any
at the absolute wall-clock instant (epoch seconds) the driver sent the gesture, on the same footing as the step's started_at. A step stamps started_at before it screenshots, reads the tree, and resolves its target, so the recording changes only at at — up to seconds later on Android. The report highlights a step row from this instant, and both readers show the lag after the step's start: the report beside the gesture, the trace timeline as +0.85s. Absent on every run recorded before schemaVersion 13

Three rules bound what a record may say, and every backend honors them:

  • Only a coordinate that was really sent. points is empty whenever no coordinate crossed to the platform (a handle-based iOS tap, an Android device-side gesture), because the far side chose the point. The record shows the resolved frame instead. It does not present the frame's centre as a measurement it never took.
  • No device work. Every value is one the actuator already had, so recording costs no extra query, read, or round trip.
  • No authored string, ever. The writer of manifest.json runs no redactor, so the record carries none of the following:
  • a type step's text, not even its length (Redactor uses a fixed-width placeholder, so no artifact discloses a secret's length)
  • a selectOption's option
  • a setPickerValue's value
  • a navigate or openUrl step's URL
  • an element's accessibility label

target is always the resolved accessibility identifier and nothing else. An element with no identifier leaves it unset.

A record is written when the gesture is attempted, before its transport answers, so a step that failed to actuate still shows what it aimed at. The step's own result says whether the step worked.

One actuation belongs to no step. The reactive system-alert guard also acts before the scenario-level expect re-check. Its record lands on the scenario's expect_actuations, beside expect_alerts. A backend that does not implement the record contributes none, and the run does not change.

The driver's log is bounded, so a pathological step (a maxScrolls in the hundreds) can lose its earliest records. A report that loads a manifest back can lose a damaged record the same way. dropped_actuations counts both kinds, so the list never reads as complete when it is not.

The Android backend also records what happened to its read channel, because a run there can pass on a different read path from the one it started on. Nothing on the verdict path reads these fields. A step's warnings name a recovery, a loss, or a return during that step. A scenario's channel_degraded says why reads were on the uiautomator dump path when the scenario ended, and it stays empty while the channel is healthy. A scenario's channel_notes hold what happened after the last step. A run recorded before schemaVersion 18 carries neither scenario field, which reads as healthy.

Default modifiers: the always-on instant baseline (below) is before — captured before the step acts, not after. A capturePolicy rule or inline capture: still defaults an unmodified instant kind to after when it fires; interval kinds (video/deviceLog) default to around (start before the action, stop after the step). Stating screenshot.before explicitly on a rule/inline capture is redundant with the baseline and is dropped rather than re-taken.

A. capturePolicy (rule-based)

Repeatedly-firing rules, written per scenario (implementation: scenario/models/evidence.py CaptureRule / Trigger).

capturePolicy:
  # On every tap of settings.*, also capture the network exchanges — screenshot and elements are
  # already guaranteed on every step by config's default policy (C, above)
  - on: { action: tap, idMatches: "settings.*" }
    capture: [network]

  # On every screen transition
  - on: { event: screenChanged }
    capture: [screenshot.around, elements]

  # On error in any step, capture the maximum (the safety net)
  - on: { result: failure }
    capture: [screenshot, video, deviceLog, elements, actionLog]

The trigger on is exactly one of action / event / result:

  • action: <tap|longPress|type|swipe|...> — optionally combined with idMatches (glob against the primary target's id). idMatches can only be used with action.
  • event: screenChanged — fires if query() changed during that step.
  • result: failure — fires if the step failed (the safety net).

The detailed firing logic is in run-loop.

Preview firing before a run (BE-0028). A loose glob or a screenChanged rule can fire on far more steps than intended, and attaching a heavy capture (video / deviceLog / appTrace / network) to such a rule quietly produces gigabytes. bajutsu trace --explain <scenario.yaml> is a read-only dry run. It counts how many times each rule would fire (and on which steps) and flags ⚠ a heavy capture on a broadly-matching rule, so a reader can tighten the match before paying for it. See cli.

B. Inline evidence

To capture just one step, attach capture: directly to the step.

- tap: { id: settings.reindex }
- wait: { for: { id: settings.reindexComplete }, timeout: 5 }
  capture: [video, deviceLog]     # record the interval of this wait

(real example in demos/showcase/scenarios/evidence.yaml)

Interval evidence (video / deviceLog / appTrace)

Implementation: bajutsu/common/evidence/intervals.py. These are subprocess child processes — simctl on iOS, adb on Android — started before the action and stopped after the step settles. Process spawning is injectable (Spawn) and testable. Web has no subprocess: its intervals are Playwright-native and supplied by the driver (see below). On iOS, appTrace is a log stream over the app's os_log subsystem. parse_app_trace pairs its markers into timed intervals.

On Android, appTrace streams adb logcat -s <tag>:V, and the app logs its markers under that tag. The tag is --log-subsystem when set, else the app's package. The stream follows the tag rather than a process id, so a relaunch leaves the stream running. The stream starts at an instant read from the device's own clock (logcat -T), to the second. The device thus decides which lines came earlier. A cutoff on the computer running Bajutsu would compare device times with its own clock. Any clock drift would then drop real lines. A line logged up to a second before the stream can still appear. A later started of the same name replaces such a line when pairing. When Bajutsu cannot read the device clock, the stream starts from the last logged line with a warning. On stop, parse_logcat_app_trace reads each line's time and pairs the markers into appTrace.json.

Interval kinds are opt-in (BE-0028). video / deviceLog / appTrace are heavy, so a scenario records an interval only when it asks for that kind — through an inline capture: or a capturePolicy rule (e.g. a result: failure rule that captures video). A scenario that requests none records none, keeping the common case cheap; the lightweight instant baseline (screenshot + elements) is always captured, so a failure still leaves evidence (DESIGN §10).

Every step captures a screenshot on both sides of its action, but a tree only after it. before.png shows the screen the step is about to act on. Once the step has acted, after.png and an elements write show what the action left behind (BE-0407). Neither screenshot depends on the capture list, so narrowing that list never removes either of a step's two screenshots.

When nothing has actuated the device since the previous step's after.png, before.png can be a byte-for-byte copy of it rather than a fresh capture. Four cases skip the copy and always shoot fresh, because an asynchronous interstitial could have arrived between the two steps:

  • an interrupt's recovery step
  • a handleSystemAlert step
  • any step in a scenario declaring interrupts
  • any step running under the system-alert guard

The guard is on by default, so a stock run shoots fresh on every step. The copy reaches a step only where --no-system-alert-handling turns the guard off.

elements.json has a single filename, so the post-action write is the tree a run keeps. It describes the screen the action produced, which is the screen after.png shows and the one every viewer draws element frames from. On a non-mutating step (assert, wait), that tree is the one the step itself settled on, reused rather than re-read (BE-0259), so it comes from a moment just before the screenshot rather than just after.

Preview what a scenario would record with bajutsu trace --explain (see cli).

Kind Start command (iOS / Android) Stop signal Filename
video simctl io <udid> recordVideo --codec h264 / adb shell screenrecord SIGINT (a hard kill would corrupt the mp4) scenario.mp4
deviceLog simctl spawn <udid> log stream --level debug --style compact [--predicate ...] / adb logcat -b main,system,crash,events -T 1 SIGTERM device.log
  • start_video / start_device_log (iOS) and start_screenrecord / start_logcat (Android) return an Interval, and Interval.stop() sends the signal and finalizes the file.
  • deviceLog waits up to 10s, then sends a kill.
  • video gets a generous 120s window to finish writing before the kill. recordVideo / screenrecord still has to flush and mux the whole clip to disk. A premature kill truncates the mp4 (no moov atom) and, on iOS, wedges the simulator's recording session.
  • screenrecord records device-side. On stop, its Interval also pulls the finished mp4 off the device and removes the device copy.
  • If that pull fails (the device vanished), the sink drops that one artifact with a warning. It does not emit a path with no file behind it, and it does not fail an otherwise-passing scenario.
  • adb screenrecord caps a single recording at ~180s. That is the platform default/maximum, not a limit bajutsu tunes. An Android video of a longer scenario ends at that mark.
  • On iOS, --predicate (NSPredicate) narrows deviceLog to a subsystem, for example (the CLI's --log-predicate).
  • adb logcat has no tag or priority filter. A logcat filterspec is a different syntax, a later knob.
  • adb logcat starts the follow from the tail, so it reflects the scenario window, not the whole ring buffer.
  • The buffer set widens past bare logcat's default (main,system,crash) to add events. An app's own uncaught exception lands in crash. A process that ActivityManager ends for memory pressure logs a structured am_kill / am_low_memory entry in events alone. Without events, that cause looks like a silent, uncaptured fault.
  • The kernel's own out-of-memory (OOM) / low memory killer (LMK) path lands in the kernel ring buffer instead. logcat -b kernel reaches that buffer where logd bridges /proc/kmsg (ro.logd.kernel, typically userdebug builds) and nowhere else. The set here leaves the buffer out, though it is not out of reach.
  • INTERVAL_KINDS = {"video", "deviceLog", "appTrace"}. The orchestrator uses this set to split "interval / instant."
  • The scenario-wide video begins before the app launches on Android, so the recording spans the app's cold start rather than missing it. There, the environment's start starts recording (after the device is booted and the app installed, but before am start) and hands the running Interval back through prestarted_intervals. At scenario start, the sink adopts that interval (intervals.adopt) instead of starting a fresh one. On stop, it finalizes the interval and relocates the file to scenario.mp4 (Android's screenrecord writes real ISO base media).

Web wires the same up-front capture into the browser context at creation, but finalizes to scenario.webm instead. Playwright's recorder always writes Matroska/WebM, so its artifact is named after that real container rather than borrowing Android/iOS's .mp4 (see PlaywrightDriver.video_extension in drivers.md).

XCUITest, the current iOS backend, records on demand instead. Nothing starts a recording before the xcodebuild runner spawns and launches the app, so its prestarted_intervals is always empty.

The flag records_video_up_front controls the up-front behavior. It is True for Android and web and False for XCUITest and the fake backend. A scenario that requests no video starts none regardless. - A confirmed start time corrects the report's step/network timestamps to the video's real origin, not the moment recording was merely requested. start_video (iOS) and start_screenrecord (Android), passed confirm_started=True at their production call sites, wait on a real signal after spawning, and store the confirmed time.monotonic() instant on Interval.true_start. - iOS waits for the Recording started line on the recorder's stderr. simctl io recordVideo writes that line once it has processed the first frame. - Android waits for the device-side file's first byte. screenrecord opens its output empty. It writes nothing until its muxer starts on the first encoded frame, so that first byte marks the recording's origin. - Android's wait runs on the device, in one bounded adb shell loop. The run clears the file before the spawn, so a leftover recording cannot answer the wait. - Android, where the wait cannot answer, falls back to the device-side process appearing. A running process is no proof its encoder emits frames yet, so that signal is weaker. It still beats a guess. - The web actuator stamps true_start right after it creates the recording page, with no poll. record_video_dir enables recording for the pages in a context. The video itself does not exist until a page does, so the stamp waits for new_page() rather than new_context().

intervals.adopt carries true_start forward unchanged when it relocates a prestarted interval. Android's confirmation, made before adopt even runs, is therefore not lost.

A poll that never confirms leaves true_start at None. The anchor then falls back to scenario_start, never a guessed number.

Waiting on the recorder's own announcement rather than on its output file is load-bearing on iOS, and was learned the hard way. simctl io recordVideo does not write progressively: the mp4 stays at zero bytes for the whole recording, and simctl writes it in one piece at the end.

An earlier confirmation polled that file for growth, so it could never succeed. It spent its entire ceiling on every scenario, green runs included. The iOS lane had raised that ceiling to 20 seconds on the belief that overshooting was free. simctl's own --help names the stderr line as the signal to wait on. That line arrives in about 0.15 seconds.

How long that wait may run is still a knob, because startup jitter in simctl and adb is worse on a loaded continuous-integration (CI) machine than on a developer's, and a wait that gives up costs the whole scenario its correction. BAJUTSU_VIDEO_START_TIMEOUT (seconds) replaces the 5-second compiled default, the same treatment the three BAJUTSU_XCUITEST_* timeouts get. No lane overrides it today. - The finished recording places its own origin, and outranks the confirmation above. Every true_start is a proxy: a recorder's own start line, a device-side process that has appeared, a browser page that exists. Each signal arrives at its own distance from the frame the recorder opens on, so a report anchored to one seeks off by that distance. The recording answers the question itself. A finalized clip states its own duration, and Interval.stop() knows the instant it ended, so the subtraction gives the origin outright: measured_start = ended_at - duration. The duration comes from the container, with no external tool: evidence/media.py reads the movie header that simctl and screenrecord write, and the Matroska segment that Playwright's recorder writes.

Which instant "ended" names is the recorder's to say. A subprocess recorder stops when the signal lands, then spends its finalize (and, on Android, a pull off the device) writing a clip it already captured. Playwright instead films right through the context close that stop() performs. A provider declares which shape it has through Interval.stops_when_stop_returns. run_scenario then resolves video_start_offset after finish_scenario_intervals, preferring measured_start and falling back to true_start, and records the result as RunResult.video_anchor_s.

The subtraction is only as good as its two inputs, and each can be wrong in a way the other cannot see. - A duration is not always a wall-clock measure. That is a property of the recorder rather than of the arithmetic. A container written at a nominal frame rate states more seconds than the recorder was ever open for, as Playwright's short clips do. The origin then lands before the spawn. - ended_at is not always when the recording ended, because a recorder can stop itself. Android's screenrecord quits at its own SCREENRECORD_TIME_LIMIT_S ceiling. A scenario that outlasts the ceiling signals a recorder that stopped minutes earlier. The origin then lands after the first frame by that whole gap.

A third issue sits in the recorder rather than in either input. The clip may stop short of the stop signal. The simctl movie header spans spawn to stop, even over a static screen. Playwright repeats its last frame until the context closes. Android's screenrecord behaves differently: it encodes a frame merely when the screen changes, and its file ends at the last one. A still screen before the stop thus makes the measured origin late by that whole stretch. One emulator run measured 0.8 seconds. That error sits well inside the window below, so no bound could catch it.

Interval.measure_origin decides whether stop() runs the subtraction at all. On Android, the anchors rank as follows:

  1. The first byte is the best anchor.
  2. The late measurement comes next.
  3. A pid instant comes last.

start_screenrecord turns the measurement off once the first-byte wait confirmed the start. That recording keeps its first-byte true_start anchor. Where the wait could not answer, the measurement stays on and still outranks the pid instant. intervals.adopt carries the flag along.

Interval.spawned_at bounds both, because it is the one instant that needs no confirmation. A recording opens on its first frame somewhere between that spawn and _ORIGIN_STARTUP_CEILING, which is a claim about how long a recorder can take to open rather than about how patient a lane chose to be — deliberately not BAJUTSU_VIDEO_START_TIMEOUT, whose 20-second iOS value had silently widened this window to admit origins no recording can have. An origin outside the window says one of the two inputs is not describing this recording, so it is discarded and that recording keeps the true_start anchor.

  • The recorded timestamps are absolute; a viewer derives the video-relative offset when it renders. run_scenario reads the wall clock once, beside its time.monotonic() stamp, giving the scenario an anchor pair: any later monotonic instant t becomes the wall-clock instant scenario_wall_start + (t - scenario_start). Every step's started_at and every network exchange's startedAt takes that form, so what survives the run is the raw timing data rather than a number a correction has already reduced. A viewer — report.html, bajutsu trace — subtracts video_anchor_s at render time to place an event on the recording's timeline. That is what makes the placement recomputable from a saved run: after a fix to how the anchor resolves, a re-render lands the events where they belong, with no need to run the scenario again (BE-0348). Every timing decision still reads the monotonic clock alone, since a wall clock can jump backward mid-run and would corrupt a wait's timeout or a step's duration. See reporting for what each recorded field means to a report reader.

Touch markers in the recording (--touch-markers)

A recording shows every consequence of a gesture and never the gesture itself, so bajutsu run --touch-markers asks the app under test to draw a marker at each touch it receives: a translucent circle at the contact, and a trail behind a contact that moves. Both are green while the touch is down and red once it has lifted, so a single still frame says whether the contact it shows is the gesture happening at that moment or the one it left behind. The marks are drawn inside the app's own process, so they reach the recorded video and each step's after.png alike. Because they are drawn from the UIEvent the app dequeues rather than from the coordinate the driver sent, a marker is evidence that the touch was delivered, which a driver-side coordinate record cannot show.

Three properties matter before a reader turns the flag on.

  • It needs an app that links BajutsuKit. The drawing lives in BajutsuKit (BajutsuTouch), which the demo apps already link. The flag sets BAJUTSU_TOUCH_MARKERS=1 on the app's launch environment, and an app that does not link BajutsuKit ignores the variable.
  • The marker is a CALayer, so it never enters the accessibility tree. A layer is not a UIResponder and conforms to no accessibility protocol, so no selector can resolve to it and it can swallow no gesture. Two checks keep the claim honest: demos/showcase/scenarios/golden/golden_xcuitest.yaml asserts the same tree golden twice, once with the markers on and once with them off, so a visualization that perturbed the pinned controls fails on device; and tests/test_touch_markers.py fails if the drawing code ever reaches for a UIView, which is the only way a marker could gain an accessibility representation at all.
  • A gesture's marks stay until the next gesture starts. No timer removes them, which is what keeps them in the step's screenshot, and equally why a run with the flag on produces screenshots that differ from a run without it. Where the in-app control channel is available (below), a visual assertion gets its own capture with the markers hidden, so the flag and a pixel comparison can share a run; everywhere else, leave the flag off for a pixel comparison. Android has no such channel: its scenario lanes turn show_touches and pointer_location on device-wide, and the tree-comparing and pixel-comparing lanes turn the pair back off by hand instead (demos/showcase/android/Makefile).

The markers are evidence only — no assertion reads them — and the flag is off by default for a plain bajutsu run. The repository's own iOS lanes do pass it: .github/actions/bajutsu-e2e and the showcase's run-swiftui / run-uikit targets run with the markers on, so a failure there shows where the gesture landed. That is safe because visual is the only assertion kind fed by a screenshot; every other kind reads the accessibility tree, the network exchanges, or the clipboard. No lane here builds its app with -DBAJUTSU_ENABLE_CONTROL_CHANNEL today. Adding a visual scenario to any of them fails until that build gains the setting (below).

The run loop hides the markers for the one capture a scenario's visual verdict compares, because that assertion reads the very image the markers are drawn into. It asks the running app to take the markers down, waits for the app to confirm, fires the shutter, then sends a second command putting them back. Masking cannot rescue the case, because the marker follows the gesture instead of sitting in a fixed region.

The request travels the in-app control channel (BE-0365). The channel is a command queue on the collector the app already reports to. The app opens no socket of its own, and the collector's per-run token guards the commands. The run loop waits for the app's acknowledgement rather than pausing for a fixed interval, and that wait makes the capture correct instead of hopeful.

The markers themselves never reach the verdict path. The channel that hides them does. Once armed, the channel fails a scenario whose app never acknowledges a command. The second command is no exception: it restores the markers after a capture that already succeeded. Both edges wait for an acknowledgement on purpose, because the run loop must never report an unconfirmed state as a confirmed one. From the moment the run loop arms the channel, --touch-markers bears on a verdict.

The channel needs a real Simulator process, since that is where BajutsuKit's poll loop runs. Carrying the channel takes the xcuitest actuator with network collection on. The other actuators fall short:

  • fake starts a collector, but nothing ever polls it.
  • playwright observes network through the driver, so it starts no such collector at all.
  • adb reports to the same receiver on the machine running bajutsu that iOS does, so it does start a collector. Nothing on the device answers a command. For that reason the actuator check keeps the channel to xcuitest, not the shape of the collector.

Where both conditions hold (the xcuitest actuator, with network collection on), --touch-markers activates the channel for a scenario whose verdict reads a screenshot.

Everywhere else, a screenshot-comparing scenario keeps the pre-BE-0365 behavior. It draws no markers, and the investigator loses the touch evidence for that one scenario. That is the same trade-off this channel exists to remove, where it can.

--backend names one target's platform, never an ordered fallback list (BE-0452). No scenario escalates past the run-level actuator to one with no pre-started collector.

The unmarked fallback stays local to the scenario that took it. One scenario can arm the channel while the next stays unmarked. Between scenarios, the run relaunches the app with each scenario's own launch env. An unmarked scenario runs in a process the marker variable never reached. The rest of the same run still draws its markers.

Where the channel is active, the app side gates it twice, and one of the two is a build setting. BajutsuKit compiles the channel out unless the build passes -DBAJUTSU_ENABLE_CONTROL_CHANNEL. Compiled in, the channel stays inert without BAJUTSU_CONTROL_CHANNEL=1 on the launch environment. --touch-markers supplies that key. An app built without the compilation setting acknowledges nothing. The scenario then fails naming both gates, rather than comparing an image carrying the markers.

The run loop reads those two launch-environment keys and nothing else, so it cannot tell a pair the flag armed from one a scenario assembled itself. It reads them from the scenario's effective launch environment. That environment merges the target's own subject.launch.env underneath the scenario's own, in the same order the launch itself uses. A target-level pin counts the same as a scenario's own pin.

Suppose the effective environment already sets BAJUTSU_CONTROL_CHANNEL=1. Adding the marker key would complete the pair. Any such scenario whose expect runs against a baselines directory would then enter the channel unasked. --touch-markers therefore leaves that scenario unmarked, and the run says so on stderr.

Sinks (where evidence goes)

class EvidenceSink(Protocol):
    def capture(self, driver, step_id, kinds, *, elements=None) -> list[Artifact]: ...   # instant captures after a step
    def wait_diagnostic(self, step_id, *, trace, elements) -> Artifact | None: ...       # the first-wait timeout diagnostic (below)
    def start_scenario_intervals(self, scenario_id, kinds) -> list[Interval]: ...        # begin video / deviceLog / appTrace for the whole scenario
    def finish_scenario_intervals(self, scenario_id, started) -> list[Artifact]: ...     # stop them and collect the files
Sink Behavior
NullSink (default) writes nothing (keeps a run side-effect-free)
FileSink(run_dir, udid, log_predicate) writes under run_dir/<step_id>/

A capture the environment already began before launch (Android's video) is adopted rather than started. On stop, the sink relocates its finalized file into the scenario dir.

Otherwise, interval captures come from the driver's driver_interval provider when it supplies one (web's Playwright-native console / video, Android's adb logcat). When the driver supplies none, FileSink takes the simctl path, which it skips when udid is absent. The CLI's run uses FileSink(runs/<runId>, udid=..., log_predicate=...) (cli).

First-wait timeout diagnostic (BE-0231)

A wait for <element> that times out writes run_dir/<step_id>/wait-timeout.json unconditionally, independent of capturePolicy. A timeout that no policy rule would have captured thus still leaves the evidence needed to decide why it fired. The file is pure diagnosis, never a verdict input. The run's pass/fail still comes from machine-checkable assertions alone.

The file is self-contained so a rerun-to-green does not discard it:

Field What it answers
readiness Whether the post-launch readiness gate had passed, on which signal (readyWhen, the target's subject.startWhen condition / namespace / count, or timeout), and whether the screen had settled — separates "the gate returned before the content" from "the content rendered but the awaited element did not". A settled: false says the gate returned while the tree was still changing, which is when a synthesized touch is dropped: read it as "the actuation before this wait may never have landed", since a dropped tap is still reported as delivered. null on a lane that carried no readiness result.
trace The poll timeline: how many polls, when the tree first became non-empty (firstNonemptySeconds, null if it never did), and how many elements were present at the timeout — separating "nothing rendered / transient-empty" from "rendered, awaited element absent" from "slow cold-boot render".
provenance A BE-0049 stamp (scenario hash, tool version, git revision), so the evidence stays identifiable independently of the run. Its scenarioHash fingerprints this scenario alone, without the file-level description the run manifest's scenarioHash folds in when present — so it can diverge from the manifest's hash even for a single-scenario run, not only for a suite/matrix run.
elements The (redacted) element tree at the moment of timeout.

It is recorded as an Artifact(kind="waitDiagnostic", provider="runner") — written by the run loop, not a backend actuator.

App-crash evidence (BE-0424)

When the app under test itself terminates abnormally mid-scenario, the failing step's message names the event and the scenario's evidence directory gains an app-crash/ subdirectory holding the platform's own report for it. Without this, the failure reads as an ElementNotFound naming a selector that should exist and normally does, so a contributor has to rule out a renamed id and an unloaded screen before suspecting the app at all.

The evidence is a diagnostic, never a verdict. The scenario fails because its step failed, as it would for any other terminal step failure. Nothing here decides pass/fail (prime directive 1).

The crash is classified in-band, so the scenario finishes normally. The failing step's screenshot, the running video recording, and any cleanup: on: failure rule land as they always do.

The backend-crash retry loop (crash-diagnostics/ above, BE-0421) never triggers. That loop names a fault in the test infrastructure. This evidence names a likely defect in the app a team is testing.

Backend What lands in app-crash/ How the crash is confirmed
iOS (XCUITest), Simulator only the .ips report macOS's ReportCrash wrote for the faulting process, under its own name. Where no report exists, unified-log.txt instead: the Simulator's own unified log for the app's launch window, bounded to its last 256 KB. On the GitHub-hosted macOS runner, ReportCrash writes no report for a Simulator app at all XCUIApplication.state answering notRunning. On a real device (runsOn.ios.kind: device) nothing is confirmed: the same answer can mean an OS memory-pressure kill, and the device's reports never reach this host
Android (adb), API 30 and above logcat-crash.txt, the crash buffer's own block for this process — managed (FATAL EXCEPTION) or native (Fatal signal) — plus tombstone_NN where the device allows adb root pidof reporting no process, corroborated by dumpsys activity exit-info reporting CRASH / CRASH_NATIVE at or after this launch. Below API 30 ApplicationExitInfo does not exist, so nothing is confirmed
web (Playwright) nothing left to a follow-up item

The check is reactive. The runner asks a driver only after a step's own action, wait, or assertion has already failed, and never polls. Polling would add a round trip to every step of every green run, to catch a fault that is rare by construction. The reactive shape still catches the event at the exact step where it happened, since that step is already failing.

That shape has two blind spots. The design accepts both deliberately. An unconditional end-of-scenario probe would close them, but it would add the every-green-run cost the design exists to avoid.

  • A crash caused by a scenario's last step can go unprobed when that step's own actuation still reports success (a tap the runner delivered before the app died). A scenario that declares neither expect nor a non-failure cleanup rule then stays green.
  • The same shape with a scenario-level expect goes red but unclassified. expect produces assertion results rather than step outcomes, so it never reaches the check. The reader sees an assertion mismatch instead of a crash.

A scenario written to exercise this signal takes one more step after the crash trigger. The later step, not the trigger itself, fails and gets probed.

A cleanup rule that runs on success or always is enough too. Its own teardown step does reach the check, so the check catches the crash one step later rather than missing it.

A crawl records the same evidence under crashes/crash-NNN/app-crash/, beside that crash's own repro scenario. A crawl's detection stays the accessibility-tree heuristic it already used, and a covering system alert can trip that heuristic. A driver must positively confirm the event before the crawl captures anything. An unconfirmed detection therefore records the crash with no artifacts, rather than paying a bounded sweep on what may be an alert.

The Android tombstone layer is run-only. The adb root it needs restarts adbd. A crawl lane could not recover from that restart, because the lane holds one environment for its entire walk.

Every write goes through the redacting text path, like the backend's own crash evidence above: a crash report is text a crashing app can echo a secret into.

Artifact provenance (provider)

Every piece of evidence is recorded as an Artifact(name, kind, provider, depicts), leaving in the manifest which provider it came from and which screen it shows.

@dataclass
class Artifact:
    name: str            # filename (e.g. "before.png")
    kind: str            # "screenshot" / "elements" / "video" / "deviceLog" / "network" / "waitDiagnostic"
    provider: str        # who supplied this artifact (see table below)
    depicts: str | None  # the screen it shows, as "<driver>:<moment>" (see below)
provider value Meaning
"driver" The actuator captured it directly (screenshots, element trees).
"runner" The run loop wrote it (the first-wait timeout diagnostic, BE-0231).
"simctl" Interval evidence from simctl (video, device log, app trace).
"adb" Interval evidence from adb (screenrecord video, logcat device log).
"collector" The app-side network collector (BAJUTSU_COLLECTOR).
"playwright" Native Playwright network observation (web backend).
"<backend> (fallback)" A read-only evidence fallback supplied the artifact (BE-0020).

When an evidence kind cannot be supplied by any backend in the list, a SkippedCapture(kind, reason) is recorded per scenario and disclosed in the manifest. The gap is never silently empty.

Each interval kind is a capability of its own (video, deviceLog, appTrace), judged against the route the scenario runs on (BE-0467):

Route video deviceLog appTrace
iOS Simulator Recorded Recorded Recorded
iOS real device (runsOn.ios.kind: device) Skipped Skipped Skipped
Live WebDriver endpoint Skipped Skipped Skipped
Android Recorded Recorded Recorded
Web Recorded Recorded Skipped

A gap is recorded for a kind the scenario asked for, and for no other kind. The sink never starts a recorder for a skipped kind: simctl reaches the Simulator alone, so the sink does not point it at a real device. Evidence never decides a verdict, so a skipped kind leaves the result unchanged.

Which screen an artifact shows (depicts)

A step's screenshot and its element tree are shown together, and a viewer draws a hovered element's frame onto the image, so the two have to describe the same screen. depicts makes that checkable. Its value, "<driver>:<moment>", names the driver whose read produced the file. It also names which side of the step's action (before or after) the read came from. A native step's before.png carries "xcuitest:before", and its after.png and tree carry "xcuitest:after".

Two artifacts describe the same screen exactly when their depicts values are equal. A consumer compares the values and never parses them. Every consumer does the comparison in evidence.step_view. The consumers are:

  • the HTML report's element viewer
  • the serve editor's element picker
  • the triage context handed to a failure investigator

None of them can disagree about which screen a step "is".

evidence.step_view resolves a step to one screenshot, one tree, and whether the two are paired. An unpaired step keeps its image and loses its frames. That is the honest rendering, because the frames would land on pixels they never described.

Two situations produce an unpaired step:

  • Inside a web block, the tree comes from the WebView, in the WebView's own coordinate space. The screenshot comes from the native driver, since a WebContextDriver cannot take one.
  • A run whose store no longer holds after.png falls back to the before.png beside it. The post-action tree does not describe that image. Restoring a run from Trash produces this case, as does syncing a run into an object store that never received the last write.

A step that fails before it acts is neither case. It records its pre-action pair alone, and that pair matches.

depicts is absent from every run recorded before it existed (manifest.json schemaVersion 8 and below). Nothing in such a manifest says which side of the action an artifact was taken on. A consumer therefore reproduces the earlier choice, after.png over the before.png beside it, and treats the pair as paired. This keeps the frames a stored run has always shown.

Visual evidence

A visual assertion produces a VisualEvidence record, which the manifest and the report carry. The record holds the run-dir-relative paths to the baseline copy, the actual screenshot, and the diff visualization (when the comparison found differences). It also holds two values:

  • diff_pct, the percentage of pixels that differed
  • engine, the comparison engine that produced the verdict ("exact" or "pixelmatch"; BE-0165)

The engine is selected per assertion (compare:) with a target-level config fallback (evidence.visualCompare), and is recorded in the manifest so the algorithm that produced each verdict is traceable. Implementation: bajutsu/assertions/visual.py VisualEvidence.

Masking (redact)

Screenshots, logs, and network data can capture personally identifiable information (PII) and tokens. Declare what to mask before writing.

The run merges config's evidence.redact with the scenario's redact (union) (configuration). Implementation: scenario/models/evidence.py Redact.

redact:
  labels: ["Card Number"]               # accessibility labels
  headers: ["X-Session"]                # extra HTTP header names (on top of the defaults)
  fields: ["token", "password"]         # JSON/body field names
  unmaskHeaders: ["authorization"]      # opt out of a default (visible, deliberate)
  unmaskSecureFields: true              # opt out of the platform-marked default (below)
  unmaskCredentialNames: true           # opt out of the credential-name default (below)

Where redaction runs

Every write into a run directory goes through one sink, so an artifact is redacted because of where it lands rather than because its writer remembered to ask (BE-0331). The sink takes content before serialization, with one entry point per shape: an element tree, network exchanges, a crawl's screen map, and free text. Two of the rules below are structural. They read an element's trait, or its identifier and label. A JSON string no longer carries that pairing, so a sink that scanned serialized text could apply neither.

Content the sink cannot inspect (a screenshot, a video, an archive) goes through a separate entry point. That entry point records the artifact as unmasked. An image is never described as protected. Implementation: evidence/sink.py RunArtifactWriter.

The boundary governs writing alone. Reading a run stays unrestricted. The web UI, the evidence readers, export, and the comparison commands all need to read, and none of those operations can create an artifact.

Two mechanical checks keep the write side closed. An import contract lets the sink alone derive a writable run path. A second check fails a run-root path literal written anywhere else. Both read source alone, so a writer nobody has written yet is covered the moment it exists.

Masking that needs no configuration

Three rules mask without a redact: block, because each covers a case a scenario author should not have to anticipate. A crawl carries no scenario at all, so these are the only rules that reach its artifacts.

  • A field the platform marks secret. An element whose backend reports the masked-input trait has its value masked, and so does a value typed into such a field. Each backend derives the trait from its own source: XCUITest's secureTextField, the web input[type=password], and the Android accessibility node's password flag. The driver conformance suite (BE-0114) pins the trait on every backend, so the rule means the same thing on iOS, web, and Android. Release it with unmaskSecureFields: true.
  • A field whose identifier or label names a credential. The vocabulary is password, passwd, secret, token, apikey, api_key, credential, otp, and pin, matched case-insensitively on word boundaries — so a field called settings.apikey is masked and one called prefs.pinned is not. The list is small and documented rather than clever, because a rule an author cannot predict is one they cannot rely on. Release it with unmaskCredentialNames: true.
  • A recognizable credential shape. As a backstop, the sink masks a set of high-confidence shapes in any text it writes. There are five: an Anthropic key (sk-ant-); an Amazon Web Services (AWS) access key id (AKIA); a GitHub token (ghp_ and its siblings); a three-segment JSON Web Token (JWT); and a Privacy-Enhanced Mail (PEM) private-key block. The sink masks a match and logs a warning naming the artifact, because a value reaching the backstop means an earlier, more precise rule should have caught it. The patterns are literal regular expressions, so no model is consulted. This rule alone needs to know neither a configured name nor the value in advance, which is why it alone can reach a value the tool itself generated. The leak that motivated this boundary was exactly that: a crawl --guide ai run wrote an artifact after the model invented a realistic API key to fill a field.

What redaction does not guarantee

Masking reduces what a shared artifact reveals; it does not make a leak impossible. Three residues remain, and an operator deciding whether to share a report should weigh them.

  • Pixels stay unmasked. A screenshot of a filled password field still shows it, which is why the runner warns up front rather than implying an image is protected (BE-0151).
  • An arbitrary value in an unmarked field can survive. A secret typed into a field the platform does not mark, whose identifier and label name nothing credential-like, that was never bound through ${secrets.X}, and whose shape matches no backstop pattern, is indistinguishable from ordinary text. Telling the two apart would need semantic judgment, which never sits on this path.
  • The backstop's vocabulary is finite. A credential format nobody added a pattern for passes it.
  • A configured key can over-mask a structured string. A redact.headers / redact.fields name is also matched in the name: value form. That form's value has no delimiter, so it runs to the end of the string. In a log line, that is right. Inside a bounded string an artifact carries, it consumes the rest of that string. Naming app rewrites an Android resource id like com.example.app:id/login to com.example.app:[REDACTED], and a crawl --continue reading such a screen map back replays a branch that resolves nothing.

Masking wins over legibility here on purpose, because a key an author named is one they meant. Keep redact.fields to app body-field names, not words that also appear in a control's own identifier.

Sensitive headers are masked by default (a scenario needs no redact: for this): the built-in set is authorization, proxy-authorization, cookie, set-cookie, x-api-key, and x-auth-token, matched case-insensitively. cookie and set-cookie are treated as one concern, so naming (or unmasking) either covers both. Header names in redact.headers add to this set and never replace it. When a reader needs a default header's raw value (for example, debugging an auth failure), the reader names it under unmaskHeaders. Turning off protection is an explicit, visible choice, never the mere absence of redact:.

Redaction is applied before evidence is written (evidence/redaction.py Redactor):

  • The device log and app trace are scrubbed by key→value patterns.
  • The element tree masks a value when its label is configured (or scrubs an embedded secret).
  • Each network exchange is masked structurally: header values by name, and the url / request / response bodies as free text, so query params and token / password body fields are caught.

Images (screenshots / video) cannot be masked and are left as-is.

Redaction also extends to secret input values: the literal values behind ${secrets.X} (resolved from the environment, declared via config's evidence.secrets configuration) are masked wherever they would appear in evidence, not just the configured labels / headers / fields. Redaction masks the longest values first, so a value that is a substring of another never leaves a partial leak.

Value matching is encoding-aware: the same secret reaches evidence verbatim but often encoded, so its literal bytes never appear. Alongside the raw value, redaction masks its common encodings:

  • percent-encoded (a URL query or form field, e.g. p@ss as p%40ss)
  • HTML-escaped and JSON-escaped forms
  • an Authorization: Basic <base64(user:pass)> token whose decoded credential carries the value

This is a fixed set of transforms applied to known values (the value is encoded, then searched for), not a decode-everything scan, so the cost and false-positive surface stay bounded.

One limitation remains. Where evidence is genuinely fragmented before redaction runs (a value split across streamed chunks that redaction never sees as one contiguous string), matching is best-effort. Assembled full-text evidence, the common case, is unaffected.

The executed scenario is also snapshotted into the run directory (scenario.yaml, and the raw YAML view in the report). A totp step's secret is a durable base32 seed, not a one-time code, so a literal seed written straight into the scenario is masked to <redacted> in that snapshot. A ${secrets.X} reference is kept as-is, because it is not the seed, and the secret-value rule above masks its resolved value. Prefer ${secrets.X} for a totp seed, so the seed never sits in the scenario file to begin with.

File permissions

Redaction reduces what a leaked artifact reveals, but a best-effort denylist cannot promise more. Who can read the artifact matters too.

The runner creates each run directory with owner-only access (0700). The sensitive files a run may hold also get owner-only access (0600), whatever the umask of the machine running bajutsu says (BE-0131). Those files are network.json, the copied scenario.yaml, the element dump (elements.json), and screenshots.

Everything else lands under the 0700 run directory. On a shared machine (a CI runner, say), another local account cannot read a run's evidence by default. Implementation: bajutsu/common/run_meta/artifact_perms.py.