English · 日本語
The evidence subsystem¶
Evidence capture for a recurring action is expressed as a repeatedly-firing rule rather than a one-shot instruction. The rule collects the same evidence on every later run, without AI.
Implementation: bajutsu/common/evidence/core.py (instant captures and Sinks) · bajutsu/common/evidence/intervals.py (interval captures: video / deviceLog / appTrace). The orchestrator decides when a rule fires (run-loop).
Related: the capture tokens in scenarios · reporting
Three ways to request evidence¶
| Way | Use | Example |
|---|---|---|
A. Rules (capturePolicy) ★ central |
automatic capture every time a particular action happens | network exchanges on every tap of settings.* |
B. Per-step (capture:) |
this one step only | video + deviceLog around a specific wait |
| C. Default policy | a baseline guarantee | config's evidence: { capture: [screenshot.after, elements, actionLog] } |
The config default (C),
evidence.capture, resolves toEffective.capture(configuration). It applies on top of every step, alongside the scenario'scapturePolicyand the per-stepcapture. Unlike those two, it fires unconditionally rather than on a trigger, so it acts as a baseline guarantee, not a rule.All three ways request evidence on top of what every step already records on both sides of its action. Naming
screenshot.afterorelementsin any of them therefore changes nothing, and leaving either out costs nothing. Whatever the three ask for, the run capturesbefore.png,after.png, and the post-actionelements.json(below).
Evidence kinds and acquisition timing¶
A capture: token is <kind>[.<modifier>] (scenarios).
| Kind | Source | Interval / instant | Status |
|---|---|---|---|
screenshot |
the driver (XCUITest's own /screenshot endpoint, adb's screencap, Playwright natively) |
instant | ✅ captured |
elements (a11y / accessibility tree) |
driver.query() as JSON |
instant | ✅ captured |
actionLog |
orchestrator internals (action · duration) plus each driver's own actuation records | — | ✅ inherent in the manifest |
video |
simctl io recordVideo |
interval | ✅ captured (needs udid) |
deviceLog |
simctl spawn log stream |
interval | ✅ captured (needs udid) |
network |
the in-app collector (BajutsuKit → network.json) |
interval | ✅ captured (the --network run flag) |
appTrace |
simctl spawn log stream over the app's os_log subsystem (iOS); adb logcat filtered to the app's tag (Android) |
interval | ✅ captured (needs a device + subsystem) |
rawTree |
the device's own reply behind elements, untouched (base.RawSourceProvider; adb and XCUITest today) |
instant | ✅ captured (opt-in, no-op elsewhere) |
appTracepairs the app's<name> started/<name> finishedlog markers into timed intervals. On iOS the markers come fromos_signpost/os_log(intervals.parse_app_trace). On Android the app logs them withandroid.util.Log, as inLog.i(tag, "checkout started").intervals.parse_logcat_app_tracepairs those (BE-0478).networkis produced by the request collector rather than the interval system — its exchanges are written to<sid>/network.json(network observation, the--networkflag).Every entry in
elements.jsoncarries the element'sidentifier,label,traits,value, andframe. One further field is diagnostic:nativeZ, the element's real front-to-back position as the app under test measured it (BE-0355). Reading it changes nothing a run decides — no selector matches onnativeZ, and the occlusion checks (is_tappable,topmost_at_point, and XCUITest's ownisHittable) behave exactly as they did before the field existed. On the web backend, an element read from a cross-origin iframe also carriescrossOriginFrame: true(drivers); every other element omits the field.What the number means is the backend's own, and only its direction carries across. On both backends a larger
nativeZis closer to the viewer. Nothing else about two values is comparable unless they came from the same backend.
- iOS reports an ordinal over the app's real compositing order, counted back to front.
CALayer.zPositionreads zero across an ordinary flat layout, and Apple documents it as the wrong tool for sibling order.- Android reports
View.getZ()(elevation plus any translation on the z axis) in device pixels. That value orders siblings within one parent and nothing wider: a child at0under a parent at8still composites in front of that parent's sibling at4.So on Android, a number tells a reader which of two elements is in front only when they are siblings. On iOS, any two values from the same read compare, since the ordinal spans the whole screen. No comparison holds across backends.
A
nullis the common case, and it is deliberate rather than a gap to be filled in by inference. Reporting a real value needs an app that opted in:
- iOS needs two things. The app links BajutsuKit, and bajutsu drives the app through its own XCUITest runner. The WebDriverAgent-backed live/record path injects no port for the responder to answer on, so it reads
nulleven for an opted-in app.- Android needs the app to call
BajutsuZOrder.report(view)in a debug build.Two toolkits report nothing even then, because each generates its own accessibility elements and does not expose the underlying one the position would be measured from. SwiftUI materializes its accessibility elements only for an assistive technology attached to the process, so an app looking at its own view tree finds no identifiers on it. Jetpack Compose forwards no app-declared extra-data key through its own node generation. UIKit and Android
Viewscreens report; SwiftUI and Compose screens readnull.The element list's own order is the paint-order proxy that
topmost_at_pointfalls back to. Deriving a position from it would read as authoritative while being wrong on exactly the layouts an investigator opens the evidence for. One such layout is an Android view whoseelevationlifts it above a sibling declared after it.Inside an iOS
app:block, every element readsnull. The responder runs in the test target. While another app owns the screen, the test target sits suspended in the background, so its positions would describe a tree the block is not reading. Requests queued to a suspended app were measured to kill it withSIGPIPEonce it resumed (the Home Screen widget spike). The driver skips the round trip until the block ends.On Android the target must also ask for it, with
driver: { nativeZ: true }under itstargets.<name>entry. The reading comes from a walk the resident server performs over every node on every screen read. That walk costs 20-100 milliseconds whether or not the app opted a single view in (BE-0407 unit 18). An app that callsBajutsuZOrder.report(view)and does not set the flag readsnull, the same honest absence every other unreported case gives.iOS needs no such flag. Its responder answers only when the app links BajutsuKit, so a target that did not opt in pays nothing to ask.
rawTreewriteshierarchy.raw<suffix>, the device's/runner's own reply, untouched by any of bajutsu's processing. For adb that is theuiautomator dump/resident XML (.xml); for XCUITest it is the undecodedGET /elementsbody (.json).On adb's resident channel, when narrowing changed something,
rawTreealso writeshierarchy.parsed-input.xml: whatparse_hierarchyactually consumed, after SystemUI decor windows were stripped. XCUITest applies no such transform, so it never writes a second file.The capture exists to diagnose a mismatch between a resolved coordinate and the real screen. It shows whether the device's/runner's own reply already looked wrong, or whether bajutsu's own parsing changed it. It is never in the default capture list; a scenario opts in with
capture: [rawTree, ...].One redaction rule refuses it outright: when
redact.labelsis configured,rawTreewrites nothing for the whole run and logs why.redact.labelsmasks a labeled element's value structurally.elements.jsonis written from the parsed tree, so the writer knows which value to blank. The raw dump is free text with no such structure, so it would ship an unmasked superset of whatelements.jsonjust masked.Every other redaction rule (
headers,fields, resolved secret values) applies to the dump as free text and leavesrawTreeenabled. One caveat applies toredact.headersandredact.fields. Their key-pattern masking is written for multi-line logs, where a matched value ends at the next newline, but a UI Automator dump is emitted as a single line. A configured key that happens to match text inside the dump itself (an on-screen label orcontent-descreading likeToken: ...) therefore masks everything after that match to the end of the file, not just the matched value. The dump still ships, just truncated. A resolved secret value (bound via${secrets.*}) is unaffected, since it is masked by matching a known literal rather than a key pattern.
actionLog — what each step actually did to the screen¶
actionLog needs no capture request and writes no file of its own: every step's outcome carries an
actuations list in manifest.json, one entry per primitive the driver performed, and the report and
the bajutsu trace timeline read it from there. It answers the question a screenshot and an element
tree cannot: where did this tap land, and how far did this swipe travel.
| Field | Meaning |
|---|---|
gesture |
the driver primitive — tap, doubleTap, longPress, swipe, scroll, pinch, rotate, the text primitives, selectOption, setPickerValue, systemAlert, back, navigate, openUrl |
via |
how the gesture reached its target: coordinate (the driver computed a point and sent it), handle (XCUITest actuated a snapshot handle), identity (the Android device resolved the element and chose the point), bridge (a WebView call addressed by element id), focused (a text primitive on whatever field holds focus), key, history, url (a page load by navigate, or a URL handed to the operating system by openUrl; the URL itself is not recorded) |
unit |
the coordinate space: point (iOS), pixel (Android), cssPixel (a browser page, or a WebView's own space) |
points |
the coordinates the driver sent, in order — one for a tap, two for a drag's start and end. A two-finger gesture records the single anchor its two contacts were derived from, not the contacts |
frame · target |
the resolved element's bounds and its accessibility identifier |
accepted |
whether the platform accepted this attempt, on the two channels that answer (XCUITest's handle actuation, Android's device-side endpoint). A refused attempt is shown struck through, so a stale-retried tap does not read as several taps; None means the channel gave no separate answer |
substitution |
why the element actuated is not the one the driver's default rule would have named — soleHittableDescendant when a refused tap was redirected to the one reachable named descendant inside its frame. Both readers show it: the report as a badge beside via, the trace timeline as ↷<token>. Absent on the ordinary path, and on every run recorded before schemaVersion 7, which reads the same way: no substitution happened |
duration_s · scale · radians |
the gesture's non-positional parameters, where it has any |
at |
the absolute wall-clock instant (epoch seconds) the driver sent the gesture, on the same footing as the step's started_at. A step stamps started_at before it screenshots, reads the tree, and resolves its target, so the recording changes only at at — up to seconds later on Android. The report highlights a step row from this instant, and both readers show the lag after the step's start: the report beside the gesture, the trace timeline as +0.85s. Absent on every run recorded before schemaVersion 13 |
Three rules bound what a record may say, and every backend honors them:
- Only a coordinate that was really sent.
pointsis empty whenever no coordinate crossed to the platform (a handle-based iOS tap, an Android device-side gesture), because the far side chose the point. The record shows the resolvedframeinstead. It does not present the frame's centre as a measurement it never took. - No device work. Every value is one the actuator already had, so recording costs no extra query, read, or round trip.
- No authored string, ever. The writer of
manifest.jsonruns no redactor, so the record carries none of the following: - a
typestep's text, not even its length (Redactoruses a fixed-width placeholder, so no artifact discloses a secret's length) - a
selectOption's option - a
setPickerValue's value - a
navigateoropenUrlstep's URL - an element's accessibility label
target is always the resolved accessibility identifier and nothing else. An element with no
identifier leaves it unset.
A record is written when the gesture is attempted, before its transport answers, so a step that failed to actuate still shows what it aimed at. The step's own result says whether the step worked.
One actuation belongs to no step. The reactive system-alert guard also acts before the
scenario-level expect re-check. Its record lands on the scenario's expect_actuations, beside
expect_alerts. A backend that does not implement the record contributes none, and the run does
not change.
The driver's log is bounded, so a pathological step (a maxScrolls in the hundreds) can lose its
earliest records. A report that loads a manifest back can lose a damaged record the same way.
dropped_actuations counts both kinds, so the list never reads as complete when it is not.
The Android backend also records what happened to its read channel, because a run there can pass on
a different read path from the one it started on. Nothing on the verdict path reads these fields. A
step's warnings name a recovery, a loss, or a return during that step. A scenario's
channel_degraded says why reads were on the uiautomator dump path when the scenario ended, and it
stays empty while the channel is healthy. A scenario's channel_notes hold what happened after the
last step. A run recorded before schemaVersion 18 carries neither scenario field, which reads as
healthy.
Default modifiers: the always-on instant baseline (below) is before — captured before the step
acts, not after. A capturePolicy rule or inline capture: still defaults an unmodified instant
kind to after when it fires; interval kinds (video/deviceLog) default to around (start
before the action, stop after the step). Stating screenshot.before explicitly on a rule/inline
capture is redundant with the baseline and is dropped rather than re-taken.
A. capturePolicy (rule-based)¶
Repeatedly-firing rules, written per scenario (implementation: scenario/models/evidence.py CaptureRule /
Trigger).
capturePolicy:
# On every tap of settings.*, also capture the network exchanges — screenshot and elements are
# already guaranteed on every step by config's default policy (C, above)
- on: { action: tap, idMatches: "settings.*" }
capture: [network]
# On every screen transition
- on: { event: screenChanged }
capture: [screenshot.around, elements]
# On error in any step, capture the maximum (the safety net)
- on: { result: failure }
capture: [screenshot, video, deviceLog, elements, actionLog]
The trigger on is exactly one of action / event / result:
action: <tap|longPress|type|swipe|...>— optionally combined withidMatches(glob against the primary target'sid).idMatchescan only be used withaction.event: screenChanged— fires ifquery()changed during that step.result: failure— fires if the step failed (the safety net).
The detailed firing logic is in run-loop.
Preview firing before a run (BE-0028). A loose glob or a
screenChangedrule can fire on far more steps than intended, and attaching a heavy capture (video/deviceLog/appTrace/network) to such a rule quietly produces gigabytes.bajutsu trace --explain <scenario.yaml>is a read-only dry run. It counts how many times each rule would fire (and on which steps) and flags ⚠ a heavy capture on a broadly-matching rule, so a reader can tighten the match before paying for it. See cli.
B. Inline evidence¶
To capture just one step, attach capture: directly to the step.
- tap: { id: settings.reindex }
- wait: { for: { id: settings.reindexComplete }, timeout: 5 }
capture: [video, deviceLog] # record the interval of this wait
(real example in demos/showcase/scenarios/evidence.yaml)
Interval evidence (video / deviceLog / appTrace)¶
Implementation: bajutsu/common/evidence/intervals.py. These are subprocess child processes — simctl on iOS,
adb on Android — started before the action and stopped after the step settles. Process spawning is
injectable (Spawn) and testable. Web has no subprocess: its intervals are Playwright-native and
supplied by the driver (see below). On iOS, appTrace is a log stream over the app's os_log
subsystem. parse_app_trace pairs its markers into timed intervals.
On Android, appTrace streams adb logcat -s <tag>:V, and the app logs its markers under that tag.
The tag is --log-subsystem when set, else the app's package. The stream follows the tag rather
than a process id, so a relaunch leaves the stream running. The stream starts at an instant
read from the device's own clock (logcat -T), to the second. The device thus decides which lines
came earlier. A cutoff on the computer running Bajutsu would compare device times with its own
clock. Any clock drift would then drop real lines. A line logged up to a second before the
stream can still appear. A later started of the same name replaces such a line when pairing.
When Bajutsu cannot read the device clock, the stream starts from the last logged line with a warning. On stop, parse_logcat_app_trace reads
each line's time and pairs the markers into appTrace.json.
Interval kinds are opt-in (BE-0028).
video/deviceLog/appTraceare heavy, so a scenario records an interval only when it asks for that kind — through an inlinecapture:or acapturePolicyrule (e.g. aresult: failurerule that capturesvideo). A scenario that requests none records none, keeping the common case cheap; the lightweight instant baseline (screenshot+elements) is always captured, so a failure still leaves evidence (DESIGN §10).Every step captures a screenshot on both sides of its action, but a tree only after it.
before.pngshows the screen the step is about to act on. Once the step has acted,after.pngand anelementswrite show what the action left behind (BE-0407). Neither screenshot depends on thecapturelist, so narrowing that list never removes either of a step's two screenshots.When nothing has actuated the device since the previous step's
after.png,before.pngcan be a byte-for-byte copy of it rather than a fresh capture. Four cases skip the copy and always shoot fresh, because an asynchronous interstitial could have arrived between the two steps:
- an interrupt's recovery step
- a
handleSystemAlertstep- any step in a scenario declaring
interrupts- any step running under the system-alert guard
The guard is on by default, so a stock
runshoots fresh on every step. The copy reaches a step only where--no-system-alert-handlingturns the guard off.
elements.jsonhas a single filename, so the post-action write is the tree a run keeps. It describes the screen the action produced, which is the screenafter.pngshows and the one every viewer draws element frames from. On a non-mutating step (assert,wait), that tree is the one the step itself settled on, reused rather than re-read (BE-0259), so it comes from a moment just before the screenshot rather than just after.Preview what a scenario would record with
bajutsu trace --explain(see cli).
| Kind | Start command (iOS / Android) | Stop signal | Filename |
|---|---|---|---|
video |
simctl io <udid> recordVideo --codec h264 / adb shell screenrecord |
SIGINT (a hard kill would corrupt the mp4) | scenario.mp4 |
deviceLog |
simctl spawn <udid> log stream --level debug --style compact [--predicate ...] / adb logcat -b main,system,crash,events -T 1 |
SIGTERM | device.log |
start_video/start_device_log(iOS) andstart_screenrecord/start_logcat(Android) return anInterval, andInterval.stop()sends the signal and finalizes the file.deviceLogwaits up to 10s, then sends a kill.videogets a generous 120s window to finish writing before the kill.recordVideo/screenrecordstill has to flush and mux the whole clip to disk. A premature kill truncates the mp4 (nomoovatom) and, on iOS, wedges the simulator's recording session.screenrecordrecords device-side. On stop, itsIntervalalso pulls the finished mp4 off the device and removes the device copy.- If that pull fails (the device vanished), the sink drops that one artifact with a warning. It does not emit a path with no file behind it, and it does not fail an otherwise-passing scenario.
adb screenrecordcaps a single recording at ~180s. That is the platform default/maximum, not a limit bajutsu tunes. An Android video of a longer scenario ends at that mark.- On iOS,
--predicate(NSPredicate) narrowsdeviceLogto a subsystem, for example (the CLI's--log-predicate). adb logcathas no tag or priority filter. A logcat filterspec is a different syntax, a later knob.adb logcatstarts the follow from the tail, so it reflects the scenario window, not the whole ring buffer.- The buffer set widens past bare
logcat's default (main,system,crash) to addevents. An app's own uncaught exception lands incrash. A process thatActivityManagerends for memory pressure logs a structuredam_kill/am_low_memoryentry ineventsalone. Withoutevents, that cause looks like a silent, uncaptured fault. - The kernel's own out-of-memory (OOM) / low memory killer (LMK) path lands in the kernel ring
buffer instead.
logcat -b kernelreaches that buffer where logd bridges/proc/kmsg(ro.logd.kernel, typically userdebug builds) and nowhere else. The set here leaves the buffer out, though it is not out of reach. INTERVAL_KINDS = {"video", "deviceLog", "appTrace"}. The orchestrator uses this set to split "interval / instant."- The scenario-wide
videobegins before the app launches on Android, so the recording spans the app's cold start rather than missing it. There, the environment'sstartstarts recording (after the device is booted and the app installed, but befoream start) and hands the runningIntervalback throughprestarted_intervals. At scenario start, the sink adopts that interval (intervals.adopt) instead of starting a fresh one. On stop, it finalizes the interval and relocates the file toscenario.mp4(Android'sscreenrecordwrites real ISO base media).
Web wires the same up-front capture into the browser context at creation, but finalizes to
scenario.webm instead. Playwright's recorder always writes Matroska/WebM, so its artifact is
named after that real container rather than borrowing Android/iOS's .mp4 (see
PlaywrightDriver.video_extension in drivers.md).
XCUITest, the current iOS backend, records on demand instead. Nothing starts a recording before
the xcodebuild runner spawns and launches the app, so its prestarted_intervals is always empty.
The flag records_video_up_front controls the up-front behavior. It is True for Android and web
and False for XCUITest and the fake backend. A scenario that requests no video starts none
regardless.
- A confirmed start time corrects the report's step/network timestamps to the video's real
origin, not the moment recording was merely requested. start_video (iOS) and
start_screenrecord (Android), passed confirm_started=True at their production call sites,
wait on a real signal after spawning, and store the confirmed time.monotonic() instant on
Interval.true_start.
- iOS waits for the Recording started line on the recorder's stderr. simctl io recordVideo
writes that line once it has processed the first frame.
- Android waits for the device-side file's first byte. screenrecord opens its output empty.
It writes nothing until its muxer starts on the first encoded frame, so that first byte marks
the recording's origin.
- Android's wait runs on the device, in one bounded adb shell loop. The run clears the file
before the spawn, so a leftover recording cannot answer the wait.
- Android, where the wait cannot answer, falls back to the device-side process appearing. A
running process is no proof its encoder emits frames yet, so that signal is weaker. It still
beats a guess.
- The web actuator stamps true_start right after it creates the recording page, with no
poll. record_video_dir enables recording for the pages in a context. The video itself does not
exist until a page does, so the stamp waits for new_page() rather than new_context().
intervals.adopt carries true_start forward unchanged when it relocates a prestarted interval.
Android's confirmation, made before adopt even runs, is therefore not lost.
A poll that never confirms leaves true_start at None. The anchor then falls back to
scenario_start, never a guessed number.
Waiting on the recorder's own announcement rather than on its output file is load-bearing on iOS,
and was learned the hard way. simctl io recordVideo does not write progressively: the mp4 stays
at zero bytes for the whole recording, and simctl writes it in one piece at the end.
An earlier confirmation polled that file for growth, so it could never succeed. It spent its
entire ceiling on every scenario, green runs included. The iOS lane had raised that ceiling to 20
seconds on the belief that overshooting was free. simctl's own --help names the stderr line as
the signal to wait on. That line arrives in about 0.15 seconds.
How long that wait may run is still a knob, because startup jitter in simctl and adb is worse
on a loaded continuous-integration (CI) machine than on a developer's, and a wait that gives up
costs the whole scenario its correction. BAJUTSU_VIDEO_START_TIMEOUT (seconds) replaces the
5-second compiled default, the same treatment the three BAJUTSU_XCUITEST_* timeouts get. No lane
overrides it today.
- The finished recording places its own origin, and outranks the confirmation above. Every
true_start is a proxy: a recorder's own start line, a device-side process that has
appeared, a browser page that exists. Each signal arrives at its own distance from the frame the
recorder opens on, so a report anchored to one seeks off by that distance. The recording answers
the question itself. A finalized clip states its own duration, and Interval.stop() knows the
instant it ended, so the subtraction gives the origin outright:
measured_start = ended_at - duration. The duration comes from the container, with no external
tool: evidence/media.py reads the movie header that simctl and
screenrecord write, and the Matroska segment that Playwright's recorder writes.
Which instant "ended" names is the recorder's to say. A subprocess recorder stops when the signal
lands, then spends its finalize (and, on Android, a pull off the device) writing a clip it already
captured. Playwright instead films right through the context close that stop() performs. A
provider declares which shape it has through Interval.stops_when_stop_returns. run_scenario
then resolves video_start_offset after finish_scenario_intervals, preferring
measured_start and falling back to true_start, and records the result as
RunResult.video_anchor_s.
The subtraction is only as good as its two inputs, and each can be wrong in a way the other
cannot see.
- A duration is not always a wall-clock measure. That is a property of the recorder rather
than of the arithmetic. A container written at a nominal frame rate states more seconds than
the recorder was ever open for, as Playwright's short clips do. The origin then lands before
the spawn.
- ended_at is not always when the recording ended, because a recorder can stop itself.
Android's screenrecord quits at its own SCREENRECORD_TIME_LIMIT_S ceiling. A scenario that
outlasts the ceiling signals a recorder that stopped minutes earlier. The origin then lands
after the first frame by that whole gap.
A third issue sits in the recorder rather than in either input. The clip may stop short of the
stop signal. The simctl movie header spans spawn to stop, even over a static screen. Playwright
repeats its last frame until the context closes. Android's screenrecord behaves differently: it
encodes a frame merely when the screen changes, and its file ends at the last one. A still screen
before the stop thus makes the measured origin late by that whole stretch. One emulator run
measured 0.8 seconds. That error sits well inside the window below, so no bound could catch it.
Interval.measure_origin decides whether stop() runs the subtraction at all. On Android, the
anchors rank as follows:
- The first byte is the best anchor.
- The late measurement comes next.
- A pid instant comes last.
start_screenrecord turns the measurement off once the first-byte wait confirmed the start.
That recording keeps its first-byte true_start anchor. Where the wait could not answer, the
measurement stays on and still outranks the pid instant. intervals.adopt carries the flag
along.
Interval.spawned_at bounds both, because it is the one instant that needs no confirmation. A
recording opens on its first frame somewhere between that spawn and _ORIGIN_STARTUP_CEILING,
which is a claim about how long a recorder can take to open rather than about how patient a lane
chose to be — deliberately not BAJUTSU_VIDEO_START_TIMEOUT, whose 20-second iOS value had
silently widened this window to admit origins no recording can have. An origin outside the window
says one of the two inputs is not describing this recording, so it is discarded and that recording
keeps the true_start anchor.
- The recorded timestamps are absolute; a viewer derives the video-relative offset when it
renders.
run_scenarioreads the wall clock once, beside itstime.monotonic()stamp, giving the scenario an anchor pair: any later monotonic instanttbecomes the wall-clock instantscenario_wall_start + (t - scenario_start). Every step'sstarted_atand every network exchange'sstartedAttakes that form, so what survives the run is the raw timing data rather than a number a correction has already reduced. A viewer —report.html,bajutsu trace— subtractsvideo_anchor_sat render time to place an event on the recording's timeline. That is what makes the placement recomputable from a saved run: after a fix to how the anchor resolves, a re-render lands the events where they belong, with no need to run the scenario again (BE-0348). Every timing decision still reads the monotonic clock alone, since a wall clock can jump backward mid-run and would corrupt a wait's timeout or a step's duration. See reporting for what each recorded field means to a report reader.
Touch markers in the recording (--touch-markers)¶
A recording shows every consequence of a gesture and never the gesture itself, so bajutsu run
--touch-markers asks the app under test to draw a marker at each touch it receives: a translucent
circle at the contact, and a trail behind a contact that moves. Both are green while the touch is
down and red once it has lifted, so a single still frame says whether the contact it shows is the
gesture happening at that moment or the one it left behind. The marks are drawn inside the app's
own process, so they reach the recorded video and each step's after.png alike. Because they are
drawn from the UIEvent the app dequeues rather than from the coordinate the driver sent, a marker
is evidence that the touch was delivered, which a driver-side coordinate record cannot show.
Three properties matter before a reader turns the flag on.
- It needs an app that links BajutsuKit. The drawing lives in
BajutsuKit(BajutsuTouch), which the demo apps already link. The flag setsBAJUTSU_TOUCH_MARKERS=1on the app's launch environment, and an app that does not link BajutsuKit ignores the variable. - The marker is a
CALayer, so it never enters the accessibility tree. A layer is not aUIResponderand conforms to no accessibility protocol, so no selector can resolve to it and it can swallow no gesture. Two checks keep the claim honest:demos/showcase/scenarios/golden/golden_xcuitest.yamlasserts the same tree golden twice, once with the markers on and once with them off, so a visualization that perturbed the pinned controls fails on device; andtests/test_touch_markers.pyfails if the drawing code ever reaches for aUIView, which is the only way a marker could gain an accessibility representation at all. - A gesture's marks stay until the next gesture starts. No timer removes them, which is what
keeps them in the step's screenshot, and equally why a run with the flag on produces screenshots
that differ from a run without it. Where the in-app control channel is available (below), a
visualassertion gets its own capture with the markers hidden, so the flag and a pixel comparison can share a run; everywhere else, leave the flag off for a pixel comparison. Android has no such channel: its scenario lanes turnshow_touchesandpointer_locationon device-wide, and the tree-comparing and pixel-comparing lanes turn the pair back off by hand instead (demos/showcase/android/Makefile).
The markers are evidence only — no assertion reads them — and the flag is off by default for a
plain bajutsu run. The repository's own iOS lanes do pass it: .github/actions/bajutsu-e2e
and the showcase's run-swiftui / run-uikit targets run with the markers on, so a failure
there shows where the gesture landed. That is safe because visual is the only assertion kind
fed by a screenshot; every other kind reads the accessibility tree, the network exchanges, or
the clipboard. No lane here builds its app with -DBAJUTSU_ENABLE_CONTROL_CHANNEL today. Adding a
visual scenario to any of them fails until that build gains the setting (below).
The run loop hides the markers for the one capture a scenario's visual verdict compares,
because that assertion reads the very image the markers are drawn into. It asks the running app
to take the markers down, waits for the app to confirm, fires the shutter, then sends a second
command putting them back. Masking cannot rescue the case, because the marker follows the
gesture instead of sitting in a fixed region.
The request travels the in-app control channel (BE-0365). The channel is a command queue on the collector the app already reports to. The app opens no socket of its own, and the collector's per-run token guards the commands. The run loop waits for the app's acknowledgement rather than pausing for a fixed interval, and that wait makes the capture correct instead of hopeful.
The markers themselves never reach the verdict path. The channel that hides them does. Once armed, the
channel fails a scenario whose app never acknowledges a command. The second command is no
exception: it restores the markers after a capture that already succeeded. Both edges wait for an
acknowledgement on purpose, because the run loop must never report an unconfirmed state as a
confirmed one. From the moment the run loop arms the channel, --touch-markers bears on a
verdict.
The channel needs a real Simulator process, since that is where BajutsuKit's poll loop runs.
Carrying the channel takes the xcuitest actuator with network collection on. The other actuators
fall short:
fakestarts a collector, but nothing ever polls it.playwrightobserves network through the driver, so it starts no such collector at all.adbreports to the same receiver on the machine running bajutsu that iOS does, so it does start a collector. Nothing on the device answers a command. For that reason the actuator check keeps the channel toxcuitest, not the shape of the collector.
Where both conditions hold (the xcuitest actuator, with network collection on), --touch-markers
activates the channel for a scenario whose verdict reads a screenshot.
Everywhere else, a screenshot-comparing scenario keeps the pre-BE-0365 behavior. It draws no markers, and the investigator loses the touch evidence for that one scenario. That is the same trade-off this channel exists to remove, where it can.
--backend names one target's platform, never an ordered fallback list
(BE-0452).
No scenario escalates past the run-level actuator to one with no pre-started collector.
The unmarked fallback stays local to the scenario that took it. One scenario can arm the channel while the next stays unmarked. Between scenarios, the run relaunches the app with each scenario's own launch env. An unmarked scenario runs in a process the marker variable never reached. The rest of the same run still draws its markers.
Where the channel is active, the app side gates it twice, and one of the two is a build setting.
BajutsuKit compiles the channel out unless the build passes -DBAJUTSU_ENABLE_CONTROL_CHANNEL.
Compiled in, the channel stays inert without BAJUTSU_CONTROL_CHANNEL=1 on the launch environment.
--touch-markers supplies that key. An app built without the compilation setting acknowledges
nothing. The scenario then fails naming both gates, rather than comparing an image carrying the
markers.
The run loop reads those two launch-environment keys and nothing else, so it cannot tell a pair
the flag armed from one a scenario assembled itself. It reads them from the scenario's effective
launch environment. That environment merges the target's own subject.launch.env underneath the
scenario's own, in the same order the launch itself uses. A target-level pin counts the same as a
scenario's own pin.
Suppose the effective environment already sets BAJUTSU_CONTROL_CHANNEL=1. Adding the marker key
would complete the pair. Any such scenario whose expect runs against a baselines directory would
then enter the channel unasked. --touch-markers therefore leaves that scenario unmarked, and the
run says so on stderr.
Sinks (where evidence goes)¶
class EvidenceSink(Protocol):
def capture(self, driver, step_id, kinds, *, elements=None) -> list[Artifact]: ... # instant captures after a step
def wait_diagnostic(self, step_id, *, trace, elements) -> Artifact | None: ... # the first-wait timeout diagnostic (below)
def start_scenario_intervals(self, scenario_id, kinds) -> list[Interval]: ... # begin video / deviceLog / appTrace for the whole scenario
def finish_scenario_intervals(self, scenario_id, started) -> list[Artifact]: ... # stop them and collect the files
| Sink | Behavior |
|---|---|
NullSink (default) |
writes nothing (keeps a run side-effect-free) |
FileSink(run_dir, udid, log_predicate) |
writes under run_dir/<step_id>/ |
A capture the environment already began before launch (Android's video) is adopted rather than
started. On stop, the sink relocates its finalized file into the scenario dir.
Otherwise, interval captures come from the driver's driver_interval provider when it supplies
one (web's Playwright-native console / video, Android's adb logcat). When the driver supplies
none, FileSink takes the simctl path, which it skips when udid is absent. The CLI's run uses
FileSink(runs/<runId>, udid=..., log_predicate=...) (cli).
First-wait timeout diagnostic (BE-0231)¶
A wait for <element> that times out writes run_dir/<step_id>/wait-timeout.json
unconditionally, independent of capturePolicy. A timeout that no policy rule would have
captured thus still leaves the evidence needed to decide why it fired. The file is pure
diagnosis, never a verdict input. The run's pass/fail still comes from machine-checkable
assertions alone.
The file is self-contained so a rerun-to-green does not discard it:
| Field | What it answers |
|---|---|
readiness |
Whether the post-launch readiness gate had passed, on which signal (readyWhen, the target's subject.startWhen condition / namespace / count, or timeout), and whether the screen had settled — separates "the gate returned before the content" from "the content rendered but the awaited element did not". A settled: false says the gate returned while the tree was still changing, which is when a synthesized touch is dropped: read it as "the actuation before this wait may never have landed", since a dropped tap is still reported as delivered. null on a lane that carried no readiness result. |
trace |
The poll timeline: how many polls, when the tree first became non-empty (firstNonemptySeconds, null if it never did), and how many elements were present at the timeout — separating "nothing rendered / transient-empty" from "rendered, awaited element absent" from "slow cold-boot render". |
provenance |
A BE-0049 stamp (scenario hash, tool version, git revision), so the evidence stays identifiable independently of the run. Its scenarioHash fingerprints this scenario alone, without the file-level description the run manifest's scenarioHash folds in when present — so it can diverge from the manifest's hash even for a single-scenario run, not only for a suite/matrix run. |
elements |
The (redacted) element tree at the moment of timeout. |
It is recorded as an Artifact(kind="waitDiagnostic", provider="runner") — written by the run loop,
not a backend actuator.
App-crash evidence (BE-0424)¶
When the app under test itself terminates abnormally mid-scenario, the failing step's message names
the event and the scenario's evidence directory gains an app-crash/ subdirectory holding the
platform's own report for it. Without this, the failure reads as an ElementNotFound naming a
selector that should exist and normally does, so a contributor has to rule out a renamed id and an
unloaded screen before suspecting the app at all.
The evidence is a diagnostic, never a verdict. The scenario fails because its step failed, as it would for any other terminal step failure. Nothing here decides pass/fail (prime directive 1).
The crash is classified in-band, so the scenario finishes normally. The failing step's
screenshot, the running video recording, and any cleanup: on: failure rule land as they always do.
The backend-crash retry loop (crash-diagnostics/ above, BE-0421) never triggers. That loop names
a fault in the test infrastructure. This evidence names a likely defect in the app a team is
testing.
| Backend | What lands in app-crash/ |
How the crash is confirmed |
|---|---|---|
| iOS (XCUITest), Simulator only | the .ips report macOS's ReportCrash wrote for the faulting process, under its own name. Where no report exists, unified-log.txt instead: the Simulator's own unified log for the app's launch window, bounded to its last 256 KB. On the GitHub-hosted macOS runner, ReportCrash writes no report for a Simulator app at all |
XCUIApplication.state answering notRunning. On a real device (runsOn.ios.kind: device) nothing is confirmed: the same answer can mean an OS memory-pressure kill, and the device's reports never reach this host |
| Android (adb), API 30 and above | logcat-crash.txt, the crash buffer's own block for this process — managed (FATAL EXCEPTION) or native (Fatal signal) — plus tombstone_NN where the device allows adb root |
pidof reporting no process, corroborated by dumpsys activity exit-info reporting CRASH / CRASH_NATIVE at or after this launch. Below API 30 ApplicationExitInfo does not exist, so nothing is confirmed |
| web (Playwright) | nothing | left to a follow-up item |
The check is reactive. The runner asks a driver only after a step's own action, wait, or assertion has already failed, and never polls. Polling would add a round trip to every step of every green run, to catch a fault that is rare by construction. The reactive shape still catches the event at the exact step where it happened, since that step is already failing.
That shape has two blind spots. The design accepts both deliberately. An unconditional end-of-scenario probe would close them, but it would add the every-green-run cost the design exists to avoid.
- A crash caused by a scenario's last step can go unprobed when that step's own actuation still
reports success (a tap the runner delivered before the app died). A scenario that declares
neither
expectnor a non-failurecleanuprule then stays green. - The same shape with a scenario-level
expectgoes red but unclassified.expectproduces assertion results rather than step outcomes, so it never reaches the check. The reader sees an assertion mismatch instead of a crash.
A scenario written to exercise this signal takes one more step after the crash trigger. The later step, not the trigger itself, fails and gets probed.
A cleanup rule that runs on success or always is enough too. Its own teardown step does reach the
check, so the check catches the crash one step later rather than missing it.
A crawl records the same evidence under crashes/crash-NNN/app-crash/, beside that crash's own
repro scenario. A crawl's detection stays the accessibility-tree heuristic it already used, and a
covering system alert can trip that heuristic. A driver must positively confirm the event before
the crawl captures anything. An unconfirmed detection therefore records the crash with no
artifacts, rather than paying a bounded sweep on what may be an alert.
The Android tombstone layer is run-only. The adb root it needs restarts adbd. A crawl lane
could not recover from that restart, because the lane holds one environment for its entire walk.
Every write goes through the redacting text path, like the backend's own crash evidence above: a crash report is text a crashing app can echo a secret into.
Artifact provenance (provider)¶
Every piece of evidence is recorded as an Artifact(name, kind, provider, depicts), leaving in the
manifest which provider it came from and which screen it shows.
@dataclass
class Artifact:
name: str # filename (e.g. "before.png")
kind: str # "screenshot" / "elements" / "video" / "deviceLog" / "network" / "waitDiagnostic"
provider: str # who supplied this artifact (see table below)
depicts: str | None # the screen it shows, as "<driver>:<moment>" (see below)
provider value |
Meaning |
|---|---|
"driver" |
The actuator captured it directly (screenshots, element trees). |
"runner" |
The run loop wrote it (the first-wait timeout diagnostic, BE-0231). |
"simctl" |
Interval evidence from simctl (video, device log, app trace). |
"adb" |
Interval evidence from adb (screenrecord video, logcat device log). |
"collector" |
The app-side network collector (BAJUTSU_COLLECTOR). |
"playwright" |
Native Playwright network observation (web backend). |
"<backend> (fallback)" |
A read-only evidence fallback supplied the artifact (BE-0020). |
When an evidence kind cannot be supplied by any backend in the list, a SkippedCapture(kind,
reason) is recorded per scenario and disclosed in the manifest. The gap is never silently empty.
Each interval kind is a capability of its own (video, deviceLog, appTrace), judged against the
route the scenario runs on
(BE-0467):
| Route | video |
deviceLog |
appTrace |
|---|---|---|---|
| iOS Simulator | Recorded | Recorded | Recorded |
iOS real device (runsOn.ios.kind: device) |
Skipped | Skipped | Skipped |
| Live WebDriver endpoint | Skipped | Skipped | Skipped |
| Android | Recorded | Recorded | Recorded |
| Web | Recorded | Recorded | Skipped |
A gap is recorded for a kind the scenario asked for, and for no other kind. The sink never starts a recorder for a skipped kind: simctl reaches the Simulator alone, so the sink does not point it at a real device. Evidence never decides a verdict, so a skipped kind leaves the result unchanged.
Which screen an artifact shows (depicts)¶
A step's screenshot and its element tree are shown together, and a viewer draws a hovered element's
frame onto the image, so the two have to describe the same screen. depicts makes that checkable.
Its value, "<driver>:<moment>", names the driver whose read produced the file. It also names
which side of the step's action (before or after) the read came from. A native step's before.png carries
"xcuitest:before", and its after.png and tree carry "xcuitest:after".
Two artifacts describe the same screen exactly when their depicts values are equal. A consumer
compares the values and never parses them. Every consumer does the comparison in
evidence.step_view. The consumers are:
- the HTML report's element viewer
- the serve editor's element picker
- the triage context handed to a failure investigator
None of them can disagree about which screen a step "is".
evidence.step_view resolves a step to one screenshot, one tree, and whether the two are paired.
An unpaired step keeps its image and loses its frames. That is the honest rendering, because the
frames would land on pixels they never described.
Two situations produce an unpaired step:
- Inside a
webblock, the tree comes from the WebView, in the WebView's own coordinate space. The screenshot comes from the native driver, since aWebContextDrivercannot take one. - A run whose store no longer holds
after.pngfalls back to thebefore.pngbeside it. The post-action tree does not describe that image. Restoring a run from Trash produces this case, as does syncing a run into an object store that never received the last write.
A step that fails before it acts is neither case. It records its pre-action pair alone, and that pair matches.
depicts is absent from every run recorded before it existed (manifest.json schemaVersion 8 and
below). Nothing in such a manifest says which side of the action an artifact was taken on.
A consumer therefore reproduces the earlier choice, after.png over the before.png beside it,
and treats the pair as paired. This keeps the frames a stored run has always shown.
Visual evidence¶
A visual assertion produces a VisualEvidence record, which the manifest and the report carry.
The record holds the run-dir-relative paths to the baseline copy, the actual screenshot, and the
diff visualization (when the comparison found differences). It also holds two values:
diff_pct, the percentage of pixels that differedengine, the comparison engine that produced the verdict ("exact"or"pixelmatch"; BE-0165)
The engine is selected per assertion (compare:) with a target-level config fallback
(evidence.visualCompare), and is recorded in the manifest so the algorithm that produced each
verdict is traceable. Implementation: bajutsu/assertions/visual.py VisualEvidence.
Masking (redact)¶
Screenshots, logs, and network data can capture personally identifiable information (PII) and tokens. Declare what to mask before writing.
The run merges config's evidence.redact with the scenario's redact (union) (configuration). Implementation: scenario/models/evidence.py Redact.
redact:
labels: ["Card Number"] # accessibility labels
headers: ["X-Session"] # extra HTTP header names (on top of the defaults)
fields: ["token", "password"] # JSON/body field names
unmaskHeaders: ["authorization"] # opt out of a default (visible, deliberate)
unmaskSecureFields: true # opt out of the platform-marked default (below)
unmaskCredentialNames: true # opt out of the credential-name default (below)
Where redaction runs¶
Every write into a run directory goes through one sink, so an artifact is redacted because of where it lands rather than because its writer remembered to ask (BE-0331). The sink takes content before serialization, with one entry point per shape: an element tree, network exchanges, a crawl's screen map, and free text. Two of the rules below are structural. They read an element's trait, or its identifier and label. A JSON string no longer carries that pairing, so a sink that scanned serialized text could apply neither.
Content the sink cannot inspect (a screenshot, a video, an archive) goes through a separate entry
point. That entry point records the artifact as unmasked. An image is never described as protected.
Implementation: evidence/sink.py RunArtifactWriter.
The boundary governs writing alone. Reading a run stays unrestricted. The web UI, the evidence
readers, export, and the comparison commands all need to read, and none of those operations can
create an artifact.
Two mechanical checks keep the write side closed. An import contract lets the sink alone derive a writable run path. A second check fails a run-root path literal written anywhere else. Both read source alone, so a writer nobody has written yet is covered the moment it exists.
Masking that needs no configuration¶
Three rules mask without a redact: block, because each covers a case a scenario author should not
have to anticipate. A crawl carries no scenario at all, so these are the only rules that reach its
artifacts.
- A field the platform marks secret. An element whose backend reports the masked-input trait has
its value masked, and so does a value typed into such a field. Each backend derives the trait from
its own source: XCUITest's
secureTextField, the webinput[type=password], and the Android accessibility node'spasswordflag. The driver conformance suite (BE-0114) pins the trait on every backend, so the rule means the same thing on iOS, web, and Android. Release it withunmaskSecureFields: true. - A field whose identifier or label names a credential. The vocabulary is
password,passwd,secret,token,apikey,api_key,credential,otp, andpin, matched case-insensitively on word boundaries — so a field calledsettings.apikeyis masked and one calledprefs.pinnedis not. The list is small and documented rather than clever, because a rule an author cannot predict is one they cannot rely on. Release it withunmaskCredentialNames: true. - A recognizable credential shape. As a backstop, the sink masks a set of high-confidence shapes
in any text it writes. There are five: an Anthropic key (
sk-ant-); an Amazon Web Services (AWS) access key id (AKIA); a GitHub token (ghp_and its siblings); a three-segment JSON Web Token (JWT); and a Privacy-Enhanced Mail (PEM) private-key block. The sink masks a match and logs a warning naming the artifact, because a value reaching the backstop means an earlier, more precise rule should have caught it. The patterns are literal regular expressions, so no model is consulted. This rule alone needs to know neither a configured name nor the value in advance, which is why it alone can reach a value the tool itself generated. The leak that motivated this boundary was exactly that: acrawl --guide airun wrote an artifact after the model invented a realistic API key to fill a field.
What redaction does not guarantee¶
Masking reduces what a shared artifact reveals; it does not make a leak impossible. Three residues remain, and an operator deciding whether to share a report should weigh them.
- Pixels stay unmasked. A screenshot of a filled password field still shows it, which is why the runner warns up front rather than implying an image is protected (BE-0151).
- An arbitrary value in an unmarked field can survive. A secret typed into a field the platform
does not mark, whose identifier and label name nothing credential-like, that was never bound
through
${secrets.X}, and whose shape matches no backstop pattern, is indistinguishable from ordinary text. Telling the two apart would need semantic judgment, which never sits on this path. - The backstop's vocabulary is finite. A credential format nobody added a pattern for passes it.
- A configured key can over-mask a structured string. A
redact.headers/redact.fieldsname is also matched in thename: valueform. That form's value has no delimiter, so it runs to the end of the string. In a log line, that is right. Inside a bounded string an artifact carries, it consumes the rest of that string. Namingapprewrites an Android resource id likecom.example.app:id/logintocom.example.app:[REDACTED], and acrawl --continuereading such a screen map back replays a branch that resolves nothing.
Masking wins over legibility here on purpose, because a key an author named is one they meant.
Keep redact.fields to app body-field names, not words that also appear in a control's own
identifier.
Sensitive headers are masked by default (a scenario needs no
redact:for this): the built-in set isauthorization,proxy-authorization,cookie,set-cookie,x-api-key, andx-auth-token, matched case-insensitively.cookieandset-cookieare treated as one concern, so naming (or unmasking) either covers both. Header names inredact.headersadd to this set and never replace it. When a reader needs a default header's raw value (for example, debugging an auth failure), the reader names it underunmaskHeaders. Turning off protection is an explicit, visible choice, never the mere absence ofredact:.Redaction is applied before evidence is written (
evidence/redaction.pyRedactor):
- The device log and app trace are scrubbed by key→value patterns.
- The element tree masks a value when its label is configured (or scrubs an embedded secret).
- Each network exchange is masked structurally: header values by name, and the url / request / response bodies as free text, so query params and
token/passwordbody fields are caught.Images (screenshots / video) cannot be masked and are left as-is.
Redaction also extends to secret input values: the literal values behind
${secrets.X}(resolved from the environment, declared via config'sevidence.secretsconfiguration) are masked wherever they would appear in evidence, not just the configuredlabels/headers/fields. Redaction masks the longest values first, so a value that is a substring of another never leaves a partial leak.Value matching is encoding-aware: the same secret reaches evidence verbatim but often encoded, so its literal bytes never appear. Alongside the raw value, redaction masks its common encodings:
- percent-encoded (a URL query or form field, e.g.
p@ssasp%40ss)- HTML-escaped and JSON-escaped forms
- an
Authorization: Basic <base64(user:pass)>token whose decoded credential carries the valueThis is a fixed set of transforms applied to known values (the value is encoded, then searched for), not a decode-everything scan, so the cost and false-positive surface stay bounded.
One limitation remains. Where evidence is genuinely fragmented before redaction runs (a value split across streamed chunks that redaction never sees as one contiguous string), matching is best-effort. Assembled full-text evidence, the common case, is unaffected.
The executed scenario is also snapshotted into the run directory (
scenario.yaml, and the raw YAML view in the report). Atotpstep'ssecretis a durable base32 seed, not a one-time code, so a literal seed written straight into the scenario is masked to<redacted>in that snapshot. A${secrets.X}reference is kept as-is, because it is not the seed, and the secret-value rule above masks its resolved value. Prefer${secrets.X}for atotpseed, so the seed never sits in the scenario file to begin with.
File permissions¶
Redaction reduces what a leaked artifact reveals, but a best-effort denylist cannot promise more. Who can read the artifact matters too.
The runner creates each run directory with owner-only access (0700). The sensitive files a run may hold also get owner-only access (0600), whatever the umask of the machine running bajutsu says (BE-0131). Those files are network.json, the copied scenario.yaml, the element dump (elements.json), and screenshots.
Everything else lands under the 0700 run directory. On a shared machine (a CI runner, say), another local account cannot read a run's evidence by default. Implementation: bajutsu/common/run_meta/artifact_perms.py.