English · 日本語
Driver abstraction, backends, and environment management¶
One
Driverinterface sits in front of the backends:xcuitest(iOS Simulator),adb(Android emulator),playwright(web browser), and the in-memoryfakefor tests. The abstraction absorbs the capability differences between them. A platform-aware registry picks the actuator from the target'splatform. On iOS asimctlwrapper launches the app (boot/launch); on Android the twinadbwrapper does.Implementation:
bajutsu/common/drivers/(base.py/xcuitest.py/adb.py/playwright.py/fake.py) ·bajutsu/common/backends.py·bajutsu/common/backend_cli/simctl.py·bajutsu/common/backend_cli/adb.py.
Related: selectors (resolution) · the stability ladder · run-loop
Driver Protocol¶
Every backend satisfies this common interface (base.py, a runtime_checkable Protocol).
Only the actuator performs actions (tap/type/swipe/wait/query).
class Driver(Protocol):
def query(self) -> list[Element]: ... # the screen's element tree
def tap(self, sel: Selector) -> None: ...
def tap_point(self, p: Point) -> None: ... # raw coordinate tap (system alerts, etc.)
def long_press(self, sel: Selector, duration: float) -> None: ...
def swipe(self, frm: Point, to: Point) -> None: ... # a raw pointer drag (coordinate form)
def scroll(self, frm: Point, to: Point) -> None: ... # a non-inertial directional scroll (BE-0227, BE-0326)
def type_text(self, text: str) -> None: ...
def wait_for(self, sel: Selector) -> bool: ... # single-shot: matches the current screen?
def screenshot(self, path: str) -> None: ...
def capabilities(self) -> set[str]: ... # provided capabilities (for actuator / fallback resolution)
About
wait_for: the contract makes it single-shot (BE-0118). It checks the current screen once and returns, and never loops. One shared helper,base.wait_until, holds the deadline poll. A caller'stimeoutthus means the same real seconds on every backend, and no driver reimplements its own loop. The run loop's own condition waits work differently: the orchestrator pollsquery()directly (_wait, run-loop). Only callers outside that loop (for examplegolden_assert) usewait_until.
Capabilities (Capability)¶
capabilities() returns a set of tokens.
The set drives actuator selection and evidence fallback resolution.
It also drives the preflight capability check (below).
The two mobile backends differ in what they support. iOS and Android driver parity lists every difference and its plan.
| Capability | Meaning | xcuitest | adb | playwright | fake |
|---|---|---|---|---|---|
query |
element-tree query | ✅ | ✅ | ✅ | ✅ |
elements |
element-dump evidence | ✅ | ✅ | ✅ | ✅ |
screenshot |
screenshot | ✅ | ✅ | ✅ | ✅ |
semanticTap |
tap directly by id/label (no coordinates) | ✅ | — | ✅ | ✅ |
conditionWait |
native condition waiting | ✅ | — | ✅ | ✅ |
network |
native network monitoring | — | — | ✅ | — |
video |
whole-scenario video recording (BE-0467; Simulator only on iOS) | ✅ | ✅ | ✅ | — |
deviceLog |
whole-scenario device log or browser console stream (BE-0467; Simulator only on iOS) | ✅ | ✅ | ✅ | — |
appTrace |
whole-scenario app log trace: os_log on iOS, a tag-filtered logcat on Android (BE-0467, BE-0478; Simulator only on iOS) |
✅ | ✅ | — | — |
multiTouch |
two-finger gestures (pinch / rotate) | ✅ | ✅ | ✅ | ✅ |
webView |
query and tap the DOM inside a WebView the app embeds; granted per run, by the BajutsuKit WebView bridge on iOS (BE-0037) and over the Chrome DevTools Protocol on Android (BE-0482), so no backend declares it statically | — | — | — | — |
textSelection |
select-all + clipboard copy and paste on the focused field | ✅ | ✅ | ✅ | ✅ |
selectOption |
set a native <select> by value (web only) |
— | — | ✅ | ✅ |
urlNavigation |
open a URL mid-scenario with navigate (web only; BE-0464) |
— | — | ✅ | ✅ |
openUrl |
hand a URL to the operating system mid-scenario with openUrl (iOS and Android; BE-0487) |
✅ | ✅ | — | ✅ |
elementAttribute |
read one HTML attribute of an element for the attribute assertion (web only; BE-0466) |
— | — | ✅ | ✅ |
handleSystemAlert |
tap an iOS SpringBoard permission-prompt button natively | ✅ | — | — | ✅ |
resolveAlerts |
answer the policy's system alert inside the request that reads the screen (BE-0461; iOS only) | ✅ | — | — | — |
pickerWheel |
set a wheel-style picker to a named row (an Android NumberPicker through the resident server; BE-0480) |
✅ | ✅ | — | ✅ |
handleTipkitTip |
dismiss a blocking Apple TipKit tip (iOS only) | ✅ | — | — | ✅ |
appContext |
activate another installed app and drive it with the app: step |
✅ | ✅ | — | ✅ |
selectPhotos |
pick images from the system photo picker by position (PHPickerViewController on iOS, dropped on an Apple Silicon Simulator; the Android 13+ photo picker over adb) |
✅ | ✅ | — | ✅ |
handleNotificationBanner |
report and swipe away a foreground notification banner (BE-0416; iOS only) | ✅ | — | — | ✅ |
deviceControl.setLocation |
set the simulated GPS location | ✅ | ✅ | — | — |
deviceControl.clipboard |
read / write / clear the clipboard | ✅ | ✅ | — | — |
deviceControl.push |
deliver a push notification | ✅ | — | — | — |
deviceControl.clearKeychain |
clear the keychain | ✅ | — | — | — |
deviceControl.background |
send the app to the background | ✅ | ✅ | — | — |
deviceControl.foreground |
bring the app to the front, launching it if needed | ✅ | ✅ | — | — |
deviceControl.statusBar |
override / clear the status bar | ✅ | ✅ | — | — |
deviceControl.permissions.location |
grant or revoke the location permission (BE-0276) |
✅ | ✅ | — | — |
deviceControl.permissions.camera |
grant or revoke the camera permission (BE-0276) |
✅ | ✅ | — | — |
deviceControl.permissions.microphone |
grant or revoke the microphone permission (BE-0276) |
✅ | ✅ | — | — |
deviceControl.permissions.contacts |
grant or revoke the contacts permission (BE-0276) |
✅ | ✅ | — | — |
deviceControl.permissions.photos |
grant or revoke the photos permission (BE-0276) |
✅ | ✅ | — | — |
deviceControl.permissions.calendar |
grant or revoke the calendar permission (BE-0276) |
✅ | ✅ | — | — |
deviceControl.permissions.notifications |
grant or revoke the notifications permission (BE-0276) |
— | ✅ | — | — |
deviceGroup |
share one device between the targets of a device group (BE-0447; Simulator only on iOS) | ✅ | ✅ | — | ✅ |
The
deviceControl.*tokens are theDeviceControlfamily split per operation (BE-0212, from the coarsedeviceControlof BE-0128). A backend can thus advertise exactly the operations it can honor. XCUITest backs the whole family throughsimctl. The Android emulator backs every operation butpushandclearKeychain, all sevenpermissions.*services included (BE-0478).pushneeds a receiver inside the app's process, and Android has no keychain forclearKeychainto clear. The split makes that subset expressible without green-lighting the rest.backgroundandforegroundwere oneappLifecycletoken until BE-0447 split them, because a device group's later member comes up at itsforeground.adb sits at the lean end, actuating by frame-center coordinates. It exposes no semantic tap, so the run loop resolves a unique element via
query()and taps its center. XCUITest sits at the rich end: it taps directly by identifier, waits on native conditions, and performspinch/rotatenatively.adb advertises these capabilities:
query/elements/screenshot.multiTouch, a two-finger gesture the resident server injects, or a rooted-devicesendeventsweep (BE-0232).textSelection, throughCtrl+A/Ctrl+Cchords (BE-0265).- The emulator-backed device-control subset:
deviceControl.setLocation,deviceControl.clipboard(BE-0211),deviceControl.foreground,deviceControl.background, anddeviceControl.statusBar(BE-0478).- The seven
deviceControl.permissions.*services, throughpm grant/pm revoke(BE-0276).deviceGroup(BE-0447).appContext, theapp:step, througham start.- The
video/deviceLog/appTraceevidence kinds:screenrecord,logcat, and a tag-filteredlogcat(BE-0467, BE-0478).
push(no receiver in the app's process) andclearKeychain(no keychain on Android) stay unadvertised.The
fakedriver advertises a richer capability set (semanticTap / conditionWait / multiTouch) merely to exercise those code paths in tests.The
playwright(web) driver advertises these capabilities:
semanticTap/conditionWait, which Playwright has natively.network, making it the first backend with native network: it observes and stubs traffic in-process with no app-side cooperation.multiTouch, which it synthesizes for pinch / rotate through the Chromium DevTools protocol'sInput.dispatchTouchEvent(BE-0054).- The
videoanddeviceLogevidence kinds, taken from the browser context's recording and console (BE-0467).
Preflight capability check (BE-0082)¶
A backend's capability set is static.
The chosen actuator may lack a capability that a scenario needs. That is knowable before any device work.
The runner checks each scenario against the actuator's capabilities at run start (bajutsu/common/capability/capability_preflight.py).
The check runs after the actuator is selected and before the first device is leased.
It fails an unsupported scenario immediately, with one aggregated UnsupportedAction-style reason.
The scenario thus never boots a device and fails partway through (prime directive #2: fail fast and clearly).
The check is a pure function of (scenario, capability set), with no device and no clock. It works per scenario: only the offending scenarios fail, and the rest run.
The check gates only the hard requirements the capability set cleanly decides:
pinch/rotateneedmultiTouch.selectOptionneeds theselectOptiontoken. It is a web-only<select>switch, so iOS and Android are rejected before any device work.navigateneedsurlNavigation(web only). The check catches it even inside aweb:orapp:block.openUrlneeds theopenUrltoken (iOS and Android). The check catches it inside anapp:block too. The scenario model already refuses anopenUrlinside aweb:block when the file loads.navigate,openUrl, andattributeare the three constructs the check follows into those blocks.- An
attributeassertion needselementAttribute(web only; BE-0466). The check finds it anywhere. The places areexpect, a step'sassert, aniforinterruptscondition, aweb:orapp:block, andinterruptsrecovery steps. The iOS WebView context is thus rejected too. select/copy/pasteneedtextSelection(select-all + clipboard copy and paste). The web context is coordinate-only for these and refuses all three.delete/clearstay ungated, because every backend backsdelete_text.- A
visualassertion needsscreenshot. handleSystemAlertneeds thehandleSystemAlerttoken, which only xcuitest declares.setPickerValueneeds thepickerWheeltoken, which xcuitest and adb declare. A picker wheel is a native mobile control, so the web backend is rejected before any work starts.- Each device-control step needs the token for its own operation.
setLocationneedsdeviceControl.setLocation, the clipboard steps needdeviceControl.clipboard,pushneedsdeviceControl.push, and so on. BE-0212 split the coarsedeviceControlfamily of BE-0128 into these per-operation tokens. - A
permissionsentry is gated per service (deviceControl.permissions.<service>). An unsupported service is thus named individually, not the field as a whole. - Every run needs
query+elements.
The check deliberately gates neither of two capabilities:
conditionWait. The run loop polls for every wait, so no backend needs the token.network. XCUITest captures traffic through the app-side collector although it does not advertisenetwork. Therequest/event/requestSequence/responseSchemaassertions and theuntil: { request }waits thus run on iOS.
gestures.py's _require_multi_touch stays as a defense-in-depth check at gesture time.
_need_control stays as the equivalent for device-control steps.
It catches a run with no DeviceControl wired at all, for example a parallel run with no pinned device.
The tokens are per-operation.
A backend that supports only part of the family thus passes preflight for what it advertises. It fails fast for the rest.
The Android emulator, which supports
setLocation, clipboard, and foreground, is the example.
Each unsupported step is named
individually, so the family is not all-or-nothing.
Worker capability check (BE-0450)¶
The capability preflight above asks whether an actuator supports a scenario's steps. A second check (BE-0450) settles two more points:
- whether this machine can run the driver at all.
- the number of targets one scenario may hold at once.
A worker is the machine that runs scenarios.
A laptop that runs bajutsu run is a worker.
A bajutsu worker that leases jobs from a server is one too.
Host requirement. Each driver runs on a fixed set of host operating systems.
Each driver class
declares that set in a HOST_OS constant beside CAPABILITIES.
The worker reads its own operating
system from the machine.
| Driver | Runs on |
|---|---|
xcuitest |
macOS only |
adb |
any host |
playwright |
any host |
fake |
any host |
The worker capability file. worker.yaml describes one worker.
It stays apart from bajutsu.config.yaml.
An application team writes that config, but an operator knows what a machine can host.
The loader validates the file against version 1 of the schema.
It refuses an unknown
key, driver, or environment, and any number below one.
| Key | Meaning |
|---|---|
version |
The schema version. It must be 1. |
environment |
Where the worker runs jobs: local (the default), appium (a live Appium grid), devicefarm-host (the run inside a Device Farm package, whose device Device Farm booted and installed the app on), or a device-cloud provider kind. devicefarm is the one provider kind known today. |
appium |
Required with environment: appium, refused otherwise: the grid's endpoint URL and the one platform it serves (ios today). |
maxJobConcurrency |
How many jobs the worker may run at once, a positive integer. Omitted means one, and a local worker accepts one alone. |
maxTargetsPerJob |
How many targets one scenario may declare, a positive integer. The count spans every driver and device group, so a Simulator target and a browser target count two. Omitted means unbounded. |
drivers |
The drivers the worker offers, from xcuitest, adb, playwright, and fake. A driver the list omits is unavailable. |
realDevices |
Whether a local worker that offers xcuitest also drives attached real iPhones, so serve routes real-device jobs to it (capability routing, BE-0467). Omitted counts as true, as such a worker served real-device jobs before. Set false to keep a Simulator-only Mac off them. |
The package ships a default at bajutsu/common/capability/worker.yaml.
The default describes a
local worker that offers every driver and sets no target limit:
version: 1
environment: local
drivers: [xcuitest, adb, playwright, fake]
An appium environment replaces the target-level deviceProvider that
BE-0452 removed.
The file names the grid and the one platform it serves:
version: 1
environment: appium
appium:
endpoint: https://grid.example.com/wd/hub
platform: ios # the one value today
drivers: [xcuitest, adb, playwright, fake]
Targets of the grid's platform take the endpoint as their device. The scenario's other targets stay on this host. Bajutsu refuses a scenario with two grid targets.
bajutsu worker accepts --worker-config <path>.
So does each command that drives a device: run / record / crawl / repl / audit / doctor / bajutsu mcp.
The file it names replaces
the default whole, and the two files never merge.
A device-driving command refuses a file that names
a device-cloud provider.
--udid no longer accepts a URL; an appium environment names the grid
instead.
No environment variable or working-directory lookup picks a file.
The file in force is always the one
on the command line.
The host rule applies to a local worker alone.
A device-cloud worker runs the driver on the provider's host.
The check. bajutsu/common/capability/worker_capability.py loads and validates the file.
Then capability_preflight.worker_capability_unsupported() judges a scenario against the file
and the host.
That function needs no device and no clock.
It stays on the deterministic path.
Two
places apply the check:
- Actuator selection. Five commands share one selection step:
run,record,crawl,audit, andrepl. The target'splatformnames one actuator. The step refuses it when the host cannot run it or the worker file omits it. The command then exits 2 with a named reason. The old generic reason wasno available actuator. The new one readsworker-capability-unsupported: xcuitest runs only on macOS; this worker runs on linux. A command started without--worker-configuses the default file. - Per scenario. Before
runresolves any target, it checks each scenario's own declared targets. A scenario fails when it declares more targets thanmaxTargetsPerJob. It also fails when it needs a driver the worker cannot run. Its reason starts withworker-capability-unsupported:.runacquires no device for that scenario and runs the rest.runalso selects and checks every target's actuator before it acquires the first device. An unsupported target thus leaves nothing reserved.
Advertising and routing. A server worker turns the same facts into routing tokens
(self-hosting).
The worker advertises
host:<os> for the operating system it runs on.
Some jobs need one host operating system: every
driver their target can resolve to runs there alone.
Such a job requires that host.
An xcuitest
job thus requires host:darwin and reaches a Mac worker alone.
At startup, a local worker checks each --platform against the file and the host.
When either
rules the platform out, the worker exits with the driver's reason.
A Linux worker started with
--platform ios thus stops.
Its reason reads xcuitest runs only on macOS; this worker runs on
linux. The check judges the platform's drivers, not a request's backend. No worker off macOS
advertises platform:ios.
An iOS target run on the fake backend thus still needs a Mac worker.
A Device Farm worker advertises environment:devicefarm and kind:device.
It also advertises its provider's platforms, platform:android and platform:ios.
An appium worker advertises environment:appium.
A serve run request reaches it by naming environment: appium.
Every worker also advertises job:run-intent, which says it can run a run-set job (BE-0484).
An appium worker advertises the token too.
A run-set job requires a platform:* token, however, and such a worker does not advertise one.
A deployment of grid workers alone thus runs its scenarios through plain run requests.
The file declares no routing token itself.
The runtime and device-class tokens (ios18, ipad)
come from the Simulator inventory.
The real-device tokens (device:ipad) come from devicectl
(BE-0468).
A
declaration could promise a runtime or a device the machine lacks.
The kind tokens follow the same two reads (BE-0484).
A worker advertises kind:simulator when its Simulator inventory holds an iPhone or iPad.
It advertises kind:device while it drives a connected iPhone or iPad.
realDevices: false turns kind:device off.
Two hand-written inputs now log a deprecation notice. Each still works for this release:
- the worker's
--capabilitiesflag - the
$BAJUTSU_WORKER_CAPABILITIESenvironment variable
Neither input may carry a reserved prefix.
environment:, host:, platform:, and kind: are among them.
The worker fills those tokens itself.
BE-0452 removed a third input, a target's requires list.
A hosted run job now requires its platform and its driver's host alone.
A run-set job requires no host, since a worker on any host may take it
(self-hosting).
XCUITest (iOS)¶
XCUITest is the sole iOS backend since BE-0290
retired idb.
It reads the XCTest automation snapshot through a resident on-device runner (BajutsuKit). A loopback HTTP channel drives the runner.
It drives an arbitrary app by bundle id, with no
app-side integration.
Implementation: common/drivers/xcuitest.py.
XCUITest needs Xcode's
xcodebuild.
It sits at the rich end of the capability model. It offers semantic tap, native condition waiting, multi-touch, and text selection. It does not resolve through frame-center coordinates.
query(): reads the XCTest automation snapshot and maps each element to anElement. The snapshot descends into group containers. A coordinate backend's frame dump is flat. The snapshot instead renders a fully-expanded element tree.AXLabel/AXValue/accessibility identifier map tolabel/value/id.query()also reads a presentedSFSafariViewController. An app opens this in-app browser for a sign-in page or a terms document. It reads fromcom.apple.SafariViewService, the process that draws the browser. BE-0316 reads a SpringBoard prompt the same way (BE-0396).- The app's own snapshot reports that browser differently per iOS version. Through iOS 18 it mirrors the whole subtree. From iOS 26 it stops at the process boundary and reports nothing below it. The driver prunes the mirror and merges the service's own tree in its place. The result is one tree that is complete on both versions and reports nothing twice.
- The dismiss control is the one chrome identity the versions disagree on. iOS 26 identifies it
Close; iOS 18 leaves it unidentified with the labelDone. The runner reports iOS 26'sCloseas the identifier on both, soid: Closeaddresses it with one selector. Only the identifier is normalized. The label stays what the platform announces (Doneon iOS 18), so alabelselector still does not travel. - A browser element actuates at its frame centre.
XCUIElement.tap()reaches the page content across the process boundary. The browser's own chrome silently drops it. - iOS 18's disabled
ForwardButtonhas no iOS 26 counterpart at all. A scenario cannot depend on it. tap(sel):_resolveconfirms uniqueness (retries not-found, fails ambiguity fast). A real-device tree can be transiently empty during transitions. The driver then taps the element directly by its accessibility identifier — a semantic tap, no coordinates. BE-0289 re-resolves a stale snapshot handle and re-actuates only on a still-unique match.- A tap XCTest refuses takes one more step. iOS can report a container inflated over the control it wraps. The driver then probes the target's named descendants.
Where exactly one is reachable, it taps that descendant and records
substitution: soleHittableDescendant. Where none or several are reachable, it fails and names the candidates rather than choosing between them (selectors). wait_for: uses the runner's native condition waiting.pinch/rotate: two-finger multi-touch gestures performed natively by the runner.select/copy/paste: native text selection and clipboard paste on the focused field.screenshot:simctl io screenshot.enter_app/leave_app: theapp:step. Unlike theSFSafariViewControllermerge above, these never merge two trees.enter_apppushes a newXCUIApplication(bundleIdentifier:)onto the runner's own app stack and.activate()s it. Every route above (query(),tap, …) then addresses that app exclusively. This holds until a matchingleave_apppops back to the app beneath. The step is gated on theAPP_CONTEXTcapability, which XCUITest and adb declare (the adb section describes the adb mechanism). On XCUITest, a feasibility spike confirmed thatactivate()reliably foregrounds an app the test target never launched, with no per-app config or app-side cooperation (docs/specs/ios-cross-app-ui-control-feasibility.md).
The generic runner uses
XCUIApplication(bundleIdentifier:), so it drives any installed app with no app-side cooperation. A Simulator run needs no runner config at all. When a target names neitherdriver.runner.testRunnernordriver.runner.build, it resolves to the Simulator runner bundled in the wheel as package data (BE-0292). An explicittestRunnerorbuildstill overrides that default.runsOn.ios.kind: deviceresolves to the runner the user built and signed withbajutsu runner build --device(BE-0456), since Bajutsu cannot ship one signed for the operator's team.The backend is validated on-device (iPhone 17 Pro, recent iOS) via
make -C demos/showcase run-swiftuiand theios-e2e.ymlCI workflow. The XCUITest backend needs no pip extra, because Xcode suppliesxcodebuild.
adb (Android)¶
The adb backend is headless and coordinate-based, and it is the only coordinate backend.
With no
semantic tap, the abstraction resolves id → frame center → coordinate tap.
Implementation:
common/drivers/adb.py + bajutsu/common/backend_cli/adb.py (roadmap
BE-0007).
Reading the tree and resolving a selector¶
query(): reads the window's UI Automator XML. A pure parser (parse_hierarchy) maps each<node>to anElement.- Resident path. When the resident UI Automator server is built, the read runs over it (
make -C BajutsuAndroidUIAutomatorServer build). One warmUiAutomationsession answersGET /sourceoveradb forward. Each read then costs ≈ 0.1–0.3 s. A freshadb -s <serial> exec-out uiautomator dump /dev/ttypays ≈ 2.4 s per invocation instead (roadmap BE-0245). The resident whole-screen dump is narrowed to the active window, so it yields the same Elements. - Fallback. Without the built server, or on a channel failure the lease cannot recover from,
query()falls back touiautomator dump.BAJUTSU_ADB_RESIDENT(0/1) pins either path. - Bounded waits. Every wait on the device has a bound that fits inside the host's 5-second read timeout. A screen that never goes idle thus costs a read about 3.6 s, rather than the channel.
- Recovery order. A timeout does not show that the server is gone. The lease therefore reads again on a fresh connection, then restarts the server before it falls back. A refused or closed connection falls back at once.
- Later non-ASCII
typestep. The dump path cannot enter that text, so such a step starts a lost server again. - Settings and record. The target's
driver.residentRecoverysets the retries and whether a restart is allowed. The run'smanifest.jsonrecords any fallback. - Selector mapping.
resource-id→identifier. The<package>:id/prefix is stripped to the local name. A ComposetestTagsurfaced viatestTagsAsResourceIdthus reproduces verbatim, while a nativeandroid:iddrops its prefix.text→label(content-descfallback).content-desc→value(the app mirrors its state value there, SPEC §2.1).- The widget
class(plus enabled / selected / checked state) →traits. - Id matching. The local name is matched exactly.
The driver does no
.↔_rewriting, which would conflate distinct ids and erode determinism. A platform's native id syntax sometimes cannot reproduce the SPEC id verbatim. Android Views are the example:android:idallows neither.nor-, sostable.refreshsurfaces asstable_refresh. In that case the scenario carries both id forms in one selector —id: [stable.refresh, stable_refresh]— and the match is an OR over the candidates (BE-0221); see scenarios. - The app-side plumbing behind both forms ships as a standalone library,
IdentifierTool(BE-0405). The plumbing setstestTagsAsResourceIdinside every modal window and resolves theandroid:ida Views lookup needs. The showcase itself consumes the library, and the library carries no dependency onBajutsuAndroid. Declaring thoseandroid:idresources stays the consuming app's own job. - An editable text field reverses the
text/content-descmapping above, as on iOS. A node counts as one when its class isandroid.widget.EditText,android.widget.AutoCompleteTextView, orandroid.widget.MultiAutoCompleteTextView. A ViewsEditText, its AppCompat and Material subclasses, and a Compose text field all reportandroid.widget.EditText. - In such a field,
text→value, so avalueassertion and aclearstep read what was typed.content-desc(else thehint) →label, so a label selector keeps its meaning as the text changes. - An empty field reports its hint as
text. When the dump carries ahintattribute (the resident server's dump does), the driver identifies that case, and the field'svalueis then empty. - The
uiautomator dumpfallback may omithint. There an empty field's hint reads as itsvalue, andtextstays the field'slabel, the only name that path has for it. The dump also has noisShowingHintText, so typed text equal to the hint reads as novalue. - A masked field (
password="true") never exposes its text asvalueorlabel. tap(sel):_resolveconfirms uniqueness (retries not-found, fails ambiguity fast). A mid-transition dump is a transient null-root that is retried, and a 2+ match fails immediately.- Device-side actuation.
tap,long_press, anddouble_tapthen send the resolved element's identity toPOST /acton the resident server (roadmap BE-0339). The identity is the element's raw accessibility fields plus an ordinal, never a host-computed coordinate. The server re-resolves that identity against its own live tree and injects from the same warm session. A gesture thus lands on the bounds the device holds at the moment it injects, never on a coordinate the host computed one round trip earlier. stalereply. Astalereply means the identity's match count moved since the host counted it. The driver retries it, bounded.- Coordinate fallback. The driver falls back to a host-computed coordinate in three cases: the
retries are exhausted, the channel has no
/actendpoint (an older server), or the channel faults outright.tapthen usesadb shell input tapat the frame center.long_pressuses a same-point swipe held for the duration. - Lost reply. When the server injects but its reply never reaches the host, the driver counts the gesture as done. A second touch thus never lands on top of the first.
double_tap. The device path stamps bothMotionEvents from one server-side call. It declares a fixed interval between them. The gap does not depend on a round trip's incidental timing. See On-device actuation fidelity below for its coordinate fallback.swipe. It adds a finite duration, so it is a real drag.type_text. For ASCII text it isinput text, with spaces sent as its%sescape. A literal%ssplits between its two characters into separate calls, becauseinput texthas no escape for its own escape.- Non-ASCII text. It has no key in the command's character map and crashes the command.
That
text goes to the resident server's
POST /typeinstead, as a base64 body. The server builds the new value from the focused field's text and selection, then applies it withACTION_SET_TEXT. This route thus exercises no input method editor (IME). On that route, four cases fail the step and name the reason: a password field, a field without the action, no focused field, or no resident server. ASCII text reaches a password field throughinput textas usual.
Waiting for the tree to catch up with a pan¶
A coordinate resolve waits for the tree to catch up with a pan. Android moves the content first. It publishes the accessibility update naming the new frames afterwards. A read landing between those two moments describes the pre-scroll screen. Repeated reads then agree with each other on frames that are already wrong. The two-consecutive-equal-reads settle cannot detect the lag on its own. The tree is self-consistently stale rather than visibly unsettled.
The driver records the frame projection the screen had beforehand after each of these actuations:
- a
swipe, ascroll, apinch, or arotate— every gesture that moves frames wholesale - every
tap,longPress, ordoubleTap, device-side or coordinate alike (BE-0332)
The next coordinate resolve then re-reads until the projection moves off that record. The projection must then hold still briefly. A wall-clock budget bounds the wait. The driver announces spending it in full.
A tap can move the layout too: it can open a menu, expand a row, or advance a stepper.
The actuator that
follows a tap must thus resolve against the tree the device published after the tap, not the
pre-tap one.
tapPoint (raw coordinates) and back (no resolved target) do not arm the wait.
They have no target-from-a-layout to postdate.
The device confirms the gesture. The device-side POST /act path above answers the barrier's own question at its source.
A gesture the device confirms thus arms no barrier at all (BE-0339).
- After injecting, the resident server waits briefly on the accessibility event stream. Its warm session already observes that stream. It reports back the device-clock time of the first event that postdates the injection.
- A gesture confirmed that way has reached the tree before the host is told the gesture landed. That is exactly what the barrier would otherwise have waited for.
- A gesture the device cannot confirm within that window arms the barrier exactly as a coordinate injection does. Three unrelated causes land there together. The gesture moved no frame, or the publish was slower than the window, or the server is old enough never to have waited at all.
- The device gives the confirmation, and the driver never assumes it. A coordinate-resolving follower (
pinch,rotate, a directionalswipe/draganchor) has nostalere-resolve to self-heal with, unlike an identity-addressed follower. - A confirmed gesture also carries the tree the device dumped once that publish had landed. The read the next settle would open with is thus already answered. The driver adopts the tree after re-checking that the tree's own mark postdates the gesture (BE-0407 unit 19).
One budget. The barrier's own wall-clock budget, not the device's publish window, is the same
number the scroll loop uses to confirm an end of content before failing (ReadLagProvider,
BE-0326 / BE-0332; see architecture).
There is one publish lag, so one budget,
spent across those paths.
Directional swipe and drag. These two are the one exception to where the resolve happens.
The step's anchor element determines their endpoints. Code above the driver computes the endpoints.
The driver thus receives two coordinates rather than a selector, and cannot settle the tree
itself.
A backend that needs the settle exposes it (SettledReadProvider). The handler takes that read in place of a bare one.
A backend that does not implement the protocol keeps its single read.
Without that seam, two consecutive directional swipes anchor the second one on the first one's pre-pan
frames.
Three conditions make the test mean "caught up". The test must mean "caught up", not merely "different".
- The hold. The catch-up is not atomic. Android republishes node bounds one node at a time. A read landing mid-catch-up thus carries some new frames and some old. Two fast reads can both land inside that window and agree.
- Degenerate reads. The wait ignores a degenerate read outright. Its empty projection differs from every real one. Counting it would spend the budget on a tree the read path is still retrying.
- Re-reading the record. The wait re-reads the recorded projection when something has actuated since the last read. A baseline predating that actuation is worse than none. The first post-gesture read moves off it, and that move would count as the gesture being published.
A gesture still waiting to publish is drained before the next one's baseline is taken. Re-reading cannot rescue that case. The read would return the pre-gesture screen. The earlier gesture's publish would later be mistaken for the newer one's.
Every read counts toward the test, not only the ones the wait itself issues.
The reads the runner already takes between the gesture and the next actuator normally close it. Examples are a wait, an assert, and a post-step capture. A run whose tree keeps up waits for nothing.
What the wait fixed. This wait fixes the intermittent gestures flake on the continuous-integration emulator. There the tree withheld a 73px scroll for over a second.
The longPress aimed 10px past the target's bottom edge. The mirrored value thus stayed idle, even though the screenshots for those steps stayed pixel-identical.
Mid-scenario extract. The same publish lag reaches a mid-scenario extract (BE-0332).
A value
an action mirrors into the tree can land a beat after the action returns.
The first reads after the
action then agree with each other on the pre-action value.
extract.yaml bound a counter's previous value. The follow-up assert against the live one failed a correct run.
The settle poll cannot use the pan's "differs from the pre-step read" test here. An extract baseline is itself a single post-action read that can be stale.
It instead requires the agreeing read to postdate the
action by the budget.
Declaring a read lag is thus a contract. A backend that returns a non-zero read_lag()
takes on two obligations:
- Every coordinate resolve after a content-moving actuation may spend up to that budget waiting for the tree to catch up.
- Every mid-scenario
extractmay do the same.
The backend pays this on a read that still matches the pre-action screen, never on one that already landed. A backend that declares none keeps its single-read, fail-fast behavior unchanged.
The read mark. The resident reader publishes a read mark the host compares against (BE-0332 Units 3–4). The mark turns that ceiling into an early-releasing wait.
- The reader observes the accessibility event stream.
It stamps every
GET /sourcewith anX-Bajutsu-Read-Markheader, the device-clock time of the newest event as of the served dump. It also adds aGET /clockendpoint. - The driver takes a device-clock mark before each barrier-arming actuation.
It requires a read whose mark postdates the actuation (
read_postdates_actuation(), theReadOrderProviderseam). - The coordinate resolve's catch-up thus releases the instant the device publishes the action's update. There is no dwell, and no idling to the budget.
- Both marks come from the device's own clock, so no host-to-device skew enters.
The mark releases the coordinate barrier only. A mid-scenario extract keeps the wall-clock
budget.
The mark answers "an accessibility event postdates the gesture", while extract needs "the
property I am copying out has been republished".
One gesture produces several events.
Compose
publishes the tapped button's own event before the Text mirroring the new count recomposes.
A read
can thus postdate the tap, still carry the previous value, and agree with the read after it.
Ordering is the right question for frames, which the coordinate resolve waits on.
It is the wrong one for a value.
GET /source?since=<mark>. This request pushes the same ordering into the reader itself.
It
blocks until an accessibility event postdates the requested mark.
A bounded settle then closes tearing
before the reader answers.
The catch-up barrier's own dwell is thereby retired at its source. That dwell is the two-identical-dumps freshness check the barrier otherwise needs. Closing that barrier pays no second dump.
The budget survives only for the one-shot uiautomator dump fallback, which carries no mark.
The driver conformance suite (BE-0114) checks that marked-read contract against the real backend.
On a read-ordering backend, a read postdates a content-moving gesture.
A mark postdate does not prove rest. The mark proves the read is ordered after the gesture. It does not prove that the screen has stopped moving since, because a fling can keep publishing well past it.
The settle poll
layered on top (_settle(), above) thus deliberately does not treat a mark-closed catch-up alone
as proof of rest (roadmap
BE-XXXX).
On the
resident channel, a coordinate resolve still pays one confirmatory read (plus a short poll sleep)
after the catch-up barrier itself closes for free.
The barrier's own dump is free; the settle poll's
is not.
The settle also guards against the platform's own node cache (BE-0490).
- Android caches accessibility nodes per connection. The platform drops a cached node when an accessibility event reaches it, and at no other time.
- Most events name the node they drop. A window-level change clears every window at once.
- A change that publishes no such event leaves two dumps agreeing on the pre-change tree.
- On Android 14 and later, the resident server calls
UiAutomation.clearCache()between settle dumps. BothGET /sourceandPOST /actsettle this way, so the agreeing pair is a re-fetched tree. - Android 13 and earlier keep the earlier settle, because no supported call clears the cache there. A static target still settles in two dumps.
The Android fault-injection lane checks both outcomes on an Android 14 emulator.
Its fixture withholds the event by construction: a tap on one view changes what a second view reports.
The tap publishes TYPE_VIEW_CLICKED alone, and the cache refreshes the clicked view for that type.
The second view's old node stays cached, so the settled read returns the new value because of the drop and nothing else.
Without the drop, the read returns the old value.
The same lane recorded why the fixture cannot be a plain Compose element.
A Compose value published TYPE_WINDOW_CONTENT_CHANGED for its own change, which evicts the changed node without any drop.
On-device actuation¶
- On-device actuation fidelity (roadmap BE-0210):
- The
backstep is the true system back (input keyevent 4/KEYCODE_BACK). Android has no on-screen back element to tap, unlike iOS's OS back button. double_tap's primary path is the resident server'sPOST /act(see above). The server builds bothMotionEvents itself and stamps a declared interval between them. It does not trust a round trip's incidental timing to land inside the platform's double-tap window (roadmap BE-0339).- Without that channel,
double_tapfalls back to a host recipe. On a rooted device with a discoverable touchscreen, a raw two-slotsendeventsequence (BE-0208) narrows the gap between the two taps. The gap shrinks to five process spawns. Otherwise both taps go out in oneadb shellround trip (input tap … ; input tap …). The transport round trip itself then does not widen the gap past the double-tap window. - A tap whose target is not in the current viewport scrolls toward it (a default up-swipe) and re-queries. A retry count bounds the scrolling. This is a condition wait, so a selector that never appears still fails deterministically.
[!NOTE] This not-found scroll recovery is adb-only. XCUITest / Playwright still fail a
tapfast when the target is not in the initial viewport. Relying on the recovery thus makes atapon a below-the-fold element pass on Android yet fail on iOS/web for the same scenario.The portable way to reach an off-screen element is the explicit
scrollaction (BE-0326). It is one deterministic, non-inertial construct that reveals a target identically on iOS, Android, and web: writescroll: { to: <selector> }, then act on the target. It supersedes the hand-tunedswipechain the showcase fixture once used. The adb auto-scroll remains a robustness net undertapfor the not-found case specifically, not the portable idiom.A different, narrower safety net now covers every backend that can hit-test a point. The one exception is the app-embedded WebView bridge,
WebContextDriver, whose protocol exposes no hit-test. Before acting,tap/double_tap/long_presscheck that the resolved target is actually reachable at its own point, not covered by another on-screen element. The check uses the idiomatic signal each platform offers:
- iOS: native
isHittable.- Web: a
document.elementFromPointhit-test.- adb: a document-order geometric proxy,
Driver.is_tappable/topmost_at_point.When the check fails, the orchestrator takes a small, bounded scroll and retries the action once, rather than failing immediately. The scroll takes up to three
downsteps. Only ifdownnever clears the obstruction does it take up to sixupsteps. Theuplimit is widened becauseupmust first retrace the grounddownalready covered before it can make any net progress of its own. Seeselectors.md.This net is not a substitute for the explicit
scrollaction above: an author who already knows a target starts off-screen still writesscroll. It insures against an obstruction the author did not expect (a transient overlay, a sticky header settling into place), the same way adb's own not-found fallback already insures against an unexpected off-screen target. - Multi-touch (BE-0232):pinch_contacts/rotate_contactscompute the two contacts' geometry from the target's frame.rotatesweeps the straight chord between the endpoints, a linear approximation of the arc, like the web backend's rotate. Two routes inject those contacts, tried in order: 1. The resident UI Automator server'sPOST /gestureinjects both fingers as one multi-pointerMotionEventsequence throughUiAutomation, which holds the shell's privilege, so no root is needed. The record'sviaisresident. 2. A rooted device with a discoverable touchscreen falls back to a raw two-slot protocol-Bsendeventsweep. The fallback applies only when the server is absent, predates the endpoint, or could not be reached before the request went out. In those three cases nothing reached the screen. The record'sviaiscoordinate.
A gesture the server rejects fails the step rather than falling back, and a lost reply is taken as landed.
Either may already have put a finger down, and a second gesture on top of it would be worse than a loud failure.
A device with neither route fails with UnsupportedAction naming both preconditions.
There is no single-touch fallback, unlike the double-tap path above.
MULTI_TOUCH is declared statically in the capability set, so preflight admits gestures_multitouch on adb.
Which route is available is decided at actuation time.
Screenshots, lifecycle, and permissions¶
screenshotwrites the PNG bytes fromadb exec-out screencap -p(binary-clean stdout).- Lifecycle (
AndroidEnvironment, the twin of the iOSsimctlsequence): - Wait for boot readiness, polling
getprop sys.boot_completedto a bounded deadline. This is a condition wait, with no fixed sleep and no unboundedadb wait-for-deviceblock. - Optionally install the APK.
pm clearfor a clean state (theeraseequivalent).am force-stop.- Pre-grant runtime permissions (
pm grant, see below). am start. The package manager resolves the launcher activity. Launch env is forwarded as intent extras.- Open the deeplink (
am start -a android.intent.action.VIEW).
The run manifest records backend: "adb", so the selected actuator is disclosed.
- Runtime permissions (BE-0210): the permissions listed in the target's config
subject.grantPermissions are granted up front with adb shell pm grant <package> <permission> at
lease time.
The grant comes after pm clear (which resets grants) and before launch, so a runtime
permission prompt never blocks a scenario.
Granting deterministically up front, rather than tapping
the dialog when it appears, keeps timing off the run path.
The list is app-specific, so it lives in
config, not the driver.
- enter_app / leave_app: the app: step.
UI Automator already reads whichever app's window is in front, so entering another app is only a launch.
The driver resolves the package's launcher activity (cmd package resolve-activity --brief).
It starts that activity with the same am start -n the foreground step uses.
It then polls dumpsys activity activities until that package holds the resumed activity.
This is a bounded condition wait.
It raises ElementNotFound when it runs out, or at once when the package has no launcher activity.
The driver keeps its own app stack.
leave_app pops it and brings back the app beneath the same way, which is the app under test after the outermost block.
The app under test keeps its process behind the other app, so the crash probe (pidof plus exit-info, BE-0424) does not read the handoff as a crash.
On a device a device group shares (BE-0447), the front-app check expects the entered package while the block is open.
Evidence and network¶
- Interval evidence (BE-0007 Unit 4):
videorecords viaadb shell screenrecord, anddeviceLogstreamsadb logcat. They are the twins of the simctl providers.screenrecordwrites device-side (it cannot stream to a host file), so the driver finalizes the recording on SIGINT and pulls it off withadb pullon stop.logcatstreams to the file and stops on SIGTERM. Both are supplied through the same driverdriver_intervalseam the web backend uses, so the backend-independentcapturepolicy drives them unchanged (see evidence). - Network. The adb backend does not observe the network natively (no
NETWORKcapability). The story is the same mocked one as on iOS. The launch env forwards the app-side collector URL as an intent extra.mocksthus work with no new code path. - Device control. It backs
setLocationand the clipboard operations. The rest of the family stays unsupported.setLocationtakes the emulator console (emu geo fix, BE-0211) on a localemulator-<port>serial. Every other target mocks thegpsandnetworkproviders through BajutsuAndroid's location receiver (BE-0481). Such a target is a real device, or an emulator reached withadb connect. Adb grants the app the mock-location app-op first. The lease's end clears the mock and resets the app-op. - Clipboard. It runs through an in-app receiver (
BajutsuAndroid, BE-0233), notcmd clipboard. On-device, that command is a silent no-op. Since Android 10 only the foreground app / default IME may touch the clipboard. Bajutsu thus sends an orderedam broadcast, which a receiver inside the app handles from the app process. Base64 carries the data both ways, so the argv needs no quoting. A missing receiver fails loudly rather than reading an empty clip. adb still advertisesclipboard, because, like XCUITest's over simctl, the backend can drive it given a cooperating app. SeeBajutsuAndroid.
The XML attribute names follow UI Automator's
uiautomator dumpschema. The scenario side resolves the Viewsandroid:id.↔_case: a selector lists both id forms and matches either (BE-0221). The shared showcase scenarios thus run unchanged on both Android toolkits.android-e2e.ymlchecks this on every push/PR, drivingshowcase-composeandshowcase-viewsover the same set.The fast gate exercises the parser, the frame-center taps, the transient-empty retry, and ambiguous-fails-fast over captured XML fixtures. Install adb with
brew install android-platform-tools.
Embedded WebViews¶
A scenario's web: block reads and taps inside a WebView that the app embeds
(BE-0482).
The adb tree often shows a WebView's content too coarsely to select by.
The run reaches the page over the Chrome DevTools Protocol (CDP) instead.
With debugging enabled, each Android WebView listens on a local socket, webview_devtools_remote_<pid>.
That socket speaks CDP.
CdpDomSource implements the same DomSource interface as the iOS bridge client.
WebContextDriver thus runs unchanged on both platforms.
The app opens the socket.
A debuggable build calls Bajutsu.enableWebViewDebugging(context) right before it creates its first WebView.
The call loads the WebView provider.
From Application.onCreate it loads the provider on every launch.
An update to the WebView package can then relaunch an activity mid-scenario, even in a scenario that never opens a WebView.
BajutsuAndroid documents the call.
The helper does nothing in a non-debuggable build, so a release build never opens the socket.
The machine running bajutsu needs the android extra (websockets).
scripts/install.sh installs it when the configured backends include Android.
Each web: block opens one connection to the page.
The block closes it when it ends, pass or fail:
- The run lists the package's processes with
adb shell ps -A. It keeps each<package>:<name>child too, because a WebView can run in another process. - It finds those processes' sockets in
/proc/net/unix. It forwards a free local port to each socket withadb forward, then reads each socket's page list,/json/list. - It keeps the attached, visible pages.
It picks the page whose rectangle contains the frame that
withinresolved to in the native tree. Zero matches or two or more fail the step as a selector error. Two overlapping WebViews thus fail rather than let the run guess. - It opens a WebSocket to the chosen page. When the block ends, it removes every forward it opened.
Inside the block, each DomSource method maps to CDP calls:
DomSource method |
CDP calls |
|---|---|
query_dom |
Runtime.evaluate of the shared QUERY_JS, the same walk the web backend and the iOS bridge run |
tap_element |
A hit-test with document.elementFromPoint, then Input.dispatchTouchEvent (touchStart, touchEnd) at the point; the tap ends once the page's click lands, after at most three seconds (or the BAJUTSU_MIN_WAIT_TIMEOUT condition-wait floor, when that is longer); a page that cannot be read again within that bound fails the tap |
type_text |
Moves the caret to the end of the focused text field, sends Input.insertText, then dispatches change |
scroll_to |
Runtime.evaluate of a scrollIntoView script |
The tap is a trusted touch, so a page that listens for touch events reacts to it.
The iOS bridge clicks through a script instead.
The page turns the touch into a click a moment after touchEnd returns.
The click also focuses a field.
The tap waits for that click.
Without the wait, a type right after tapping a field would find nothing focused.
A page that cancels the click never sends one.
Once that bound passes, the tap ends with the touch alone and logs a warning.
Every transport error raises one exception type, WebViewTransportError.
It ends the block with its reason, not the run.
Three causes are common.
The socket is missing, CDP returns an error, or an app restart closes the socket.
The reason for a missing socket names the helper above.
Flutter (via the native backends)¶
The existing XCUITest / adb backends, unchanged, drive Flutter apps. Flutter adds no backend of its own (roadmap BE-0008).
Flutter renders its own pixels through Skia / Impeller, but the native backends never read pixels. They read the OS accessibility tree. Flutter maintains a semantics tree that its engine bridges into that tree:
- Android's
AccessibilityBridgeturns eachSemanticsNodeinto a virtualAccessibilityNodeInfo. - The iOS engine exposes
UIAccessibilityelements.
A widget that sets Semantics(identifier: …) thus surfaces as a resolvable element on both
backends.
A bounds-center tap lands via the semantics node's on-screen rect and Flutter's own
hit-testing.
The selector model, machine assertions, and the runner stay byte-for-byte unchanged.
The id convention sits alongside the iOS and Android ones above.
It needs Flutter 3.19+, when SemanticsProperties.identifier began mapping into the platform tree:
Selector field |
Flutter (via the native backend) |
|---|---|
id (primary) |
Semantics(identifier: "…") → accessibilityIdentifier (iOS) / resource-id (Android) |
label (auxiliary) |
the widget's semantics label (visible text) |
value |
the widget's semantics value (the state mirror, Semantics(value: …)) |
traits (role filter) |
the semantics role surfaced as the platform widget class / trait (button, selected, …) |
The app must meet two preconditions. Both concern Flutter's semantics state, not the renderer:
- Semantics is built lazily. Flutter constructs the tree only at one of two moments: an accessibility client connects, or the app calls
SemanticsBinding.instance.ensureSemantics(). On both backends the connection triggers construction on its own. Android's UI Automator connects as an accessibility service. On iOS, as this item verified on device, the XCUITest runner's accessibility query triggers it too. The driven path thus needs noensureSemantics()call. The showcase app keeps the call behind an off-by-default--dart-define=ENSURE_SEMANTICS=true. That flag is a documented fallback for an app that is driven without an accessibility client. - Only widgets that carry semantics appear. Standard Material / Cupertino widgets and text carry
semantics automatically.
A
CustomPaint-drawn control that is not wrapped inSemanticsnever enters the tree. Wrapping it inSemantics(identifier: …)is the same convention that surfaces the id. Flutter draws its own navigation chrome. The app thus also sets the back control's identifier to the platform conventionBackButton(base.OS_BACK_BUTTON). The iOS backend'sbackstep taps that identifier. On Android the system back key pops as usual.
Verified on device by the showcase-flutter (iOS, XCUITest) and showcase-flutter-android
(Android, adb) targets.
Both drive a Flutter twin of the native showcase apps
(demos/showcase/flutter), which the shared scenarios/ set drives
unchanged.
The verification covers these cases:
- id-based selectors.
valueassertions over the state mirror.- scroll-to-element over the lazily-built (culled) Notices list.
- native two-finger
pinch/rotate.
Run it with make -C demos/showcase run-flutter (iOS) / run-flutter-android (Android).
Out of scope (see the roadmap item for the reasoning):
- Features that need the in-app collector / receiver library the Flutter twin does not link.
Two capabilities depend on
BajutsuKit(iOS) /BajutsuAndroid(Android) being linked into the app. The Flutter app does not link them, to stay plugin-free: networkcapture andmocksroute app traffic through an in-app interceptor (BajutsuKitURLProtocolon iOS,BajutsuAndroid's OkHttp interceptor on Android). Flutter's DartHttpClientflows through neither, sonetworkevidence andmocksdo not observe Flutter traffic. The app's*.statusmirror still drives the deterministic wait/assert the scenarios rely on. Routing Dart HTTP through the native stack (viacupertino_http/cronet_http) is a follow-up.- The Android device-control
clipboardround-trips throughBajutsuAndroid's in-app receiver (BE-0233), so thesetClipboard/clipboardsteps ofdevice.yamlandtext_selection_android.yamlfail on the Flutter Android target. iOS device-control clipboard goes through simctl with no app cooperation. It works on the Flutter iOS target. This gap is Android-only.
Beyond these, the Flutter targets pass the same on-device scenario sets the native twins run. The exceptions are limited by the platform regardless of Flutter:
- Multi-touch (
gestures_multitouch) needs a rooted emulator on adb, as for the native Android apps. text_editingand thepushhalf ofdeviceare iOS-only in the native suite too.- Deeplink-to-tab routing. The Flutter targets register the per-flavor scheme (the Android
VIEWintent-filter, the iOSCFBundleURLTypes). The native twins route the URI to a tab: they select it, pop to root, and dismiss any modal. The Flutter app does not yet handle the incoming URI. The scheme is registered so the BE-0007 deeplink-actuation follow-up can driveam start -a VIEW -d <scheme>://<tab>/simctl openurl. The app-side handler lands with that slice. No shared scenario drives a literal-scheme deeplink today (navigation.yaml/notices.yamluse launch env and taps only), so the on-device verification is unaffected. - Flutter Web (CanvasKit). It paints to a canvas and surfaces no DOM elements. The Playwright backend cannot resolve them.
- The iOS
noaxtwin. The a11y build is the surfacing proof. A distinct no-identifier iOS bundle needs Flutter-flavor bundle-id separation, a follow-up. The Androidnoaxtwin ships (showcase-flutter-android-noax), built via a Gradle product flavor.
Playwright (web)¶
The Playwright backend drives headless Chromium via Playwright (Python).
It runs on Linux with no
Mac and no Simulator, so it fits the same toolchain as make check.
Implementation:
common/drivers/playwright.py (roadmap
BE-0041).
query(): oneevaluate()per document walks the visible / interactive / a11y-relevant DOM nodes, the main page first and then each iframe. A pure parser (parse_dom) maps each node to anElement. The id convention is the web equivalent of iOS accessibilityIdentifier:data-testid→Selector.id.- ARIA
role(or tag) →traits. - Accessible name /
aria-label/ text →label. valuecomes from each tag's own source:hreffor<a>,srcfor<img>, anddatetimefor<time>.- These elements report their live
valueproperty even when they also carry arole:<input>,<select>,<textarea>,<meter>, and a<progress>that carries avalueattribute. Theiraria-valuenowcan lag the property. - A
<progress>with novalueattribute, and any other element with an explicitrolesuch asrole="slider", reportaria-valuetext— oraria-valuenowwhenaria-valuetextis absent or empty. - Plain text:
query()also returns an element that holds its own text yet matches no rule above. A heading, a paragraph, and a<span>are typical. Its label is the element's text. Its one trait isstaticText, the trait iOS reports for a label, so a selector such as{ label: Welcome, traits: [staticText] }reads the same on both platforms. The iOS WebView bridge runs the same script, so a hybrid app's WebView reads plain text this way too. Two kinds of text stay out. Each would make a once-unique selector match twice: - Text inside an interactive control (a button, a link, or an input). The control's label already carries it.
- Text that repeats the label of its nearest ancestor with a
data-testidor arole.
Text elsewhere on the page still counts.
Take a page with a heading and a button that both read
"Sign in".
There tap: { label: Sign in } matches both and fails as ambiguous, as on iOS.
A text element's label also includes the text of the controls inside it.
Take a paragraph that
holds a "Terms" link.
Both the paragraph and the link match labelMatches: Terms.
A count
over that pattern grows by one.
Add traits: [button], within, or index to name the one you
mean.
Text elements count toward index too, in document order. A { label: Settings, index: 1 } used to skip a "Settings" heading. It now lands on that heading, without an error.
Pair index with
traits.
- Iframes: query() also reads every iframe's document, nested ones included.
Those
elements join the same snapshot.
- The parent page's contentDocument stops at a cross-origin iframe.
Playwright's frame tree
reaches one, so the driver walks that tree.
- The driver offsets each element by its iframe's content-box origin.
Every frame then reads in
main-page CSS pixels.
- An element scrolled outside its iframe's visible box is off screen and stays out.
- Some iframes are not read at all: those under any CSS transform but a pure translation.
A zoomed
device preview and a mirrored frame are examples.
Their inner coordinates would not match the
screen, so a tap would miss without an error.
- An element from a cross-origin iframe carries crossOriginFrame: true.
Selectors and assertions
treat it like any other element.
doctor's score and the crawl's screen model leave it out.
The app cannot instrument or explore an embedded third-party page.
- A selector resolves over the whole snapshot.
When two iframes share a data-testid, that
selector matches twice; index picks one.
An embedded third-party service counts too.
A
video player's "Share" button can make an app's { label: Share } ambiguous, and within or
traits narrows it back to the app.
- tap and selectOption hit-test with elementFromPoint inside the element's own document.
First, the driver checks that the point lands on each enclosing <iframe>.
An overlay above
the iframe raises ElementNotTappable, naming the overlay.
- tap(sel): like the adb backend, it resolves a unique element through the shared
resolve_unique/find_all against a query() snapshot and clicks the frame center by
coordinate (page.mouse.click).
It deliberately does not use Playwright's own
get_by_test_id().click(), so selector semantics stay byte-identical to every other backend.
- read_attributes(reads) (BE-0466) backs the attribute assertion.
It takes every
(sel, name) pair of an assertion block at once.
One page.evaluate runs the same walk as
QUERY_JS, with each record also carrying getAttribute for every requested name, so the whole
block reads from one instant.
Each sel resolves with the same resolve_unique a tap uses.
A pair
gets its value, None for a missing attribute, or the SelectorError its selector raised.
The
function returns that error rather than raising it, so the other pairs keep their reads.
Iframe,
covered, and off-viewport elements read like any other listed element.
The other backends
(XCUITest, adb, and the iOS WebView context) raise UnsupportedAction.
- type_text types via page.keyboard (the orchestrator taps into first, focusing the field).
screenshot is page.screenshot.
wait_for is single-shot via find_all, like every backend;
the shared base.wait_until supplies the deadline poll.
- The driver owns the lifecycle. A fresh BrowserContext is the erase equivalent. navigate() (page.goto(subject.url)) is the launch. close() tears the browser down.
There is no simctl
device, so the run uses a dummy lease and no device control.
- Device mode (BE-0228): a web target's runsOn.browser.emulate config selects how each
BrowserContext is created.
The value is desktop (the default when omitted, a plain desktop
context, unchanged from before) or a Playwright device preset name (for example iPhone 13).
- A preset is resolved against playwright.devices.
Its descriptor (viewport /
device_scale_factor / is_mobile / has_touch / user_agent) is merged into
new_context(**kwargs) alongside reduced_motion="reduce", so the target is driven as that
mobile device.
- The descriptor is resolved lazily (config load never imports Playwright) and memoized.
A
reset_context (crawl clean start) and a relaunch (BE-0077) thus rebuild the identical
context.
The mode is stable across the browser's whole lifecycle, the same invariant the engine
and reduced_motion already hold.
- An unknown preset fails loudly with a ValueError at driver start.
- Device mode is desktop-browser emulation. It gives a desktop-class browser a mobile viewport and touch input, as Chrome DevTools' device toolbar does.
It is not a real mobile browser on
a real device or a device cloud.
For a real mobile OS, the Android backend is the path.
- Directional swipe scrolls (BE-0227): the directional form swipe: { on, direction } means
"scroll", and a mouse drag does not scroll a web page.
The web backend thus dispatches the input primitive that actually scrolls. The context's input mode (the runsOn.browser.emulate above) picks the primitive.
- On a desktop (pointer) context it emits a page.mouse.wheel(...) over the gesture's start.
The wheel is the reverse of the travel.
An up swipe thus scrolls the page down, as a trackpad or wheel would.
- On a touch context (a mobile runsOn.browser.emulate) it uses a real single-finger touch drag over CDP, the same path pinch / rotate take. The page's touch and scroll listeners thus fire.
- The coordinate form swipe: { from, to } is unchanged. It stays a literal page.mouse drag. That is the raw-drag last resort for a canvas, a map pan, or a drag handle.
- codegen emits the desktop wheel scroll for the directional form.
A generated Playwright test
thus scrolls in the physically correct direction instead of the old inert drag.
It uses a
fixed default distance, as codegen has no viewport to scale amount against.
- The separate drag action (element-anchored pointer drag — a resize divider, a slider) routes to
the driver's swipe.
On web it is a real page.mouse drag that moves the grabbed element,
where a directional swipe would only scroll.
- Multi-touch (BE-0054): pinch / rotate are synthesized as two-finger drags via the Chromium
DevTools protocol (Input.dispatchTouchEvent).
mouse is single-pointer. Gestures thus go through CDP, the same path a real touch takes, so the page's touch listeners fire.
The element center
anchors the two fingers.
scale spreads/closes their gap, and radians rotates them about it.
- Native network (BE-0054): Playwright sees every request the page makes, so --network works
on web with no app-side cooperation.
network_collector() hooks the page's requestfinished event into the same NetworkExchange the iOS collector produces. request assertions and network.json evidence are thus unchanged.
A scenario's mocks are fulfilled in-process via
page.route: a matching request gets the canned response and is recorded with mocked: true.
Mock matching reuses the deterministic request matcher, and no model is consulted.
- Console / page-error & video evidence (BE-0054): the deviceLog capture kind streams the
browser console and uncaught page errors to <scenario>/device.log, and video records the whole
scenario.
Both are Playwright-native (no simctl), the web analogues of the iOS os_log / simctl video.
- The pool enables recording only when video is in the scenario's capture (the BrowserContext
is created with record_video_dir).
The video interval then finalizes it into
<scenario>/scenario.webm on close.
- Playwright's recorder always writes Matroska/WebM.
PlaywrightDriver.video_extension therefore
names the artifact after its real container, rather than the .mp4 every other backend produces
(simctl/adb write genuine ISO base media).
- The pool injects the driver's driver_interval (the driver-supplied interval seam, shared with
the adb backend) and video_extension into the FileSink.
The same backend-agnostic capture
policy carries both, while each artifact still names its own bytes honestly.
playwrightis imported lazily (only when a browser is actually started), so it never loads on the default CLI path (locked bytests/serve/test_import_guard.py). Install withuv sync --extra web+uv run playwright install chromium. The demo atdemos/web(make -C demos/web e2e) drives a tiny static web app end to end.
FakeDriver¶
FakeDriver is an in-memory implementation for testing the orchestrator / runner / record without a
device.
Implementation: common/drivers/fake.py.
- It holds a
screen(a list ofElement) and returns it fromquery(). tap/long_pressgo throughresolve_uniquelike the real thing (ambiguous / not-found =SelectorError).- A
reactcallback lets you script "the screen changes in response to an action." actionsrecords the performed actions (for assertions).
def react(driver, kind, arg):
if kind == "tap":
driver.screen = [...] # swap in the post-tap screen
FakeDriver(screen=[...], react=react)
Backend selection and the actuator¶
Implementation: bajutsu/common/backends.py.
PLATFORMS = { # a platform token expands to its actuators (stability order)
"ios": ("xcuitest",), # the sole iOS actuator since BE-0290 retired idb
"android": ("adb",), # the sole Android actuator (BE-0007)
"web": ("playwright",), # implemented (BE-0041)
"fake": ("fake",), # the in-memory test/demo driver
}
COST_ORDER: dict[str, tuple[str, ...]] = {} # empty: no platform's cost order differs from its stability order
IMPLEMENTED = {"fake", "playwright", "xcuitest", "adb"} # actuators with a driver today
def default_available(actuator) -> bool: # implemented + backing tool present (playwright: package import; fake: always)
def resolve_actuators(backends) -> list: # expand each token (platform or actuator) to actuators
def select_actuator(backends, available) -> str: # first implemented + available, in stability order
def select_actuator_cost_first(backends, available) -> str: # cheapest available, no scenario in hand (BE-0267)
def select_actuator_for_scenario(backends, scenario, available, caps) -> str: # cheapest available + sufficient (BE-0240)
def make_driver(actuator, udid, *, base_url=None, runner_port=None) -> Driver: # "xcuitest"→XcuitestDriver, "playwright"→PlaywrightDriver, "fake"→FakeDriver
- The platform decides the actuator. A target's required
platform(ios/android/web/fake) decides its actuator. Thebackend:config key no longer exists (BE-0452). Each platform today resolves to a single actuator:iostoxcuitest(BE-0290 retired idb),androidtoadb,webtoplaywright. --backend(and a serve request body'sbackend) is a check, not an ordered fallback list. It accepts the target's platform, that platform's actuator name, orfake, so--backend iosand--backend xcuitestmean the same. Anything else, or a comma list, exits 2.--backend fakeruns any target on the fake driver. That driver has no device to read, so it evaluates norunsOncondition and forms no runs. The invocation prints one notice saying so (configuration).serve --backendandprovision --backendkeep their own meaning.- The machinery for a multi-actuator platform (per-scenario resolution in cost order; BE-0240) stays in place for a future platform. No platform exercises it today.
- Two orderings answer two questions.
- Stability order (
PLATFORMS, most-capable-first; concepts) drivesselect_actuator. This is the availability-only pick. It serves where no scenario is in hand yet and cost doesn't matter:doctor, the pool's up-front setup, and an explicit single-actuator pin. - Cost order (
COST_ORDER, cheapest-first) drives bothselect_actuator_for_scenarioandselect_actuator_cost_first. The two share a candidate-resolution prefix (_cost_ordered_available). WithCOST_ORDERnow empty, a platform's cost order is just its stability order, so these fall through to a single candidate. select_actuator_for_scenarioadditionally reusescapability_preflight.unsupported(BE-0082) against each candidate's capability set. It returns the first candidate that is both available and sufficient for that scenario's steps.select_actuator_cost_firstis the same cost-first pick with no scenario to check against. It serves a live session that needs the cheapest actuator it can bring up, without capability escalation (serve's Author-tab Capture and Enrich; BE-0267).- Both delegate to
select_actuator(keeping its diagnostics) whenever the resolved candidates collapse to one. With every platform single-actuator today, that is always the case. - If no actuator is available, the selection raises
RuntimeError(the CLI exits with code 2). webresolves toplaywrightandandroidresolves toadb, both implemented (vision → reach).- The availability check
availableis injectable (swappable in tests). The default isshutil.whichfor PATH-backed actuators.playwrightis gated on whether its Python package is importable, andfakeis always available. - The actuator is fixed per scenario and held for that scenario's whole execution (BE-0240), so two drivers never operate one device at once. Fixing the actuator per scenario narrows the earlier "fixed per invocation" unit without relaxing the single-actuator rule. At every instant exactly one actuator acts on the leased device. There is never a mid-scenario driver swap.
Actuation stays with the single actuator. Non-actuator backends in the list can serve as read-only evidence fallbacks (DESIGN §9, BE-0020):
- A same-platform backend whose
capabilities()advertises a kind the actuator lacks (for exampleCapability.NETWORK) is resolved as the provider for that kind. - The narrow
EvidenceProviderProtocol is the only access to the fallback. It offers no tap/type/swipe, a type-level guarantee. - When no backend can fill a gap, the kind is skipped with a recorded reason (
SkippedCapture). This is graceful degradation, never a run failure. - Only
networkcan be filled this way. The interval kinds (video,deviceLog,appTrace, each a capability of its own since BE-0467) are judged against the actuator's route alone, and skip where it cannot record them.
See evidence — provider for provenance details.
Environment management (simctl)¶
Implementation: bajutsu/common/backend_cli/simctl.py.
Command builders are pure functions
(unit-tested), and execution goes through an injectable RunFn.
| Method | Command | Notes |
|---|---|---|
erase() |
simctl erase <udid> |
clean environment |
boot() |
simctl boot <udid> |
idempotent if already booted (swallows the error) |
launch(bundle, args, env) |
simctl launch --terminate-running-process <udid> <bundle> <args> |
env injected via SIMCTL_CHILD_* |
terminate(bundle) |
simctl terminate <udid> <bundle> |
ignored if not running |
openurl(url) |
simctl openurl <udid> <url> |
deeplink |
screenshot(path) |
simctl io <udid> screenshot <path> |
— |
Every call is bounded (BE-0363): the shared runner passes a deadline to every one-shot
simctlsubprocess. The command itself chooses the deadline: a long one for the commands whose duration the device or the app sets rather than simctl (bootstatus,boot,erase,install), a short one for everything else.A call that never returns is the observable symptom of a wedged CoreSimulator. It raises
simctl.DeviceTimeout, naming the command and the deadline it exceeded, instead of hanging until CI cancels the whole job with no cause.DeviceTimeoutsubclassesDeviceError, so a handler that already converts a device fault needs no change.
DeviceTimeoutalso subclasses the platform-neutraldevice_errors.DeviceTimeout(BE-0374), a second base rather than a replacement. The backend-agnostic run pipeline can thus act on a wedged host — failing the scenario instead of retrying onto it — without importing an iOS backend module to name the fault. The adb surface passes no deadline to any command yet, so it has no timeout type to place beneath that base; bounding it is a separate item's work.Where the timeout lands differs by caller. The best-effort probes (
device_booted,device_available,device_catalog, and the rest) fold it into their documented fallback and log it, so the recovery ladder still decides on what it observed. Every other call raises, including the idempotentshutdown/boot/uninstall/terminate, whose suppressions absorb a failing call and not a hanging one.Injecting launch env: an env var to pass to the app is set on the parent process as
SIMCTL_CHILD_<NAME>, which reaches the child (the app) as<NAME>.child_env()does this conversion. The showcase's launch hooks likeSHOWCASE_UITESTuse this mechanism (showcase).
The video / deviceLog interval captures also use simctl io recordVideo / simctl spawn log
stream, but those live in the evidence subsystem (evidence/intervals.py)
(evidence).