fix(devices): agents can test the mobile app on remote device hosts

.agents/skills/test-t3-mobile/SKILL.md

@@ -7,28 +7,67 @@ description: Test T3 Code's native iOS and Android app through its Device panel
 
 ## Open the device
 
-Call `device_list`, then `device_open` with the selected host and device IDs.
-T3 boots the device and shows its live stream in the Device panel. Follow its
-returned `quickStart`, using the exact `agentDevice.command` and all `targetArgs`
-on every operation. Use `device_screenshot` to inspect the screen.
+Call `device_list`, then `device_open` with both `hostId` and `deviceId`.
+Prefer a device on `local`; otherwise use a host whose platform is
+`available`, preferring a booted device. `device_open` boots the one you pick. A `ready` host with a `detail` warning is usable, and
+a null agent `runningVersion` before `device_open` is normal.
+
+Follow the returned `quickStart`. Store `agentDevice.command` in
+`agent_device_command` and `targetArgs` in the Bash array
+`agent_device_target_args`, and pass both on every AgentDevice call. Shell
+variables do not persist between tool calls, so set both at the top of each
+command that uses them. Later
+steps use the `device.id` that `device_open` returns: an Android AVD name
+becomes an `emulator-NNNN` serial when it boots.
+
+AgentDevice runs on this machine and carries each operation to the device
+host: `install` uploads a local file and `screenshot` writes a local file.
+T3 owns the SSH connection to device hosts, and a host id is a T3 name, not an
+SSH target. All work on a remote device goes through AgentDevice.
+
+If a `local` device fails to boot, run the printed emulator command once under
+`timeout 30` to read the cause. Report the cause in one line and open a device on another host. Host fixes
+such as `/dev/kvm` access are the user's call. Report the blocker and stop
+only when no host can open a device. Use only T3's device tools for automation.
+
+## Pick the layout
+
+The **layout** is where the device runs relative to this checkout:
+
+- **Same machine:** the device host is `local`. The device reaches this
+  machine's backend and Metro through loopback.
+- **Split:** the device host is another machine (SSH or EAS). On the device,
+  `127.0.0.1` and `10.0.2.2` point at that host, not here. T3 forwards device
+  control only, never Metro or the backend. The Tailscale addresses below
+  assume an SSH host on this tailnet; an EAS cloud device needs public Metro
+  and backend URLs, so report that blocker when none exist.
+
+## Start an isolated backend
+
+Reuse this task's healthy backend. Otherwise start one from the repository
+root with `vp run dev`, adding `--share` for split. Start
+each long-running process as its own process group, recorded in a file, and
+read ports and URLs from its log. This works on Linux and macOS:
 
-If T3 device tools or the selected device are unavailable, report the blocker
-and stop verification. Do not install or switch to another automation system.
-
-## Use an isolated backend
+```bash
+bash -c 'mkdir -p "$0"; set -m; vp run dev < /dev/null > "$0/backend.log" 2>&1 & echo $! > "$0/backend.pgid"' "$PWD/.t3/mobile-test"
+```
 
-Reuse this task's healthy backend. Otherwise run `vp run dev` from the
-repository root, retain its terminal session, and read the actual backend port
-from the dev-runner output. Use the worktree's ignored `.t3` state. Never run
-against `~/.t3/userdata`. The Browser panel is not required for this workflow.
+The server port and base dir are on the `[dev-runner]` line (`serverPort=`,
+`baseDir=`). For split, the backend origin is
+the `shared on tailnet:` URL; the printed `pairingUrl` belongs to the user. Use
+the worktree's ignored `.t3` state. Never run against `~/.t3/userdata`. The
+Browser panel is not required for this workflow.
 
 Test with meaningful project and thread data. Read the shared
 [SQLite fixture reference](../test-t3-app/references/sqlite-fixtures.md) only
 when inspecting or seeding SQLite. Stop the test server before fixture writes.
 
-## Launch T3 Code Dev
+## Install T3 Code Dev
 
-From the checkout being tested on the selected device host, run:
+Authorized mobile verification includes this step unless the user prohibits it.
+
+**Same machine.** From the repository root, run:
 
 ```bash
 vp run mobile:install <ios|android> <device-id>
@@ -37,36 +76,86 @@ vp run mobile:install <ios|android> <device-id>
 This installs the dev client for the checkout's native fingerprint: from the
 device, the local cache, or the shared cache in seconds, building once only
 when no machine has that fingerprint yet (5-15 minutes, logged to the file it
-names). The device id may be omitted when exactly one device is booted. It
-prints one JSON line; on failure it names the next command. Authorized mobile
-verification includes that step unless the user prohibits it.
-
-Start `vp run dev:client` from `apps/mobile`, or reuse a healthy Metro belonging
-to this checkout. Open its printed development-client URL with AgentDevice
-`open com.t3tools.t3code.dev <url>` and all returned target arguments.
-The device must be able to reach both Metro and the isolated backend.
-Append `&disableAutoLaunch=1&disableFab=1` to the development-client URL query
-when developer chrome would obscure screenshots or taps; the SDK 58 dev client
-applies these preferences before the app loads.
+names). It prints one JSON line; on failure it names the next command.
+
+**Split.** For Android, run AgentDevice `apps`. If it lists
+`com.t3tools.t3code.dev` and the change touches no native code, reuse it and
+report that its native fingerprint is unverified. Otherwise fetch or build the
+client file here, as a background task when it may build, then upload it:
+
+```bash
+vp run mobile:install android --artifact --abi arm64-v8a
+"$agent_device_command" install com.t3tools.t3code.dev <artifact> "${agent_device_target_args[@]}"
+```
+
+`<artifact>` is the `artifact` path in the JSON line. Use `arm64-v8a` for a
+`physical: true` phone or an emulator on a host that lists iOS `available` (a
+Mac), and `x86_64` for an emulator on a Linux host. If install fails with
+`NO_MATCHING_ABIS`, use the other ABI. `install` replaces an existing client and
+keeps its pairings; on `INSTALL_FAILED_UPDATE_INCOMPATIBLE`, uninstall it first,
+which clears them. iOS works the same way with
+`vp run mobile:install ios --artifact` when the local or shared cache has this
+fingerprint, or builds it as a background task on an Apple silicon Mac;
+otherwise it names the next step.
+
+## Start Metro
+
+Reuse a healthy Metro belonging to this checkout, or start one from the
+repository root the same way as the backend:
+
+```bash
+bash -c 'mkdir -p "$0"; set -m; (cd apps/mobile && exec vp run dev:client < /dev/null) > "$0/metro.log" 2>&1 & echo $! > "$0/metro.pgid"' "$PWD/.t3/mobile-test"
+```
+
+When another process owns port 8081, use a free port and leave that process
+running. For split, Metro must advertise this machine's `tailscale ip -4`:
+
+```bash
+REACT_NATIVE_PACKAGER_HOSTNAME=<tailscale-ip> bash -c 'mkdir -p "$0"; set -m; (cd apps/mobile && exec vp run dev:client --port <free-port> < /dev/null) > "$0/metro.log" 2>&1 & echo $! > "$0/metro.pgid"' "$PWD/.t3/mobile-test"
+```
+
+Open the development client with AgentDevice
+`open com.t3tools.t3code.dev <url>` and all target arguments, where the URL is
+`t3code-dev://expo-development-client/?url=<encoded Metro URL>&disableAutoLaunch=1&disableFab=1`.
+The Metro URL is `http://127.0.0.1:<metro-port>` on a same-machine iOS
+Simulator, `http://10.0.2.2:<metro-port>` on a same-machine Android emulator,
+and `http://<tailscale-ip>:<metro-port>` for split. The last two parameters
+hide developer chrome that blocks screenshots and taps. For a USB Android phone
+on this machine, run `adb -s <device-id> reverse tcp:<port> tcp:<port>`
+(`adb` is in the Android SDK's `platform-tools`) for the Metro and
+server ports, then use `http://127.0.0.1:<port>` for both.
 
 ## Pair and verify
 
-Use the helper from the repository root, with the returned executable and target
-arguments stored in `agent_device_command` and the Bash array
-`agent_device_target_args`:
+Use the helper from the repository root:
 
 ```bash
 .agents/skills/test-t3-mobile/scripts/pair-client.sh \
   <server-port> <base-dir> <device-reachable-backend-origin> \
   "$agent_device_command" "${agent_device_target_args[@]}"
 ```
 
-It issues a fresh credential and opens T3 Code Dev's existing pairing route
-through AgentDevice. For a backend on the device host, use
-`http://127.0.0.1:<server-port>` on iOS or `http://10.0.2.2:<server-port>`
-on Android. For a remote backend, use its reachable origin.
+It mints a fresh credential locally and opens T3 Code Dev's existing pairing
+route. For same machine, the origin is `http://127.0.0.1:<server-port>` on iOS
+and `http://10.0.2.2:<server-port>` on an Android emulator. For split, it is
+the `--share` origin.
 
 Confirm the intended projects appear, exercise the affected flow, and capture
-evidence. Retain the app and environment while iterating. At teardown, remove
-the disposable connection, close the AgentDevice session, call `device_close`,
-and stop only your backend and Metro processes.
+evidence. Verify with `snapshot -i` and `wait text "<expected label>"`. If `device_screenshot` fails, run
+AgentDevice `screenshot <file>` and read that file.
+
+## Teardown
+
+Keep the app, Metro, and backend running while the user inspects or iterates.
+The end of a turn is not teardown. At teardown:
+
+1. Remove the connection you added, the row whose URL is the origin you
+   paired: Settings, Environments, `Manage <label>`,
+   `Expand details for <label>`, `Remove environment`, then confirm `Remove`.
+   On iOS it is the trash button at the end of the expanded row. Leave T3 Code Dev installed for the next run.
+2. Run AgentDevice `close` with all target arguments, then call
+   `device_close`. Pass `shutdown: true` on EAS hosts, or for a device you
+   booted when the user wants it off.
+3. Stop each process group you started with
+   `kill -TERM -$(cat .t3/mobile-test/<name>.pgid)`, then confirm its ports
+   are free.

AGENTS.md

@@ -134,7 +134,7 @@ An empty database is a bad test. Seed your worktree's `.t3` with a copy of real
 
 - Upon request, user-visible frontend changes should get one integrated pass in a real client: `test-t3-app` for web, `test-t3-mobile` for mobile. The primary agent does this once after integrating. Subagents do not launch their own dev servers. Ask permission before doing computer use or spinning up browsers.
 
-For authorized mobile verification, a missing or outdated native client is an install step, not a blocker. Run `vp run mobile:install <ios|android> [device-id]` on the device host before starting Metro; it targets the only booted simulator or connected Android device when the id is omitted. It installs the dev client for this checkout's native fingerprint from the shared cache in seconds and builds only when no machine has built it yet, so prefer it over `expo run:*`, `xcodebuild`, or `gradlew`, which rebuild from scratch. Its stdout is one JSON line; build output goes to the log it names. See `test-t3-mobile` for the full workflow.
+For authorized mobile verification, a missing or outdated native client is an install step, not a blocker. When the device runs on this machine, run `vp run mobile:install <ios|android> [device-id]` before starting Metro; it targets the only booted simulator or connected Android device when the id is omitted. It installs the dev client for this checkout's native fingerprint from the shared cache in seconds and builds only when no machine has built it yet, so prefer it over `expo run:*`, `xcodebuild`, or `gradlew`, which rebuild from scratch. Its stdout is one JSON line; build output goes to the log it names. For a device on another host, `--artifact` (with `--abi <abi>` for Android) prints the client file to install with AgentDevice; see `test-t3-mobile` for that layout and the full workflow.
 
 ## Pull requests
 

apps/server/src/device/DeviceService.test.ts

@@ -39,6 +39,25 @@ const decodeJson = Schema.decodeUnknownSync(Schema.fromJsonString(Schema.Unknown
 const decodeBootError = Schema.decodeUnknownSync(Schema.fromJsonString(DeviceBootError));
 const encodeBootError = Schema.encodeSync(Schema.fromJsonString(DeviceBootError));
 
+const pngChunk = (type: string, length: number) => {
+  const chunk = new Uint8Array(12 + length);
+  new DataView(chunk.buffer).setUint32(0, length);
+  chunk.set(new TextEncoder().encode(type), 4);
+  return chunk;
+};
+const minimalPng = Uint8Array.from([
+  137,
+  80,
+  78,
+  71,
+  13,
+  10,
+  26,
+  10,
+  ...pngChunk("IHDR", 13),
+  ...pngChunk("IEND", 0),
+]);
+
 const fixture = Effect.fn("fixture")(function* (
   onBoot: Effect.Effect<void> = Effect.void,
   bootError?: string | { error?: string; errors?: Array<{ message: string; error?: string }> },
@@ -59,13 +78,14 @@ const fixture = Effect.fn("fixture")(function* (
   const devices: Array<Omit<DeviceSummary, "hostId">> = [];
   const discoveryErrors = yield* Ref.make<Array<{ message: string; error?: string }>>([]);
   const encoderResult = yield* Ref.make({ code: 0, stdout: " V....D libx264", stderr: "" });
+  const screenshotBody = yield* Ref.make(minimalPng);
   let booted = false;
   let shutDown = false;
   const http = HttpClient.make((request) =>
     Effect.gen(function* () {
       requests.push(request.url);
       if (request.url.includes("/api/screenshot")) {
-        return HttpClientResponse.fromWeb(request, new Response(new Uint8Array([137, 80, 78, 71])));
+        return HttpClientResponse.fromWeb(request, new Response(yield* Ref.get(screenshotBody)));
       }
       if (request.url.endsWith("/shutdown")) {
         shutDown = true;
@@ -229,6 +249,7 @@ const fixture = Effect.fn("fixture")(function* (
     settings,
     devices,
     encoderResult,
+    screenshotBody,
     discoveryErrors,
     endDevice,
   };
@@ -480,12 +501,37 @@ describe("device discovery after server restart", () => {
       expect((yield* service.state).devices).toEqual([]);
       const capture = yield* service.screenshot({ deviceId: DeviceId.make("Pixel_API_35") });
       expect(capture.device.id).toBe("Pixel_API_35");
-      expect(Array.from(capture.png)).toEqual([137, 80, 78, 71]);
+      expect(capture.png).toEqual(minimalPng);
       expect(requests.some((url) => url.endsWith("/api/devices"))).toBe(true);
     }).pipe(Effect.scoped),
   );
 });
 
+describe("device screenshots", () => {
+  it.effect("drops text that adb prints around the PNG", () =>
+    Effect.gen(function* () {
+      const { service, settings, screenshotBody } = yield* fixture();
+      yield* Ref.update(settings, (current) => ({ ...current, enableDeviceSupport: true }));
+      const warning = new TextEncoder().encode("[Warning] Multiple displays were found\n");
+      yield* Ref.set(screenshotBody, Uint8Array.from([...warning, ...minimalPng, 10]));
+      const capture = yield* service.screenshot({ deviceId: DeviceId.make("Pixel_API_35") });
+      expect(capture.png).toEqual(minimalPng);
+    }).pipe(Effect.scoped),
+  );
+
+  it.effect("fails instead of returning bytes that hold no complete PNG", () =>
+    Effect.gen(function* () {
+      const { service, settings, screenshotBody } = yield* fixture();
+      yield* Ref.update(settings, (current) => ({ ...current, enableDeviceSupport: true }));
+      yield* Ref.set(screenshotBody, minimalPng.subarray(0, 30));
+      const error = yield* Effect.flip(
+        service.screenshot({ deviceId: DeviceId.make("Pixel_API_35") }),
+      );
+      expect(error).toMatchObject({ _tag: "DeviceOperationError", reason: "hub_rejected" });
+    }).pipe(Effect.scoped),
+  );
+});
+
 const missingPlistMessage =
   "[apple-utils] Failed to read CoreSimulator plist at /Users/test/Library/Developer/CoreSimulator/Devices/deleted-simulator/device.plist:";
 

apps/server/src/device/DeviceService.ts

@@ -28,6 +28,7 @@ import {
   LOCAL_DEVICE_HOST_LABEL,
   type ThreadId,
 } from "@t3tools/contracts";
+import { extractPng } from "@t3tools/shared/png";
 import { SshCommandError } from "@t3tools/ssh/errors";
 import * as FileSystem from "effect/FileSystem";
 import { resolveNodeExecutable, nodeRuntimeUnavailableMessage } from "@t3tools/shared/nodeRuntime";
@@ -757,7 +758,16 @@ export const makeWithHosts = Effect.fn("DeviceService.makeWithHosts")(function*
   const screenshot: DeviceService["Service"]["screenshot"] = Effect.fn("DeviceService.screenshot")(
     function* (input) {
       const { ready, device } = yield* resolveDevice(input.hostId, input.deviceId);
-      return { device, png: yield* ready.devices.screenshot(device) };
+      const bytes = yield* ready.devices.screenshot(device);
+      const png = extractPng(bytes);
+      if (png === undefined) {
+        return yield* new DeviceOperationError({
+          operation: "screenshot",
+          reason: "hub_rejected",
+          cause: `The hub returned ${bytes.length} bytes without a complete PNG.`,
+        });
+      }
+      return { device, png };
     },
   );
 

apps/server/src/mcp/toolkits/device/handlers.ts

@@ -56,8 +56,8 @@ export function agentDeviceQuickStart(
     `  ${executable} screenshot /tmp/shot.png ${target}        # or call device_screenshot`,
     `  ${executable} install <app> <path-to-.app-or-.apk> ${target}`,
     `Prefer snapshot refs over coordinates. Run ${executable} help for workflow guides and ${executable} <command> --help for flags.`,
-    "Prefer agent-device for driving this device. simctl, adb, and xcrun remain available for anything it does not cover.",
-    "For remote hosts, arrange builds, app installation, and any Metro reverse forwarding yourself. T3 provides discovery, streaming, and control only.",
+    "Prefer agent-device for driving this device. On local hosts, simctl, adb, and xcrun remain available for anything it does not cover.",
+    "For remote hosts, build apps here and install them with this command; it uploads local files. The device reaches Metro and your backend over the network, not localhost. T3 provides discovery, streaming, and control only, and the host id is not an SSH target.",
     "Keep the returned --config and --session flags on every command. Other hosts can be used concurrently; opening one does not switch these commands.",
     platformNotes,
   ].join("\n");

packages/client-runtime/src/device/screenshot.ts

@@ -1,4 +1,5 @@
 // @effect-diagnostics globalFetch:off - This browser transport runs without an Effect runtime, like the live stream.
+import { extractPng } from "@t3tools/shared/png";
 import { withDeviceHubQuery } from "./hubAccess.ts";
 import type { DeviceStreamTarget } from "./stream.ts";
 
@@ -23,7 +24,7 @@ export async function captureDeviceScreenshot(target: DeviceStreamTarget, signal
     signal: AbortSignal.any([signal, AbortSignal.timeout(20_000)]),
   });
   if (!response.ok) throw new DeviceScreenshotError(response.status);
-  const image = await response.blob();
-  if (!image.size) throw new Error("The device returned an empty screenshot. Try again.");
-  return image;
+  const png = extractPng(new Uint8Array(await response.arrayBuffer()));
+  if (!png) throw new Error("The device returned an incomplete screenshot. Try again.");
+  return new Blob([png], { type: "image/png" });
 }

packages/shared/package.json

@@ -3,6 +3,10 @@
   "private": true,
   "type": "module",
   "exports": {
+    "./png": {
+      "types": "./src/png.ts",
+      "import": "./src/png.ts"
+    },
     "./changedFileKind": {
       "types": "./src/changedFileKind.ts",
       "import": "./src/changedFileKind.ts"

packages/shared/src/png.ts

@@ -0,0 +1,24 @@
+const PNG_SIGNATURE = [137, 80, 78, 71, 13, 10, 26, 10];
+
+/**
+ * Returns the first complete PNG in `bytes`, or undefined. `adb exec-out screencap -p` can print
+ * warnings, such as a foldable's multiple displays, around the image.
+ */
+export const extractPng = <Buffer extends ArrayBufferLike>(
+  bytes: Uint8Array<Buffer>,
+): Uint8Array<Buffer> | undefined => {
+  const start = bytes.findIndex((_, index) =>
+    PNG_SIGNATURE.every((value, offset) => bytes[index + offset] === value),
+  );
+  if (start < 0) return undefined;
+  const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+  for (let chunk = start + PNG_SIGNATURE.length; chunk + 12 <= bytes.length;) {
+    const end = chunk + 12 + view.getUint32(chunk);
+    if (end > bytes.length) return undefined;
+    if (String.fromCharCode(...bytes.subarray(chunk + 4, chunk + 8)) === "IEND") {
+      return bytes.subarray(start, end);
+    }
+    chunk = end;
+  }
+  return undefined;
+};