references/chromium.md
# Chromium (CDP) Targets
Everything in the related SKILL.md still applies; this page covers what differs.
A Chromium (CDP) target is an Electron app or a Chromium-family browser (Chrome/Brave/Edge) exposing a Chrome DevTools Protocol endpoint. `describe`, `gesture-tap`, `keyboard` and `screenshot` work as on a phone. What differs:
- Scroll with `gesture-scroll` and drag with `gesture-drag`; `gesture-swipe` is touch-only.
- Tabs and windows: `chromium-tabs`. Every other tool acts on the **active** tab, so `select` before driving a different one. A cross-process navigation (some redirects) can swap a tab's CDP target; `list` again to pick it up under a fresh id.
- Cookies, HttpOnly included: `chromium-cookies`. Web Storage: `chromium-storage`. Use them to seed auth before an e2e test or assert app state after one.
references/foldables.md
# Foldable Devices
Read this for a device `list-devices` marks `foldable: true`: a foldable iOS simulator (the iPhone Duo) or Android emulator. Everything in the related SKILL.md still applies, and the `fold` description covers postures, panels and the result fields. What it does not say:
- Keep one screenshot-diff baseline per posture: the screenshot size changes with the panel.
- Fold between gestures, not during one. A gesture stays on the panel where it started.
- iOS: if argent cannot find the active panel, it uses the cover panel and sets `warning` (a screenshot-diff summary gets a `panel:` line). Take a screenshot to see what the device shows, then do the check the warning names.
- Android: on a closed generic foldable AVD, tap only full-width rows; test the cover panel on a Pixel Fold AVD.
references/harmonyos.md
# HarmonyOS Targets
HarmonyOS targets exist only after `argent enable harmonyos`; with the flag off `list-devices` lists none and every `harmony-*` id is refused.
Everything in the related SKILL.md still applies. The HarmonyOS specifics of each tool (key and button sets, `durationMs` clamping, `rotation` refused, the suspended-panel `hint`/`note`, `open-url`'s `aa start -U` note) are in that tool's description. This page covers what spans tools.
## Ids and booting
- Drive the `kind: "device"` entry (`harmony-<connectKey>`). A `kind: "emulator"` entry (`harmony-emulator-<name>`) is only a `boot-device` target; `boot-device` returns the `harmony-<connectKey>` id to drive.
- A plugged-in phone (`state: "Connected"`) is listed as ready alongside booted simulators. Pick a `harmony` entry only when the user asked for HarmonyOS.
## What runs and what is refused
These run with the same tool names as on iOS and Android: `describe`, `screenshot`, `await-ui-element`, `await-screen-idle`, `gesture-tap`, `gesture-swipe`, `gesture-pinch`, `keyboard`, `button`, `launch-app`, `restart-app`, `reinstall-app`, `open-url`, `run-sequence`, `screenshot-diff`.
Refused as unsupported on harmony:
- `gesture-rotate` and `gesture-custom`: the device moves each contact along one straight line per call, so there is no arc and no long-press. There is no long-press tool on HarmonyOS.
- `gesture-drag` and `gesture-scroll` (Chromium-only), and `rotate`: the device has no orientation command.
- `native-describe-screen`, `native-view-at-point`, `native-user-interactable-view-at-point`, `debugger-component-tree`, the rest of the debugger (`debugger-evaluate`, `debugger-log-registry`), `view-network-logs` and the profilers.
- `paste`, so a `{{secret:...}}` placeholder goes through `keyboard` only.
- Screen recording: HarmonyOS exposes no readable frame stream. Use `screenshot` for a still frame.
So `describe` and `screenshot` are the whole discovery surface; when `describe` fails, `screenshot` is the only fallback. An empty tree during app startup is expected: call `describe` again rather than switching tools.
## Gotchas
- A suspended panel still dumps its last frame, so `await-ui-element` can succeed and `await-screen-idle` can settle on it (each with a `note`). That is not clearance to act: wake the panel with `button` `power` first.
- Text cannot contain a newline or another control character: type the text, then send `key: "enter"`.
- `reinstall-app` takes one `.hap` file; a multi-module `.app` bundle is not accepted.
## Building and signing a HAP
Both blocks below use the default macOS install, the only layout verified; elsewhere point `C` and the Java path at the matching DevEco Studio directories. The build assumes the module `entry` and the product `default`, which the output path depends on. Build from the project root without the IDE:
```bash
C=/Applications/DevEco-Studio.app/Contents
export DEVECO_SDK_HOME=$C/sdk PATH=$C/tools/node/bin:$C/tools/ohpm/bin:$C/jbr/Contents/Home/bin:$PATH
ohpm install --all
rm -rf entry/build/default/outputs
node $C/tools/hvigor/bin/hvigorw.js --mode module -p module=entry@default -p product=default -p buildMode=debug assembleHap --no-daemon
```
With no `signingConfigs` in `build-profile.json5` this writes `entry/build/default/outputs/default/entry-default-unsigned.hap`. A `signingConfigs` naming key files this machine lacks (DevEco writes absolute paths under its author's `~/.ohos/config`) fails the build at `SignHap`, e.g. `Invalid storeFile value`, after that HAP is written; sign it below. Any other failed build leaves no HAP.
The emulator installs any HAP, including an unsigned one or one whose signature fails verification, so an emulator install proves nothing about signing. A phone refuses an unsigned HAP (`9568320 error: no signature file`). Sign it with a debug profile for the app's bundle name that lists the phone's UDID (`hdc -t <connectKey> shell bm get --udid`, last line; `hdc` is in `$C/sdk/default/openharmony/toolchains`). `L` below is `$C/sdk/default/openharmony/toolchains/lib`, set in the signing block.
**Test keys, no account.** `$L/OpenHarmony.p12` (password `123456`) is OpenHarmony's public test PKI; use it for development only. A HAP signed this way installs and launches on a HarmonyOS 6.0 phone.
1. Certificate chain `app-chain.cer`, leaf first: the `bundle-info.development-certificate` from `$L/UnsgnedDebugProfileTemplate.json`, then the `OpenHarmony Application CA` and `OpenHarmony Application Root CA` certificates from `$L/OpenHarmonyProfileDebug.pem`.
2. Profile `profile.json`: a copy of the template with your `bundle-info.bundle-name`, the phone's UDID in `debug-info.device-ids`, and a current `validity` window in epoch seconds (the template's expired in January 2024).
3. Sign the profile, then the HAP, from the project root with DevEco's Java:
```bash
C=/Applications/DevEco-Studio.app/Contents L=$C/sdk/default/openharmony/toolchains/lib
export PATH=$C/jbr/Contents/Home/bin:$PATH
rm -f app-signed.hap
java -jar $L/hap-sign-tool.jar sign-profile -mode localSign -signAlg SHA256withECDSA \
-keystoreFile $L/OpenHarmony.p12 -keystorePwd 123456 -keyPwd 123456 \
-keyAlias "openharmony application profile debug" -profileCertFile $L/OpenHarmonyProfileDebug.pem \
-inFile profile.json -outFile profile.p7b &&
java -jar $L/hap-sign-tool.jar sign-app -mode localSign -signAlg SHA256withECDSA -signCode 1 \
-keystoreFile $L/OpenHarmony.p12 -keystorePwd 123456 -keyPwd 123456 \
-keyAlias "openharmony application release" -appCertFile app-chain.cer -profileFile profile.p7b \
-inFile entry/build/default/outputs/default/entry-default-unsigned.hap -outFile app-signed.hap
```
**Real keys, Huawei ID.** These need the user's Huawei account; ask before using it.
- DevEco Studio: sign in, connect the phone, and tick _Automatically generate signature_ under File > Project Structure > Project > Signing Configs. Tick _Support HarmonyOS_ too if it is offered; without it the signature is an OpenHarmony one. It writes `signingConfigs` into `build-profile.json5`, and the build above then emits `entry-default-signed.hap`.
- AppGallery Connect: create a `.p12` and CSR (DevEco: Build > Generate Key and CSR), then request a debug certificate (`.cer`) from the CSR, register the phone's UDID and the app, and request a debug profile (`.p7b`) for that app, that certificate and the device. Run the signing block without `sign-profile`, giving `sign-app` your `.p12`, key alias, passwords, `.cer` and `.p7b`; AppGallery Connect has already signed the profile.
In the signing block's shell, `java -jar $L/hap-sign-tool.jar verify-app -inFile <hap> -outCertChain c.cer -outProfile p.p7b` checks the signature on the host, but not that the profile's bundle name matches the HAP's. A phone refuses that mismatch with `9568329 verify signature failed`, so only an install proves the phone accepts a HAP. Install it with `reinstall-app` (`appPath` the HAP's absolute path, `bundleId` its bundle name), then `launch-app` it. `reinstall-app` uninstalls first, so a HAP the phone refuses also costs the installed copy and its data; to keep them, install with `hdc -t <connectKey> install <hap>` instead, which leaves the installed copy on a refusal.
## E2E tests
The e2e runner (`e2e-execute`, skill `argent-create-e2e`) runs on HarmonyOS only when you name the device: `device: "harmony-<connectKey>"` (`--device` on the CLI). A HarmonyOS device is never auto-resolved, and `platform` / `--platform` does not accept `harmony`. HarmonyOS is not an authorable platform either: `when: { platform: harmony }` and a `harmony:` launch-map key are rejected when the YAML is parsed, so launch with a bare app id or `native:`.
The runner has no UI tree on HarmonyOS (`describe` reads the `uitest` layout dump, but no runner tree source exists), so no selector resolves:
- Every selector step errors with a reason containing `ui-tree matching is not supported on platform "harmony"`: a selector `tap:` / `swipe:` / `pinch:`, `type:` (selector-only), `scroll-to:`, `await:` / `assert:` and `when:` UI conditions, and `snapshot:` with `cropOn:`.
- `await: { idle: true }` errors immediately, with no polling, since no relaunch can restore a tree that does not exist. It names `wait:` as the replacement and skips the rest of the run.
- `long-press:` and `rotate:` are refused up front, with or without a selector.
What runs: a coordinate `tap:`, a selector-less `swipe:` and `pinch:`, `launch:`, `snapshot:` without `cropOn:`, `wait:`, `echo:`, `script:`, and raw `tool:` steps for the tools that support HarmonyOS. For anything that needs a selector, run the scenario as a live interact-screenshot-verify loop instead.
SKILL.md
---
name: argent-device-interact
description: Interact with an iOS simulator or iPhone, Android emulator or phone, or Chromium (CDP) app using argent MCP tools. Use when tapping UI elements, performing gestures, scrolling/swiping, typing text, pressing hardware buttons, launching apps, opening URLs, taking screenshots, waiting for an element to appear or disappear, or checking visible app state after interactions. Not for TV targets.
---
Each tool's own description carries its parameters, defaults, return shape and examples. This skill covers choosing between tools and the gotchas those descriptions don't state.
## 1. Before You Start
- Get a target id from `list-devices`; if none is ready, `boot-device` one. Full setup flow: `argent-ios-simulator-setup` / `argent-ios-device-setup` / `argent-android-emulator-setup`. The same tool names drive iOS, Android and Chromium; each takes the `udid` from `list-devices`.
- Physical iPhone (`kind: "device"`): read `argent-ios-device-interact` first. Only the tools it lists exist there, and `describe` sees only the app `launch-app` registered.
- Read the matching reference before driving these targets: `references/chromium.md` for a `chromium` target, `references/harmonyos.md` for a `harmony` target, `references/foldables.md` for a device `list-devices` marks `foldable: true` (or before calling `fold`).
- Load a deferred tool's schema with ToolSearch before its first call (e.g. `gesture-pinch`, `gesture-rotate`, `gesture-custom`); without it, numbers go out as strings and fail validation.
- If you delegate device tasks to sub-agents, make sure they have MCP permissions.
## 2. Choosing the Right Tool
| Action | Tool | Notes |
| ------------------------------ | ------------------- | -------------------------------------------------------------- |
| Several known steps | `run-sequence` | One call, one screenshot at the end (§5) |
| Open an app | `launch-app` | **Never tap home-screen icons** |
| Restart an app | `restart-app` | Clean in-memory state |
| Open URL / deep link | `open-url` | Web pages, deep links, URL schemes |
| Tap | `gesture-tap` | Buttons, links, checkboxes |
| Scroll / swipe | `gesture-swipe` | Touch targets; not Chromium |
| Scroll (Chromium) | `gesture-scroll` | Mouse wheel |
| Drag (Chromium) | `gesture-drag` | Sliders, drag-and-drop, text selection |
| Long press, drag & drop | `gesture-custom` | Also any other non-linear touch sequence |
| Pinch / zoom | `gesture-pinch` | |
| Two-finger rotation | `gesture-rotate` | |
| Hardware key | `button` | Home, back, power, volume, appSwitch, actionButton |
| Type text or press a named key | `keyboard` | Every platform; text or one key per call (§6) |
| Paste text | `paste` | Only where a user would paste (§6). Sim/emu only |
| Rotate device | `rotate` | Orientation changes |
| Fold device | `fold` | Foldables only: `references/foldables.md` |
| Shake device | `shake` | Shake handlers (sim/emu only), Undo-typing prompt, RN dev menu |
| Wait for UI | `await-ui-element` | Block until an element is visible/hidden/exists/contains text |
| Wait for idle | `await-screen-idle` | Block until a non-empty screen tree stops changing (§4) |
## 3. Finding Tap Targets
After an action moves you to a different screen, or whenever you don't know an element's coordinates, discover before tapping. Interaction results already carry the element tree (`--- Elements after action (describe) ---`): use it, and call a discovery tool only when no fresh tree covers the current screen.
| Need | Tool |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| Any screen: app, system dialog, Home screen | `describe` |
| App-scoped UIKit properties (iOS simulator) | `native-describe-screen` (needs `bundleId`) |
| What sits at a known raw point (iOS simulator) | `native-user-interactable-view-at-point` (the view that receives touch) or `native-view-at-point` (the visually deepest view) |
| Final visual fallback | `screenshot`, only when the tools above cannot expose the control. Do not derive routine in-app navigation targets from pixels |
If `describe` returns an empty tree, follow its `hint`; with no hint, the screen is blank, loading or unlabeled: check the attached screenshot and retry once content has loaded.
For a React Native testID or component name that `describe` does not show, use `debugger-component-tree`; take tap points from `describe`.
**React Native:** LogBox toasts sit over the bottom of the screen and catch taps meant for what is under them. Argent hides them when a `debugger-*` tool connects and after `debugger-reload-metro`; after `restart-app`, call a `debugger-*` tool again. After any other reload they come back: close each by tapping the right end of its `describe` frame (the X), not its middle.
**System and permission dialogs:** on a simulator or emulator, `describe` returns a visible dialog's buttons with tap coordinates; keep using it for in-app modals too. Fall back to `screenshot` only if it does not expose the controls: then tap one obvious button (`Allow`, `OK`, `Don't Allow`, `Not Now`, `Continue`) at a time, check the returned screenshot, and go back to `describe` once the dialog is gone.
> **Prefer the dialog over the Settings tool.** When the app triggers its own permission prompt, answering it is the real user path. Use `settings-permissions` only when you can't make the change through the app: set a permission _before_ the app asks, re-enable one the user denied (iOS won't re-prompt), or reset it so the prompt reappears. See `argent-settings-permissions`.
## 4. Waiting: `await-ui-element` and `await-screen-idle`
**Never poll `screenshot`/`describe` in a loop to wait for something.** `await-ui-element` blocks server-side on the same tree `describe` reads. It has no bare-timer mode; for a plain pause, use your own harness sleep. Its description covers conditions, selector matching and defaults. What it does not tell you:
- A `hidden` check that succeeds **immediately** may be a false pass: its `note` then says the selector never matched anything. Treat that as a failed check and fix the selector; it does not mean "the element went away".
- The synthetic `ROOT` container `describe` prints is never matched, so a `role` like `AXGroup`/`html` won't trivially "match the screen".
- To disambiguate a loose selector, pin `role` to a text role like `StaticText`; that skips a same-named button.
`await-screen-idle` is for live diagnosis after a launch or navigation, before a raw tap on an element that may still be moving. Continue only when `settled: true`, and pair it with an `await-ui-element` for the destination: stillness does not identify a screen, and the tool can return during a presentation-layer animation. Do not record it in an e2e test; tests use `await: { idle: true }`, which also compares pixels.
## 5. Action Sequencing with `run-sequence`
Use `run-sequence` when every step is known in advance; if a step depends on what the previous one shows, use separate calls. Its description lists the allowed tools and their arguments. Typical batches:
- "scroll to bottom", "scroll until X": 3-5 swipes
- "clear and retype a field": triple-tap to select all, then type the new value
- typing then submitting: a `keyboard` text step, then a `keyboard` `enter` step (one call cannot carry both)
- "submit form": fill every field, then tap submit
- "go back to X": a known tap path
Gate a step that depends on a screen transition with an `await-ui-element` step rather than a fixed `delayMs`. If its condition is not met before the timeout, the sequence **stops there** and the later steps do not run, so a tap can't fire against a screen that never appeared. Any failing step also stops the sequence, with partial results.
## 6. Text Entry and Secrets
- Tap the field first. Typing or pasting with no focused field is a silent no-op.
- `paste` is **not** a faster `keyboard`. `keyboard` types the way a user types and stays the default for every text entry. Use `paste` only where a real user would paste: a 2FA / OTP code from another app, a long link or token, or testing how the app handles pasted input. That it can carry text `keyboard` can't type on a platform (multi-line text, non-ASCII on Android) is not a reason on its own.
- **Credentials:** type them as a `{{secret:<NAME>}}` placeholder in `text` (`keyboard`, `paste`, `run-sequence` keyboard steps, e2e `type` steps), never as plaintext, and never ask the user for the value. Do **not** `describe` or `screenshot` a non-secure field you just filled with a secret: submit or navigate away first, then verify the resulting screen.
## 7. Screenshots
Interaction tools already attach a screenshot. Call `screenshot` for the initial state, for a baseline before you change visible UI, to check state after a delay, or when the attached capture shows a transitional or loading frame.
Keep the default downscaled capture for UI context and state checks; do not use `scale: 1.0` as a readability or tapping aid. Full resolution is only for saving baseline/current PNGs for comparison (`scale: 1.0` with `includeImageInContext: false`); for visual regression checks use the `argent-screenshot-diff` skill.
If `screenshot` times out, restart the device transport with `stop-simulator-server`.
## 8. Android notes
- **First-launch permission prompts**: `reinstall-app` on Android installs with `-g`, so runtime permissions are pre-granted on first launch.