AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

agent-device

Automates interactions for Apple-platform apps (iOS, tvOS, macOS) and Android devices. Use when navigating apps, taking snapshots/screenshots, tapping, typing, scrolling, or extracting UI info across mobile, TV, and desktop targets.

Original source
callstackincubator
License
MIT
Pinned revision
59e9d17446a248b21d86f078fcd0bd2d36db1ca4

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read callstackincubator--agent-device at revision 59e9d17446a248b21d86f078fcd0bd2d36db1ca4, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 8

references/macos-desktop.md

macOS Desktop

When to open this file

Open this file only when --platform macos is involved or the task needs frontmost-app, desktop, or menubar surfaces.

Main commands to reach for first

  • open <app> --platform macos
  • open --platform macos --surface frontmost-app|desktop|menubar
  • snapshot -i
  • get
  • is
  • click --button secondary

Most common mistake to avoid

Do not treat every macOS surface the same. Use the normal app surface when you want to act inside one app. Use frontmost-app, desktop, or menubar mainly to inspect what is visible before switching back to app for most interactions.

Canonical loop

agent-device open TextEdit --platform macos
agent-device snapshot
agent-device close

Surface rules

  • app: default surface and the normal choice for click, fill, press, scroll, screenshot, and record.
  • frontmost-app: inspect the currently focused app without naming it first.
  • desktop: inspect visible desktop windows across apps.
  • menubar: inspect the active app menu bar and system menu extras.

Use inspect-first surfaces to understand desktop-global UI, then switch back to app when you need to act in one app.

Snapshot expectations

  • snapshot -i should describe UI visible to a human.
  • desktop snapshots can include multiple windows from multiple apps.
  • menubar snapshots can include both app-menu items and system menu extras.
  • Finder-style rows, sidebar items, toolbar controls, search fields, and opened context menus should appear when visible.
  • Finder and other native apps may expose duplicate-looking row, cell, and child text nodes. Treat them as distinct AX nodes unless you have a stronger selector anchor.

Context menus

Context menus are not ambient UI. Open them explicitly, then re-snapshot.

agent-device click @e66 --button secondary --platform macos
agent-device snapshot -i

Expected loop:

  1. Snapshot visible content.
  2. Secondary-click the target item.
  3. Snapshot again.
  4. Interact with the new menu-item nodes.

Targeting rules

  • Prefer selectors or @ref values over raw coordinates.
  • On macOS, window position can vary across runs, so coordinate-only flows are fragile.
  • If the task only needs shared exploration rules, return to exploration.md.

Selector guidance:

  • Good selectors usually anchor on stable labels or app-owned identifiers such as label="Downloads" or role=menu-item label="Rename".
  • Avoid relying on framework-generated _NS:* identifiers as stable selectors.

Use snapshot --raw --platform macos only when debugging AX structure or collector filtering. Do not make raw snapshots the default agent loop.

Things not to rely on:

  • Mobile-only helpers such as install, reinstall, or push.
  • Desktop-global click or fill parity from desktop or menubar sessions.
  • Raw coordinate assumptions across runs.

Troubleshooting:

  • If visible content is missing from snapshot -i, re-snapshot after the UI settles.
  • If desktop is too broad, retry with frontmost-app.
  • If menubar is missing the expected menu, make the app frontmost first and retry.
  • If the wrong menu opened, retry secondary-clicking the row or cell wrapper rather than the nested text node.
  • If the app has multiple windows, make the correct window frontmost before relying on refs.