AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

playwright-interactive

Persistent browser and Electron interaction through `js_repl` for fast iterative UI debugging.

Original source
openai
License
Apache-2.0
Pinned revision
49f948faa9258a0c61caceaf225e179651397431

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read openai--playwright-interactive at revision 49f948faa9258a0c61caceaf225e179651397431, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 3

SKILL.md · 24000

ow:

await emitElectronScreenshotCssScaled({ electronApp });
await clickCssPoint({ surface: appWindow, x, y });

Clipped Electron region using CSS pixels from the renderer:

var clip = await appWindow.evaluate(() => {
  const rect = document.getElementById("board").getBoundingClientRect();
  return {
    x: Math.round(rect.x),
    y: Math.round(rect.y),
    width: Math.round(rect.width),
    height: Math.round(rect.height),
  };
});

await emitElectronScreenshotCssScaled({ electronApp, clip });
await clickCssPoint({ surface: appWindow, clip, x, y });

Raw Screenshot Exception Examples

Use these only when raw pixels matter more than CSS-coordinate alignment, such as Retina or DPI artifact debugging, pixel-accurate rendering inspection, or other fidelity-sensitive review.

Web desktop raw emit:

await codex.emitImage({
  bytes: await page.screenshot({ type: "jpeg", quality: 85 }),
  mimeType: "image/jpeg",
  detail: "original",
});

Electron raw emit:

await codex.emitImage({
  bytes: await appWindow.screenshot({ type: "jpeg", quality: 85 }),
  mimeType: "image/jpeg",
  detail: "original",
});

Mobile raw emit after the mobile web context is already running:

await codex.emitImage({
  bytes: await mobilePage.screenshot({ type: "jpeg", quality: 85 }),
  mimeType: "image/jpeg",
  detail: "original",
});

Viewport Fit Checks (Required)

Do not assume a screenshot is acceptable just because the main widget is visible. Before signoff, explicitly verify that the intended initial view matches the product requirement, using both screenshot review and numeric checks.

  • Define the intended initial view before signoff. For scrollable pages, this is the above-the-fold experience. For app-like shells, games, editors, dashboards, or tools, this is the full interactive surface plus the controls and status needed to use it.
  • Use screenshots as the primary evidence for fit. Numeric checks support the screenshots; they do not overrule visible clipping.
  • Signoff fails if any required visible region is clipped, cut off, obscured, or pushed outside the viewport in the intended initial view, even if page-level scroll metrics appear acceptable.
  • Scrolling is acceptable when the product is designed to scroll and the initial view still communicates the core experience and exposes the primary call to action or required starting context.
  • For fixed-shell interfaces, scrolling is not an acceptable workaround if it is needed to reach part of the primary interactive surface or essential controls.
  • Do not rely on document scroll metrics alone. Fixed-height shells, internal panes, and hidden-overflow containers can clip required UI while page-level scroll checks still look clean.
  • Check region bounds, not just document bounds. Verify that each required visible region fits within the viewport in the startup state.
  • For Electron or desktop apps, verify both the launched window size and placement and the renderer's initial visible layout before any manual resize or repositioning.
  • Passing viewport-fit checks only proves that the intended initial view is visible without unintended clipping or scrolling. It does not prove that the UI is visually correct or aesthetically successful.

Web or renderer check:

console.log(await page.evaluate(() => ({
  innerWidth: window.innerWidth,
  innerHeight: window.innerHeight,
  clientWidth: document.documentElement.clientWidth,
  clientHeight: document.documentElement.clientHeight,
  scrollWidth: document.documentElement.scrollWidth,
  scrollHeight: document.documentElement.scrollHeight,
  canScrollX: document.documentElement.scrollWidth > document.documentElement.clientWidth,
  canScrollY: document.documentElement.scrollHeight > document.documentElement.clientHeight,
})));

Electron check:

console.log(await appWindow.evaluate(() => ({
  innerWidth: window.innerWidth,
  innerHeight: window.innerHeight,
  clientWidth: document.documentElement.clientWidth,
  clientHeight: document.documentElement.clientHeight,
  scrollWidth: document.documentElement.scrollWidth,
  scrollHeight: document.documentElement.scrollHeight,
  canScrollX: document.documentElement.scrollWidth > document.documentElement.clientWidth,
  canScrollY: document.documentElement.scrollHeight > document.documentElement.clientHeight,
})));

Augment the numeric check with getBoundingClientRect() checks for the required visible regions in your specific UI when clipping is a realistic failure mode; document-level metrics alone are not sufficient for fixed shells.

Dev Server

For local web debugging, keep the app running in a persistent TTY session. Do not rely on one-shot background commands from a short-lived shell.

Use the project's normal start command, for example:

npm start

Before page.goto(...), verify the chosen port is listening and the app responds.

For Electron debugging, launch the app from js_repl through _electron.launch(...) so the same session owns the process. If the Electron renderer depends on a separate dev server (for example Vite or Next), keep that server running in a persistent TTY session and then relaunch or reload the Electron app from js_repl.

Cleanup

Only run cleanup when the task is actually finished:

  • This cleanup is manual. Exiting Codex, closing the terminal, or losing the js_repl session does not implicitly run electronApp.close(), context.close(), or browser.close().
  • For Electron specifically, assume the app may keep running if you leave the session without executing the cleanup cell first.
if (electronApp) {
  await electronApp.close().catch(() => {});
}

if (mobileContext) {
  await mobileContext.close().catch(() => {});
}

if (context) {
  await context.close().catch(() => {});
}

if (browser) {
  await browser.close().catch(() => {});
}

browser = undefined;
context = undefined;
page = undefined;
mobileContext = undefined;
mobilePage = undefined;
electronApp = undefined;
appWindow = undefined;

console.log("Playwright session closed");

If you plan to exit Codex immediately after debugging, run the cleanup cell first and wait for the "Playwright session closed" log before quitting.

Common Failure Modes

  • Cannot find module 'playwright': run the one-time setup in the current workspace and verify the import before using js_repl.
  • Playwright package is installed but the browser executable is missing: run npx playwright install chromium.
  • page.goto: net::ERR_CONNECTION_REFUSED: make sure the dev server is still running in a persistent TTY session, recheck the port, and prefer http://127.0.0.1:<port>.
  • electron.launch hangs, times out, or exits immediately: verify the local electron dependency, confirm the args target, and make sure any renderer dev server is already running before launch.
  • Identifier has already been declared: reuse the existing top-level bindings, choose a new name, or wrap the code in { ... }. Use js_repl_reset only when the kernel is genuinely stuck.
  • browserContext.newPage: Protocol error (Target.createTarget): Not supported while working with Electron: do not use appWindow.context().newPage() or electronApp.context().newPage() as a scratch page; use the Electron-specific screenshot normalization flow in the model-bound screenshots section.
  • js_repl timed out or reset: rerun the bootstrap cell and recreate the session with shorter, more focused cells.
  • Browser launch or network operations fail immediately: confirm the session was started with --sandbox danger-full-access and restart that way if needed.