Playwright test agents: the healer vs a screenshot test
Playwright test agents plan, generate and heal tests. We ran all three on a real shop, then asked the healer to fix a failing screenshot test ten times.

On this page
In six of ten runs, the Playwright healer agent made a failing screenshot test pass by making it stop checking the screenshot. Four times it marked the test test.fixme(), which skips it. Twice it commented out the toHaveScreenshot line. Two more times it fixed the app's CSS instead, and twice it gave up. That was the last experiment of a day spent running all three Playwright test agents on a real demo shop, and the earlier parts went very well.
Playwright test agents are three agent definitions that ship with Playwright: a planner that explores your app and writes a Markdown test plan, a generator that turns the plan into Playwright tests, and a healer that runs failing tests and repairs them. They arrived in Playwright 1.56 on 2025-10-06 and run inside a coding agent such as Claude Code, Codex, or VS Code's Copilot. We ran them on Playwright 1.64.1 with Claude Code 2.1.316 on 2026-11-02, against a local copy of our Shoppy demo storefront.
How do you set up Playwright test agents?
One command, npx playwright init-agents --loop=claude, run in a project that already has Playwright Test. It took under a second and wrote six files: the three agent definitions in .claude/agents/, an .mcp.json that starts Playwright's test MCP server, a specs/ folder for plans, and an empty tests/seed.spec.ts. The seed test is where you put whatever every test needs first; ours got one line, await page.goto('/').
Two details differ from most guides. The docs list four loops, but the 1.64.1 CLI accepts six: claude, codex, copilot, opencode, vscode, and vscode-legacy. And for the Claude loop the definitions go to .claude/agents/, not the .github/ folder the docs show for VS Code. Playwright also asks you to regenerate the definitions after each upgrade, since the agents' tools change with the release.
The planner: 19 scenarios in under two minutes
We asked the planner for a plan covering the core shopping flow: browse and filter, configure a product, add to cart, apply a promo code. It explored the shop through the browser for 1 minute 48 seconds, 28 turns, $0.23 at list prices, and saved a 9.7 KB Markdown plan with three suites and 19 scenarios, each with steps, expected results, and a target file name. It did not take a single screenshot; its instructions say to avoid them.
The plan was better than a checklist. It noticed that the cart empties on a full page reload, so tests must navigate inside the app; that an unknown promo code silently removes the discount with no error message; and it flagged one flow to check as a possible bug. One practical note: our first attempt ran the planner as a background subagent from an ordinary session, the session ended, and the subagent never finished. Running it directly with claude -p --agent playwright-test-planner worked.
The generator: tests that passed first time
The generator took two scenarios from the plan, the quantity stepper and applying the SPRING15 code. It ran each step live in the browser, read the log of what worked, and wrote one test per file with a comment per plan step and role-based locators. That took 1 minute 22 seconds. Its own summary said it had not run the files. We did, without editing anything, and both passed on the first run.
The healer on a broken locator
Next we staged the kind of change the healer is built for: the promo button's label went from Apply to Redeem, and the generated test timed out waiting for a button named Apply. We gave the healer the prompt from Playwright's docs, "Fix the failing test", and named the test. It ran the test, read the error context, changed one locator from Apply to Redeem, reran it, and finished in 59 seconds for five cents. It even pointed out that it had left the test's title saying Apply. For locator drift, the healer is very good.
That matches what an engineering manager at a SaaS company described about their team: a few hundred end-to-end tests, a hard cap of about ten minutes for the run, and the healer fixing broken tests most of the time, with people still checking that the product does what it should.
What does the healer do when a screenshot test fails?
It looks at the images, makes a reasonable diagnosis, and then does whatever makes the suite pass with the tools it has. We added one screenshot assertion to the generated promo test, toHaveScreenshot on the order summary panel, recorded the baseline, and confirmed three stable passes. Then we staged two different changes, each committed with a neutral message. The first was a regression: the summary rows lost justify-content: space-between, so every price slid next to its label. The second was an intended redesign: the heading became Your order and the total got larger.
We gave the healer the same prompt five times for each change, from a clean working tree each time. In every run it opened the expected, actual, and diff images and described the change correctly. What it did next varied from run to run on identical input. On the regression it restored the CSS twice and marked the test test.fixme() three times, with a comment explaining the layout bug it had found. On the redesign it marked the test fixme once, gave up twice, and twice commented out the screenshot assertion and added two text checks in its place.
None of this is the healer misbehaving. Its definition tells it to fix assertions and expected values, to mark a test test.fixme() when it believes the test is right and the app is wrong, and, in its last line, to "do the most reasonable thing possible to pass the test" without asking questions. Its tools include Edit and Write but no shell, and its test-run tool cannot pass --update-snapshots, so updating the baseline was never an option. Two runs said so in as many words. One of the redesign runs summed up its own result: "The visual coverage is gone until someone regenerates the baseline."
Would an agent with a shell do better?
In our runs it made the right call every time, and that is its own lesson. We gave the same prompt to a plain Claude Code session with a shell, three times per change. On the regression it reverted the CSS three of three times and left the baseline alone. On the redesign it ran --update-snapshots three of three times and the suite went green. All of them read the commit to decide what was intended.
Look at where each green outcome came from. A skipped test, a deleted assertion, a reverted stylesheet, a regenerated baseline: every one is a change to the repository, made by the agent, in the same pull request as the UI change it was judging. When a commit message is honest, a capable agent reads it and gets the answer right. The visual question, whether this screen should look like this now, is still being answered by the party that changed the screen.
Where the visual baseline should live
The fix is structural: keep the screenshot baseline and the decision to accept a change out of the repository the agent edits. UI Verify, our visual regression testing tool for coding agents, works with the tests the agents just wrote. Swap the @playwright/test import for @uiverify/playwright and every page those tests open is captured and diffed against a baseline held outside your repo; a baseline only moves when someone accepts the change in review, a person in the dashboard or an agent over MCP acting for the team.
That changes what each of the healer's moves can do. There is no baseline PNG in the working tree to regenerate and no toHaveScreenshot line to comment out. A test that fails is still captured, so its page shows up in the build as a change. And a capture that existed on main but stops arriving shows up in the pull request's visual changeset as removed, so a skipped test is visible in review instead of silently green. The Playwright quickstart has the full setup.
Use the healer for what it does well, which is keeping locators current, and let the visual check be the one test it cannot rewrite. For the principle behind that split, see agent-written tests need a check the agent can't edit, and for the baseline workflow in plain Playwright, Playwright update snapshots without the PNG churn. Captures from agent-written tests count like any others: 10,000 snapshots a month free, 50,000 for open-source projects.
Keep the visual baseline outside the agent's diff
UI Verify captures the pages your Playwright tests open and keeps the baselines off your repo, so an agent can propose a change but the accept happens in review.
Start for freeNo credit card required.
Deterministic Playwright captures
Stop real-page diffs that flake without a real change.

