Agent-written tests need a check the agent can't edit

Here is a commit from our own repository, co-authored by a coding agent. It shortens the preview line of our welcome email. Two files changed: copy.ts, where the string lives, and copy.test.ts, where a test asserts that string. The old expectation went out and the new one came in, in the same commit, from the same author. The suite passed.

Nothing went wrong in that commit. The new line was the one we wanted, and a person read it before it merged. But look at what the test contributed: nothing. Whoever writes the new string also writes the new expected value, so the assertion can only ever agree. The review of the copy was the check. The test was a second copy of the string.

When one author writes both a change and the tests for it, the tests stop being an independent check: whatever the author got wrong in the code, they can get wrong in the test, and a failing test is one more thing the author is free to edit. Humans have always been able to do this. Coding agents do it routinely, because an agent asked to make the suite pass treats a red test as part of the task. Three shapes show up again and again.

• The expectation follows the code. An assertion is rewritten to match the new output, as in the commit above. Correct when the change is right, silent when it is not.
• The snapshot is regenerated. A markup or JSON snapshot fails, the agent runs the update flag, and the new snapshot lands in the same diff as the code that produced it.
• The test disappears. A refactor removes a file and its test file together, or a test that kept failing is deleted as obsolete. Coverage drops and nothing turns red.

A startup CTO whose team leans heavily on coding agents described the last one to me plainly: the test gets updated at the same moment as the code it is supposed to test, and now and then the test is simply gone when it should have stayed. A coverage report counts that test right up until the commit that removes it, and then stops counting without complaint.

To see the failure end to end, I staged it in a scratch repository. A React PlanCard renders a plan name, a price, two features, and an Upgrade button. It has three Vitest tests: two that check text is present, and one toMatchSnapshot() over the rendered markup. Then I made the kind of commit an agent makes when asked to move a component onto design-system variants: the button's class went from btn-primary to btn-accent. The stylesheet never defined btn-accent. The snapshot test failed, vitest run -u rewrote the snapshot, and the commit went in with both files.

Read the pull request the way a reviewer would. Two one-line changes, the same rename in both, a green run. Nothing in that diff says the button now renders with the browser's default grey background behind white text, which is what happened. The snapshot test serializes markup, so it faithfully recorded the new class name and has no idea what that class looks like. The two text checks still found Upgrade to Pro in the HTML, because the label is still there. It just cannot be read.

Halfway. A screenshot assertion looks at pixels, so it catches what the markup snapshot missed. In the same scratch repo, Playwright's toHaveScreenshot against a baseline PNG committed with the component failed on the agent's commit: 58,005 pixels, 14% of the card, different. That is the right answer. The trouble is where the baseline lives. The PNG sits in the repository next to the test, and regenerating it is one flag away.

With --update-snapshots the run passes and the baseline PNG joins the diff, from the same author, in the same pull request, approved with the same click as the code. A committed screenshot is a much better test than a markup snapshot, but it has the same weakness: the thing it compares against is a file the change is allowed to edit.

A check is independent when the change under test cannot alter the reference it is compared against. For tests of UI, that comes down to three properties, and the scratch repo failed all three:

• The reference lives outside the diff. If the expected output is a file in the pull request, the pull request can rewrite it.
• Moving the reference is a separate, visible decision. Updating a baseline should be an action someone takes after looking at the change, recorded where reviewers see it, not a side effect of a command that makes the suite green.
• A missing check is reported, not just absent. A deleted test fails nothing. Something has to notice that coverage existed on the base branch and is gone on the head.

In the service, not in your repository. UI Verify keeps one baseline per branch, resolved from your git history, and there are no screenshot files in the repo for a commit to overwrite. A baseline changes only when a change is accepted: by a person in the dashboard, or by an agent you gave that access to over MCP. Accepting is limited to members of the project. The AI judge labels each change as a likely regression or a likely intended change with a reason, but it never moves a baseline by itself.

On a pull request this shows up as a status check. When a build has changes to review, the check is pending; when they are accepted, it flips to success on the same commit, with no new push and no CI re-run. Here is that sequence on one of our own pull requests, read straight from GitHub's status history:

The deletion case is covered by the This PR vs base changeset, which compares the whole pull request against its base branch and lists removed stories next to new and changed ones. A story that existed on main and is absent from your head shows up as removed, which is the accidental-deletion catch a per-commit diff cannot make. Make the check required in branch protection and the pull request cannot merge while changes wait for review.

One caveat, because independence is a property of how you run it, not of the tool. If the agent that wrote the code is also the one accepting its own visual changes with nobody looking, you have rebuilt the loop from the scratch repo with more steps. Let the agent triage over MCP and write up what it found, and keep the accept with a reviewer for changes the judge marks as likely regressions.

Yes. They catch logic errors, they document intent, and an agent writes them faster than anyone. They just need a discipline the agent cannot skip. Mine, for every agent working in our repo: a test for a behavior change is not done until you have watched it fail. Revert the change, or mutate the one line that implements it, run the test, see red, then restore. That rule exists because a regression test once passed with the entire change it was written for reverted, since it asserted on data it had seeded itself. It read as coverage and it covered nothing.

Watching a test fail makes the author's own tests better. It does not make them independent, because the same author is still the one running the experiment. That is why I run both: unit tests the agent has to prove can fail, and a visual baseline the agent cannot rewrite in the pull request.

1. Capture from what you already have: Storybook, Vitest browser mode, or Playwright. The agent's existing tests become the capture path; the baselines go to the service.
2. Make the UI Verify: <project-slug> check a required status check, so an unreviewed visual change blocks the merge.
3. Connect the agent to the build over MCP so it can triage and explain each change, and decide deliberately whether it may also accept.
4. Before approving, read the PR changeset for removed stories, not just the changed list.
5. Tell your agent in its instructions file not to regenerate snapshots or screenshot baselines to make a suite pass without saying so in the pull request.

If you are heading toward merging agent pull requests without a human reading every line, this is the piece that makes the UI part of that safe; auto-merging agent PRs covers the rest of that lane. The goal is simple to state: something in the pipeline has to remember what the screen looked like before the author of the change decided what it should look like now.