Can AI tell a real regression from a flaky diff?

442,813 pixels changed in one of our dashboard's Storybook stories, 30% of the frame, and no element on the page looked any different. The page had simply rendered taller than its baseline, so the diff painted the whole extra band. A few days earlier another story changed in a 13-pixel strip: a URL whose id was different on every render. Both came back as changes. Neither was one.

So can AI tell a real regression from a flaky diff? It can help, but asking a model the flake question directly is the wrong design. Whether a diff is a flake is a fact you can prove by rendering the same commit again. Whether a real change was meant is a judgment, and that is the question a model is good at. Keeping those two apart is most of what makes an AI-reviewed visual check trustworthy.

A flaky visual diff is a difference between two screenshots that does not come from a code change: render the same commit twice and the pixels disagree. The usual causes are anything the render reads besides your code. A live clock, an animation caught on a different frame, a web font that arrives a beat late, live data, a random seed, an id minted at render time, or a layout that settles at a different height depending on timing.

The second one is a classic that any app can hit. Our API key settings story shows a Live Storybook URL, and the fixture behind it minted a fresh time-ordered id every time the story rendered. Each render produced a slightly different URL, so the diff fired on a sliver of text that a human would never notice. The fix was to freeze the story to a static id.

Because the answer is not in the two pictures. A model shown the baseline and the new capture of that settings card sees a URL that changed. A pull request that renamed the Storybook host would produce the same pair. A page that grew 346 pixels could be a new section that failed to load or a layout that happened to settle later. From one pair of images the flaky case and the real case can be identical, so any verdict the model gives is a guess, however fluent the reasoning around it.

Two properties of vision models make that guess risky. Their stated confidence is not a reliable filter: on our own labeled set of real changes, nearly every verdict came back marked high confidence, including most of the wrong ones. And they are not perfectly repeatable: the same diff judged three times can come back with different calls. Neither is a problem for a judgment a human double-checks. Both are a problem for a label that hides a change from review, because a flake label that is wrong hides a regression instead of surfacing one.

The question has a cheaper, exact answer. Render the story again from the same commit. If two renders of identical code disagree with each other, the difference cannot have come from the code. No inference, no confidence score, no tokens.

A changed screenshot raises three questions, and only one of them needs a model. UI Verify answers the other two by checking, not judging:

• Does the change reproduce? UI Verify renders a changed story again against the same commit. If the renders disagree, the story is marked flaky and auto-ignored: it does not block the build, it sits in a collapsed section out of the review queue, and it never becomes a baseline on its own. You can still deny it to force a failure. It is decided per build, with no sticky quarantine list. Details in automatic flake detection.
• Have these exact pixels appeared before? If a story you did not touch comes back changed and the exact rendering already appeared earlier in your branch's history, the change is inherited, and UI Verify points at the branch and pull request it came from instead of asking you to review someone else's work.
• Is the new look what the pull request meant? This is the judgment, and it goes to the AI judge.

Reading intent. A pixel diff knows that something changed. It cannot know that the pull request said "give the star rating on product cards a soft coral badge", and that the badge in the new capture is that sentence made visible. The judge gets the baseline, the new capture, a close-up of the region that changed, and the pull request's own description, and it answers one narrow question: does the new look match what the change set out to do?

That narrow question is where a model beats both a threshold and a tired reviewer. A threshold cannot read a pull request. A reviewer can, but on a pull request that restyles a shared component, opening fifty screenshots to confirm each one matches the description is how real regressions get waved through at the end of the day. The judge does that first pass on every changed story and puts the ones that do not match the stated intent at the top, with a sentence saying why.

It still advises rather than decides. A verdict never changes a baseline by itself; accepting is an explicit action by a person or by an agent you authorized. That is the right place for a model whose confidence you cannot fully trust: ranking and explaining, with a human or a reviewing agent one click away from overruling it.

No. Flake detection is a safety net and the judge is a reviewer; neither is a reason to leave the clock running. A story that flakes on every build is telling you about variation in the render, and the cheapest fix is at the source: freeze time, pin ids and seeds, mock live data, wait for fonts. Our URL story stopped flaking the day its fixture got a static id, and it no longer costs a re-render to prove it. The determinism checklist covers the usual sources. The order that works is determinism first, re-rendering for what slips through, and the model for the one question only it can answer.

If your coding agent does the triage, the same split carries over: the agent reads verdicts and diffs over MCP, flakes are already out of its queue, and it spends its attention on whether each real change matches the pull request it just wrote. Triage with your agent shows that loop end to end.