Visual testing: build vs buy, the honest version

Build versus buy for visual testing gets argued as a matter of taste, and it should not be. It is an arithmetic problem: does standing up a homegrown screenshot setup and keeping it running cost less, over a couple of years, than the tool it would replace? Both answers are defensible. Some teams should build. Most teams who think they should build are pricing only the first afternoon of the project and none of the years after it. This post is the flat version of the decision, both directions, so you can run the numbers instead of the debate.

The reason the two sides look closer than they are is that the entry cost is genuinely small. A Playwright screenshot and a pixel diff against a committed baseline is an afternoon of work, and it does catch things. What that afternoon does not price is the four parts that turn a demo into something a team trusts on every pull request, and the maintenance on those parts is yours forever once you own the code. So the real comparison is not build-cost versus subscription. It is total cost of ownership on both sides.

Frame the decision around four cost buckets, not one. Build cost is the sprint to stand it up. Maintenance cost is every week after, when a Chromium bump reshuffles antialiasing or a new component needs new plumbing. Opportunity cost is the engineer-weeks spent on infrastructure that is not your product. And trust cost is the quiet one: a homegrown check that flakes gets muted, and a muted visual check protects nothing while still costing render time. Buy shifts most of the last three onto a vendor and turns them into a line item. Build keeps all four, and only the first is visible on day one. Whichever way you lean, price all four buckets, because the decision is almost always won or lost in the three you cannot see yet.

There are real cases where build is the right call, and a vendor pitch that pretends otherwise is not worth trusting. Building wins cleanly when:

• Your UI surface is small and stable. A dozen components that change a few times a quarter never generate the volume that makes a hosted service pay for itself. A committed baseline and Playwright's local toHaveScreenshot gate entirely inside your own CI, cost nothing, and you review the handful of diffs by hand. A tool here is overhead you do not need.
• You have hard self-host, compliance, or bring-your-own-model requirements. If procurement will not clear a cloud service, if your pixels cannot leave your network, or if the AI judgment has to run on a model you host, a SaaS that renders centrally and calls a hosted judge is a non-starter no matter how good it is. That is a legitimate build-it case, and it is the one most vendors quietly skip.
• You already have a working setup your team trusts. If you built visual regression from scratch, froze every render, mocked the state, and it has caught real bugs for a year, the migration cost is real and the payoff has to clear it. Keep what works. The build-vs-buy question is for the team that has not paid the build cost yet.

Notice what those three have in common. They are all reasons the tool cannot do the job or the job is small enough that the tool is not worth it. None of them is "it looks easy." The looks-easy case is where the arithmetic usually flips.

The afternoon gets you a screenshot and a diff. Everything a team actually relies on lives in four parts underneath, and each is a standing maintenance cost rather than a one-time build. This is the same anatomy I laid out in visual testing for coding agents, read from the cost side instead of the engineering side:

• Determinism across machines. The same page screenshots differently on two machines: fonts hint differently, an antialiased edge lands on a different sub-pixel, the GPU rasterizes a hair apart, and the build fails for no real reason. Pinning one render environment, freezing animations, and mocking live data is the price of admission before you catch a single regression, and it re-breaks on every Chromium and font bump. This is exactly why hosted tools render centrally rather than on your laptop (see why the cloud render exists).
• A diff an agent can reason over. A pixel delta tells you something moved, not whether it should have. Telling an intended restyle from a regression is a verdict, and a bounding-box pixel count cannot produce one. This is the part homegrown setups never really solve, so a human ends up in the loop on every diff (see the AI judge).
• Capture that scales off your laptop. A real UI has hundreds or thousands of states. Rendering them on your own hardware makes the visual suite fight your dev loop for cores and memory, so eventually someone spends a week wiring capture onto a fleet of remote browsers, and any drift across that fleet reopens the determinism problem.
• An MCP to close the loop. Stable captures and a good verdict still sit in a CI log the agent cannot read. To make visual review part of the agent loop, the agent needs the specific diff image and its verdict in a form it can parse and act on, the way it reads a stack trace. That is a whole second service on top of the first (see triage with your agent).

Any one of these is buildable. Together they are a product, and the buy case is simply that a tool has already amortized all four across every customer, so you rent the maintenance instead of hiring it. Whether that trade is worth it for you is what the number decides.

The most useful thing I heard on this came from an infrastructure lead at a large org who owns visual testing across a lot of frontends. He had already built the homegrown version once, for a side project: Storybook stories, baselines on S3, screenshot compare, the diff dashboard, the lot. Standing it up again for work was, in his words, not the hard part. And yet he could not make the call, for one blunt reason: he did not know what his current vendor bill actually was. Without that figure there was no case to build against, and no case to buy against either. The decision was not blocked on engineering. It was blocked on a missing number.

That is the honest core of build vs buy, and it is why so many of these debates go in circles. Put an illustrative shape on it. Suppose a mid-size suite of 500 stories across a couple of browsers and viewports, on a per-snapshot plan, running on every push from a team whose commit volume has climbed in the agent era. The vendor bill could be a rounding error or it could be real money, and the two land you in completely different decisions, yet plenty of teams argue the architecture for weeks without ever pulling the actual invoice. Get the number first. If the bill is negligible, there is nothing to rebuild and you keep buying. If it is real, then you weigh it against a fully loaded build estimate, all four hard parts and their maintenance included, not just the afternoon.

The clean way to settle it is to price both sides on real builds rather than in the abstract. Point a tool at the tests you already have and let it run alongside whatever you do today, so the buy side is a measured number instead of a guess. UI Verify reads your existing Storybook, Playwright, or Vitest tests, renders in the cloud so determinism is handled, and the free tier is 10,000 snapshots a month, which is enough to see your real volume and diffs before anyone signs anything. Paid plans start at $89 a month for 30,000 snapshots, with overage at $0.004 per snapshot, so the buy line is a flat, public figure you can drop straight into the comparison.

Then estimate the build side just as honestly: the sprint to stand it up, plus the recurring cost of keeping the four hard parts alive as your dependencies move. If the two-year totals favor building, build, and you will do it with your eyes open. If they favor buying, you will have skipped a rebuild you did not need. Either way you decided on arithmetic. The fastest way to get the buy number is to let your agent wire it up: point it at the set up visual testing skill and you will have a running comparison this afternoon, which is about the same time the homegrown demo would have taken, and a lot more decisive.