Flaky tests: what causes them and how to fix them

Six out of a hundred. That is how often a four-line Playwright test failed on unchanged code: fill a search box, wait 300 milliseconds, count the results. The search usually answered in about 90 milliseconds and never took longer than 232 in sixty measured requests, so the wait looked generous. On a busy machine it was not, six times. Nothing about the test or the app changed between the passing runs and the failing ones.

A flaky test is a test that both passes and fails on the same code. Google's definition is the one most people use: "a test that exhibits both a passing and a failing result with the same code" (Google Testing Blog, 2016). The failure is real, in the sense that something in the run really went differently, but it is not caused by the change under review. To see what actually produces flakiness, we wrote one small flaky test per cause in a scratch project and ran each one 100 times on Playwright 1.64.1 and Vitest 5.0.5, on 2026-11-02, on a laptop under heavy load from other jobs, which is roughly what a shared CI runner looks like.

Common enough that every large test suite has them. In 2016 Google reported that about 1.5% of all its test runs produced a flaky result, that almost 16% of its tests showed some flakiness, and that about 84% of transitions from passing to failing involved a flaky test. A year later it broke the rate down by test size: 0.5% of small tests, 1.6% of medium tests, and 14% of large ones were flaky. The bigger the test, the more moving parts can go differently, which is why end-to-end suites suffer most.

The best evidence on causes is still Luo and colleagues' 2014 study, "An Empirical Analysis of Flaky Tests", which classified 161 fixes of flaky tests across 51 Apache projects. Three causes covered over three quarters of them: async waits (45%), concurrency (20%), and test order dependency (12%). Two other findings matter for how you work: 78% of flaky tests were flaky from the day they were written, and of the 20 fixes that added or lengthened a sleep, none removed the flakiness. All 20 only made it rarer.

A timing race happens when a test checks the UI before the UI has finished updating. Our search test is the classic form: a fixed waitForTimeout(300) followed by a one-time count. It failed 6 times in 100 with Expected: 3, Received: 0. The fix is an assertion that retries until the condition holds: await expect(page.locator('.result')).toHaveCount(3) waits up to its timeout and checks repeatedly. That version failed 0 times in 100, with one worker and with four. A longer sleep would only have made the failure rarer, which is what Luo found in every sleep-based fix.

Two tests that touch the same data can each pass alone and fail together. Ours shared one in-memory cart. One test checked that a new visitor sees an empty cart; the other added an item and cleaned up after itself. Run one at a time, both passed 100 of 100. Run with four parallel workers, the empty-cart test failed 5 times, each time finding "1 item(s) in your cart" because it ran during the other test's window. The cleanup worked; the overlap was the bug. The fix is isolation, a fresh cart or user per test, rather than going back to one worker.

An order-dependent test passes or fails depending on which test ran before it. Our Vitest file had three price-formatting tests, and one of them set a module-level currency to EUR without resetting it. In file order the suite passed every time. With --sequence.shuffle and 100 different seeds, it failed 57 times: in every order where the euro test ran before the dollar test, the dollar test saw €12.00.

This kind of flakiness is not random at all. It is fixed for any given order, which is why it can hide for months and then appear when someone adds a file, enables parallelism, or upgrades the runner. Shuffling on purpose in CI, as Vitest's --sequence.shuffle or Jest's --randomize does, turns it into a failure you can reproduce with a seed. The fix is resetting shared state in a beforeEach.

Some tests are not flaky on one machine, only between machines. Ours formatted an invoice due at 9:30 pm New York time and expected "Nov 4". Under five time zones it passed in New York and Los Angeles and failed in UTC, London, and Tokyo, every time, because in those zones the moment is already November 5. A developer in New York sees it pass, a CI runner set to UTC sees it fail, and the team calls it flaky.

The same shape comes from locale, installed fonts, screen size, CPU speed, and browser version. The fix is to pin what the test depends on: pass an explicit timeZone, set TZ in the test script, and run browser tests in a container or a fixed environment so local and CI agree.

A test that calls a real external service inherits that service's bad days. Ours asked the public GitHub API for a repository's star count. It passed 55 times in a row and then failed 45 times in a row with HTTP 403, because GitHub allows 60 unauthenticated requests an hour per IP address and our earlier calls had used the rest. In CI the effect looks random, because other jobs behind the same outbound IP spend the same quota.

Mock the boundary in tests that are about your code, and keep a small, separate set of tests that check the real integration on a schedule, with credentials, where a failure tells you about the service rather than about the change.

Because a screenshot records everything, including things that are still moving. We built a small profile card whose avatar fades in and whose loyalty meter animates to 70%, two of the most common motions in real UIs, and captured it with a plain locator.screenshot() compared against a baseline. All 100 runs failed, and they produced 11 different images: the avatar at different opacities, the meter at different widths, 3,149 to 3,428 changed pixels each time. That test was not sometimes red. It was always red, with a different picture each run.

Playwright's toHaveScreenshot() on the same page passed 100 of 100, because it disables CSS animations by default and retakes the screenshot until two consecutive frames match. The assertion you pick matters as much as the page. The other usual suspects in screenshot tests are the clock, live data, randomness, and web fonts that arrive late; our guide to fixing flaky Playwright screenshot tests goes through each one.

No, retries hide them. We reran the timing race with retries: 2. All six failures passed on a later attempt, and Playwright filed them under "6 flaky" next to "194 passed". The run ended green, because Playwright's failOnFlakyTests option defaults to false. The bug was exactly as present as before; the only thing that changed was that nobody would see it.

Retries and quarantine are useful as a bridge, to keep a team shipping while someone fixes the root cause. They become harmful when they are the plan. An engineer at a large collaboration-software company described the end state: about one in ten of their end-to-end tests flaky at any given time, fixed in batches when someone has the time, and a red check that people learned to rerun before reading. Once a team reruns by reflex, a real regression gets rerun too.

1. Make them visible. Turn on failOnFlakyTests in CI (or your runner's equivalent), or at least report the flaky count, so a retry that rescued a test is still a signal.
2. Reproduce on purpose. Run the suspect test with --repeat-each=100, shuffle order with a seed, vary TZ, and run with several workers. A flaky test you can reproduce is a bug you can fix.
3. Fix the cause, not the timing. Replace sleeps with retrying assertions, isolate shared data per test, reset module state in beforeEach, mock external services, and pin the environment.
4. Fix it the day it appears. Since most flaky tests are flaky from day one, run new tests many times before merging them.

For visual checks, UI Verify removes most of the causes above at the source and catches the rest. Every capture renders in one pinned browser and environment, so font and machine differences never reach the diff, and every capture is taken with reduced motion emulated, so animations written to respect it stay still. When a story still comes back changed, UI Verify renders it again a few more times on the same commit. If those renders disagree with each other, the change cannot come from the code, so it is marked flaky, kept out of the review queue, and never becomes a baseline, while staying visible in a collapsed section rather than turning green silently.

The difference from a retry is the question being asked. A retry asks whether the test can pass once. Flake detection asks whether independent renders agree. The deterministic captures guide covers what to pin on your side, like the clock and live data, and flake detection covers the rest. UI Verify has a free plan with 10,000 snapshots a month, raised to 50,000 for open-source projects.