Playwright flaky tests

What flaky means in Squally

A test is flaky in a run when it passed, but only after a retry. Across runs, its flaky rate is the share of its runs - CI and local by default, only CI runs under CI runs - that ended that way. The exact rules - which runs count, which attempt decides - are on How Squally measures tests.

Turn on retries first

Without retries a failed test gets no second attempt that could pass. Every failure then counts as failed, and the flaky rate stays 0 %. Turn retries on in CI, in playwright.config.ts:

import { defineConfig } from "@playwright/test";

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: {
    // So the failed attempt has evidence to look at.
    screenshot: "only-on-failure",
    trace: "retain-on-failure",
  },
});

Find your flaky tests

• Test explorer, sorted by Flaky (click the column header): every test with its runs, failed runs, flaky runs, stability and time lost over the period. Hover a Flaky count for its flaky rate.
• A test’s page: its rates, its runs and their results, and the branches it ran on.
• Overview: the Flaky tests tile counts the tests with at least one flaky run - each test once across Playwright projects - and opens the Test explorer sorted by flaky rate. Most flaky lists the tests with the highest share of flaky runs, among tests with at least three runs.
• Analytics: Flaky more often lists the tests whose share of flaky runs rose against the period before, and Better than before - marked was flaky - the ones whose share fell. Each test once across Playwright projects; new tests are counted in a line under Flaky more often instead.

Common causes

• Fixed waits. waitForTimeout guesses how long the app needs. Wait for the state you need instead - web-first assertions such as await expect(locator).toBeVisible() retry on their own.
• A missing await. An action or assertion that is not awaited races the next step.
• Weak selectors. CSS chains and nth() break when the page changes shape. Prefer getByRole, getByLabel or a test id.
• Shared state. Tests that depend on data, accounts or storage another test changed pass or fail with the order they run in.
• Parallel tests colliding. Workers writing the same record or the same file get in each other’s way. Give each test its own data.
• Slow third-party services. Payment, email or map providers answer at their own pace. Mock them, or wait for their response explicitly.

From a flaky test to a fix

1. Open the test from the Test explorer, then one of its runs where it was flaky.
2. The test’s page in that run opens on the last failed attempt - for example Attempt 1, when Retry 1 is the one that passed.
3. Read the error, then the screenshot, the trace (Open in Trace Viewer) and the browser console. The browser console is read from the trace, so it needs tracing on.
4. Hand the failure to your coding agent: Fix, then Fix in your agent. With MCP the agent fetches it through the MCP server; without it, the prompt carries the full context.
5. After the fix, watch the test over its next runs: its stability goes up and its flaky rate down.