How Squally measures tests

Squally describes each test with a few numbers, computed from your test results whenever you look: how often it passed on the first try, how often it needed a retry to pass, how often it failed, and how much time its retries and failures cost.

Squally does not judge a test. Every number says what happened in the period you chose, and nothing more - what to make of it is up to you, or to the agent you hand it to.

Turn on retries

Squally can only tell a flaky test from a broken one when Playwright retries failed tests. Without retries a test that fails once has no second attempt that could pass: the run counts as failed, and the flaky rate stays 0 %. Turn retries on in CI, in playwright.config.ts:

import { defineConfig } from "@playwright/test";

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
});

Which runs count

A run is one test run as a whole: when your suite is split into shards (--shard), all shards of one CI run are one run, and every number below counts it once.

A run counts when all of these are true:

• It is of the chosen source - CI and local runs by default. A run is a CI run when the reporter recognised a CI provider (which providers); a run from your own machine is a local run. The source menu in the filter sentence of the Overview, the Test explorer and a test’s page - “CI and local runs”, “CI runs” or “local runs” - narrows the numbers to one of them. Both is the default because most projects start with a local run; the price is that local runs - single tests, deliberate failures while writing a test, usually no retries, another machine’s durations - move the numbers. Pick CI runs to read your CI on its own.
• It finished. A run finishes when all its shards have finished - or, once the run timeout has passed since its last activity, when every shard that started has finished, timed out or was stopped by the monthly result limit. A shard that died, never started or hit the monthly result limit holds a run back only until then; after that the run is incomplete and counts like a cancelled run: the results its shards reported count. A run with a shard still running does not count yet.
• It is not a mass failure. When a run holds at least 20 tests and more than 80 % of them failed - judged over the whole run, all shards together - Squally treats it as an outage - a failed deploy, a test environment that was down - and it counts toward no test. Tests marked test.fail() do not count as failures for this check.
• It started inside the period, by the start of its first shard: the last 7, 14, 30 or 90 days, counted back from the moment you look. The default is 30 days.

Runs on every branch count, unless you choose one branch; then only that branch’s runs do. No branch is treated differently from another.

The read API names this set of runs - see Conventions.

One result per test and run

A run can hold several attempts of the same test. Only the final attempt counts - the one with the highest retry number. If the test was reported from two shards of one run, the lower shard number decides a tie; with Playwright’s repeatEach, the first repetition counts. A test marked test.fail() is read the way Playwright reads it: failing as expected counts as a pass. If the final attempt was skipped, the run does not count for that test at all.

Each run that counts ends in one of three results:

Result	Final attempt
stable	passed on the first try
flaky	passed, but only after at least one retry
failed	failed, whether it was retried or not

A final attempt that ran into its time limit is a failed one - see Timeouts.

The numbers

• Runs - the runs that count for the test: stable + flaky + failed.
• Stability = stable runs ÷ runs. A test that passed on the first try in 16 of its 20 runs has a stability of 80 %.
• Flaky rate = flaky runs ÷ runs.
• Failure rate = failed runs ÷ runs.
• Time lost - how much CI time the test’s retries and failures cost: a failed run lost every attempt it made, a flaky run the attempts before the one that passed, a stable run nothing. Each lost attempt is counted at the duration of the run’s final attempt.

The three rates add up to 100 %. A test with no run in the period has no rates - the app shows “—” and the API returns null, which is not the same as 0 %: 0 % means it ran and never did that. The app shows one decimal and the counts behind each figure in a tooltip (“16/20 runs passed first try”); a rate that is not zero never reads 0 % - one flaky run in 3,000 reads <0.1%. The API returns the exact fraction (5 of 8 is 0.625).

With each test the app and the API also give its last runs (each with its run number, branch and result), the branches it ran on, and the branch where it failed most - or, if it never failed, had the most flaky runs.

Colours

The Test explorer, a test’s page and the Overview’s Suite stability tile colour only a bad value, and only in red:

Number	Red
Stability	below 70 %
Failure rate	20 % and above

Every other value - a flaky rate of any size included - and a rate with no value (“—”) is drawn in the normal text colour. The colour is there to catch the eye; it is not a label the test carries.

Where you find them

• In the app: the Test explorer (every test, sorted by one column, the worst end first - lowest stability by default) and each test’s page, with one row per Playwright project. The Overview’s Failing tests (tests with at least one failed run) and Flaky tests (tests with at least one flaky run) and its Most failing and Most flaky lists count the same runs, but each test once across projects: a run counts as failed for a test when it failed in any project. So the Overview’s counts can be lower than the Test explorer’s rows. On the Overview they follow its branch menu, which starts on all branches. Analytics shows how these shares change over time.
• The Overview’s Suite stability tile counts differently: it counts mass-failure runs, which every per-test number leaves out. See Overview for what each tile and card counts.
• In the read API: List tests and Get test metrics for one test; the project overview carries the Overview’s run figures, and its testMetrics count per Playwright project, like the Test explorer - not once across projects like the Overview’s tiles. Each takes source (all by default, ci or local) - see Conventions.
• For your coding agent: the MCP server’s squally-list-tests and squally-get-test-metrics, which take source too (since squally-mcp 0.4.0), and the line “Last 30 days on CI: …” in every Copy-for-AI prompt.