Glossary

Attempt - one execution of a test within a run. The first is “Attempt 1”; Playwright’s retries follow as “Retry #1”, “Retry #2”. See Debugging basics.

Cancelled - a run stopped from outside, by Ctrl+C or a cancelled CI job, whose reporter could still send the results that had finished and report the stop. See Cancelled runs.

CI run / local run - a run is a CI run when the reporter recognises a CI provider, or CI is true or 1; anything else is local. Squally’s pages and the read API count both by default; CI (source=ci) or Local narrows them. Only CI runs post to GitHub and Slack and are in the weekly report. See Local runs.

Error (group) - failures with the same error message, once values such as durations and IDs are removed, in the same test file. Squally groups them, so one broken selector is one error, not many failed tests.

Expected failure - a test marked with Playwright’s test.fail(). Failing as expected counts as a pass. See Expected failures.

Failed (run) - for one test in one run: its final attempt failed. See How Squally measures tests.

Failure rate - the share of a test’s runs that failed.

Flaky (run) - for one test in one run: it passed, but only after a retry.

Flaky rate - the share of a test’s runs that were flaky. See Playwright flaky tests.

Incomplete - a run that will not finish: it never reported its end and Squally’s hourly check found it running too long or a shard missing, or the monthly result limit was reached. Until the run timeout has passed it is waiting - its counts can still change and its page says “Partial”; after that it is settled and counts like a cancelled run: with the results the run’s shards reported, but not in run rates. See Incomplete: waiting or settled and Timeouts.

Ingest key - a project’s key, starting with sqly_, that the reporter uses to send results. See Keys.

Job key - SQUALLY_CI_JOB_KEY (squally-reporter 0.11.0): a name for one job of a CI run that has several jobs which are not shards of each other - a browser matrix, an api and a ui suite. The shards of one job share it. Each job with a key is a run of its own, with the key as a chip beside its number, its own GitHub check, “Squally · firefox”, and its own block in the pull request’s comment; a re-run of the job links to that job’s earlier attempt. See Matrix jobs and GitHub.

Mass failure - a run of at least 20 tests in which more than 80 % failed, usually an outage rather than broken tests. It stays in the runs list and counts in the error figures, marked, but is left out of the test metrics.

Read key - an organization’s key, starting with sqly_ro_, for the read API and the MCP server. See Read keys.

Retry - an attempt Playwright makes after a failed one, when retries is set.

Run - one Playwright test run, reported by squally-reporter. It appears when the first test starts.

Run status - how a run ended: passed, failed, cancelled or incomplete, and running while it is still going - the first that applies, in the order cancelled, incomplete, running, failed, passed. A run fails when a test’s final attempt failed. A run whose tests all passed in the end is passed, even when some needed a retry; it shows how many were flaky. A cancelled or incomplete run counts with the tests that finished, but not in run rates such as the Overview’s Passed runs. The runs list, the run page, the comparison and the read API use the same five words. On screen each has its mark: a green check (passed), a red cross (failed), a spinning circle (running), a half-filled yellow circle (incomplete) and a small grey stop square (cancelled). See Run list.

Re-run - on GitHub Actions, a new attempt of a workflow run with Re-run failed jobs or Re-run all jobs. Squally keeps it as a run of its own. When the attempt it re-ran reported to Squally too, the re-run is marked Re-run of #N with a link to it, and that attempt’s page links forward with Re-run: #M. A re-run of failed jobs waits only for the shards it re-ran, and shows the others as “passed in #N · not re-run”. Its GitHub check, pull request comment and Slack message count only the shards that ran again, and say so (“Run #28 · re-run of #27 · …”). Not to be confused with a retry, which repeats one test inside a run. See GitHub Actions.

Shard - one part of a test suite split with Playwright’s --shard.

Sharded run - all shards of one CI run, joined into one run in Squally. See Sharded runs.

Source - which runs a page counts, by where they ran: All (the default), CI or Local. See Local runs.

Stability - the share of a test’s runs that passed on the first try.

Stable (run) - for one test in one run: it passed on the first try.

Time lost - the CI time a test’s retries and failures cost.

Timeout - two different limits. The test timeout is Playwright’s, per attempt; a test that hits it fails and is marked “Timeout · 30s”. The run timeout is Squally’s, for a whole run that never reports its end. See Timeouts.