Overview

The Overview opens on CI and local runs, all branches and the last 30 days. The line under the title says so as a sentence:

Showing CI and local runs on all branches from the last 30 days

Each of the three choices is a small button that opens a menu:

• Source - “CI and local runs” (the default), “CI runs” or “local runs”. See local runs.
• Branch - “all branches” (the default), or one or more branches that ran in the period. Squally remembers the branches you pick for each project in this browser: the next time you open the Overview without a branch in the address, it opens on them again. Pick “all branches” to go back to the default.
• Period - the last 7, 14, 30 or 90 days.

A choice that is not the default is drawn in dark ink, so you see at a glance that the page is not on its usual scope. When runs are still going, the sentence ends with “· 2 unfinished”.

Above the line, a sentence sums up the period in its own numbers - for example “42 of 75 runs passed in the last 30 days. 2 tests are failing in a row; most failures share one error.” It counts the tests that are failing in a row - the same tests the section of that name lists - and says “most failures share one error” when one error makes up more than half of the period’s failures. When every finished run passed it says “All 40 runs passed in the last 30 days.”, when none did “No run passed in the last 30 days.”; when the period has only cancelled or incomplete runs, it says so - “Only cancelled runs in the last 7 days.” - and Passed runs reads “—” while the other tiles still count those runs.

The ⓘ at the end of the line says which runs are counted: “Counted CI runs only”, “Counted local runs only” or “All counted runs”. A run counts once all its shards have finished, a sharded run as one run - or, once the run timeout has passed, as an incomplete run with the results its shards reported. Runs that are still going are in no figure.

Why all runs are the default

Most projects start on a laptop: the first run you send is usually a local one, and CI comes later. So the Overview counts CI and local runs together until you narrow it, and a project with only local runs shows its numbers right away.

The price is that local runs mix into the figures. Local runs are a different kind of run: you start a single test instead of the suite, you let a test fail on purpose while you write it, retries are usually off, and your machine is not the CI machine, so durations do not compare. To read your CI on its own, pick “CI runs” - the choice stays in the address, so a bookmark keeps it.

What Squally sends out counts CI runs only: the weekly email report, the GitHub check and pull request comment and Slack. Nobody picks a source there, and a run from someone’s laptop should not change what the team is told about CI.

Tiles

Five tiles sit in a row under the sentence.

Tile	What it shows
Passed runs	finished runs without a failed test - passed or flaky - out of all finished runs, for example “9 of 57”. Cancelled and incomplete runs are in neither number
Suite stability	of every executed test in the period’s runs, the share that passed on the first try. A flaky pass is not a first-try pass; skipped tests are left out
Failing tests	tests with at least one failed run, out of the tests that ran. A test that runs in several Playwright projects counts once, and a run counts as failed for it when it failed in any of them
Flaky tests	tests with at least one run that passed only after a retry and failed in no project, out of the tests that ran - each test once, across projects
Time lost	the time spent on attempts a later one replaced, with its share of the runs’ run time

The last line of each tile compares it with the period before, of the same length and on the same source and branch:

• “↓ from 12 of 48” or “↑ from 9min 10s” - the value went down or up; the text is the previous value as the tile showed it.
• “same as before” - both read the same as shown.
• “nothing before” - the period before had nothing to compare with, or it was mostly empty: when the project’s first run lies in the second half of the period before, a comparison would rest on a few days, so there is none.

The arrow only says the direction. It is never green or red: more passed runs can just mean more runs.

Suite stability counts the runs the line above the tiles names - its period and source. A run’s report counts one run and the email report one week of CI runs, so their stability can differ from the tile’s.

Suite stability is red below 70 %; no other tile is coloured. Hover a tile’s title for its definition. Clicking a tile opens what is behind it: Passed runs and Suite stability the runs list, Failing tests the Test explorer sorted by failure rate, Flaky tests sorted by flaky rate, Time lost sorted by time lost. The Test explorer opens on the Overview’s source, branch and period.

Suite stability and Passed runs count runs where most of the suite failed at once, and the tests that finished in a cancelled or incomplete run. Failing tests, Flaky tests, Most failing and Most flaky leave out runs where most of the suite failed at once, as every per-test number does. The definitions are on How Squally measures tests.

The Overview counts a test once across Playwright projects: “adds an item” in chromium and in firefox is one test, and a run is one run for it. The Test explorer lists each project’s variant as its own row, and the read API’s project overview counts per project too, so their counts can be higher than the tiles’.

Sections

Below the tiles come the sections, in this order. Each has a heading, a short hint beside it, and its rows in a card; every list has at most five rows. Failing in a row comes first, and only when a test is failing in a row - otherwise the section is left out.

Every row of a list is two lines: the name - a test, an error - and under it where it lives. Each line stays on one line; a name too long for it ends in “…”, and hovering it shows the whole name. So every row of a card has the same height.

Section	What it shows
Failing in a row	the tests that failed in each of their latest runs, at least three, the longest streak first - see below. All tests opens the Test explorer sorted by failure rate
Most failing	the tests with the highest share of runs that failed, among tests with at least three runs - each test once across Playwright projects, with its file and projects under the name, the share (“45.2%”) and the runs behind it (“33 of 73”). All tests opens the Test explorer sorted by failure rate
Most flaky	the same for runs that passed only after a retry. All tests opens the Test explorer sorted by flaky rate
Top errors	the period’s errors with the most failed attempts. A row’s first line is the error’s title, as the Error explorer and the error’s page show it - the test’s own assertion message when it wrote one, otherwise “Timeout · test limit 30s”, or “Not visible” with its locator beside it in code font. The second line says where it lives: “tests/cart.spec.ts · 3 tests”, or “4 spec files · 9 tests” when it appears in more than one, with “· 2 projects” when it hit more than one Playwright project; when the title is the test’s own message, the kind leads it - “Not visible · tests/cart.spec.ts · 1 test”. On the right, how often it happened (“66×”). A row opens the error, All errors the Error explorer
Where the time goes	the tests that lost the most time to repeated attempts - every attempt a later one replaced, retries and failed runs alike - each test once across projects, with a grey bar and the time. A test needs to have lost 1s or more
Branches	the most recently run branches, two lines each: the branch’s name, and under it its runs and their median duration - “38 runs · median 4min 5s”. On the right the share of its finished runs without a failed test - “59.5% passed” - and under it its newest run in the period: its status icon, its number, linked to the run, and how long ago it started (“#1 · 9m ago”). The row opens the runs list on that branch; All runs opens the runs list
Runs	runs per day by result: one bar per day of the period, its runs stacked by how they ended - Failed at the bottom, then Flaky, Passed and Cancelled or incomplete. The legend gives each result’s total for the period. A day without runs stays empty, a short grey mark on the baseline. Hover a day for its date and its runs per result, each with the run’s status icon; the flaky row reads “Flaky (passed on retry)” - the runs list shows such a run as passed, with its flaky tests counted

A row in a test section opens the test’s page on the same source, branch and period.

When a list has nothing to show, it says why:

• Most failing / Most flaky - “No test failed in this period.” (or “No test needed a retry in this period.”) when none did. When tests did, but none ran three times, the list cannot rank them and says “Needs at least 3 runs per test.” - with “Try a longer range.” and a link to the next longer period when the project has runs before this one.
• Where the time goes - “No test lost 1s or more in this period.” The Time lost tile can still show a few hundred milliseconds.
• Top errors - “No error in this period.”; Branches - “No runs in this period.”; Runs - “No runs in this period.”
• Failing in a row has no empty state: with no test failing in a row, the section is not there.

Cancelled and incomplete runs are left out of a branch’s pass rate and median duration: they are runs, but what they came to is not the suite’s result. A branch whose runs were all interrupted reads “— passed”. The Runs chart counts them as what they are - Cancelled or incomplete. The chart’s first day also holds the period’s hours before it, and its tooltip says from when (“Oct 7, 14:00 – Oct 8”). Trends over time - pass rate, run duration, what changed against the period before, failures by spec file and error category - are on Analytics.

Failing in a row

A test is failing in a row when it failed in each of its latest runs, at least three of them, and still fails in the newest - it has not passed since. The section lists such tests, the longest streak first; the sentence above the tiles counts them.

How a streak is counted, run by run from the newest back:

• One test across Playwright projects, as in Most failing: a run counts as failed for the test when it failed in any of its projects.
• A failed run extends the streak. A passed run ends it, and so does a flaky one - a test that passed on retry is not failing.
• A run where the test did not run - or was only skipped - and a cancelled or incomplete run neither extend nor end it: they are stepped over.
• The streak may reach back before the period, as far as runs are stored; its newest run must lie in the period. Runs where most of the suite failed at once are left out, as in every per-test figure.

Each row shows the test’s name and under it its file, the projects it failed in and the title of its newest failure - “tests/account.spec.ts · firefox · Timeout · test limit 30s”. On the right, how long the streak is - “27 runs in a row” - and under it where it began - “since #134 · Nov 1”, linked to that run. The name opens the test’s page.

What follows which switch

Every tile and section follows the source, the branch and the period - the Runs chart too. One exception, which the ⓘ names (“Branches lists every branch of the period.”):

• Branches lists every branch of the period, whatever the branch filter.

Failing in a row follows the source and the branch like every section; only its streaks may reach back before the period (see Failing in a row).

No runs yet

A project that has never reported a run shows Send your first run in place of the tiles and sections: four steps for one run from your machine - Install the reporter (Node 20.18.1+, Playwright 1.30+), Set the key in your shell (macOS / Linux or PowerShell), Add it to the reporters in playwright.config.ts and Run your tests - each with a copy button. Under the last one: look for the [squally-reporter] lines in your terminal - when nothing arrives, one of them says why (Troubleshooting).

A fifth step, Run it in CI, is optional and folded until you click Show. It has the same four tabs as the setup page: GitHub Actions, GitLab CI, Azure Pipelines and Other CI.

Beside the steps, a card lists the four Getting started items - Getting started · 2 of 4 for a new project - with Waiting for your first run… under them, a link to regenerate the ingest key in the project’s settings, and links to the setup guide, troubleshooting and the reporter on npm.

The page opens the dashboard as soon as the first run arrives - a local one too. The local steps then live on the setup page.

Getting started

A rocket beside the logo in the sidebar - in the top bar on a phone - counts what is left to set up in this project. Click it for Getting started: how many of the four items are done, a progress bar, and the items, the first open one highlighted.

Item	Done when
Create your organization	always - you are in one
Create your project	always - you are in one
Send a CI run	the project has a run the reporter recognised as CI. A local run never ticks it, whether it came first or not: checks, trends and the weekly report count CI runs. Show CI setup opens the setup page.
Connect an integration	a GitHub repository is linked to the project (GitHub) or Slack is connected (Slack). Admins get Open integrations; members read “Ask an admin to connect one.” Marked GitHub only - optional: once it is the only item left, “Not using GitHub? Skip this step” closes the guide, like Skip setup guide.

The rocket’s number is the count of open items. Under the items, Setup docs opens the reporter setup and Skip setup guide hides the rocket - for you, in this project: your teammates still see it, and you still see it in your other projects. The skip is stored with your account, so it holds in every browser; the app has no button to bring it back.

When all four items are done, the rocket goes for good - removing an integration later does not bring it back. It never shows on the organization’s own pages.

The setup page

Run it in CI is the project’s CI setup, a page of its own at /o/<organization>/p/<project>/setup. The rocket’s Show CI setup, the card’s Show CI setup and the Setup guide link under Settings › Project › Ingest key open it.

It has four tabs - GitHub Actions, GitLab CI, Azure Pipelines and Other CI - each with what goes where for that CI, a link to its guide in these docs, and where checks and pull-request comments stand for it (GitHub repositories only). GitHub Actions and Azure Pipelines show the whole workflow or pipeline file to copy, with Squally’s lines highlighted. Every tab ends with Turn on retries and the line on what to do when nothing arrives.

Add ?ci=gitlab, ?ci=azure or ?ci=other to the address to open that tab - a link you can send to whoever owns the pipeline. Without it, or with any other value, the page opens on GitHub Actions.

Once a project has a local run, the page says when it arrived - First local run 2h ago · Show local setup - and Show local setup opens the four local steps there (?steps=local).