Run comparison

Open it with a run’s Compare tab, beside Summary, Tests, Errors and Metadata; the run’s header stays as on its other tabs - title, meta line and tab bar with its counts. When the project has no other completed run to compare with, it says so.

To compare two runs you pick yourself, use Compare runs on the runs list: tick two runs - a third box stays locked until you untick one - and Compare opens the newer run’s Compare tab against the older one.

The page opens with one sentence: the two runs and what changed between them, for example “Run #128 against #127, the previous run on main. …”.

Which run it is compared with

A re-run on GitHub Actions is compared with the attempt it re-ran - the question a re-run is opened for is what changed between the two attempts. That attempt is picked first, whatever its size, as long as it is not still running.

For any other run, Squally picks a partner from the 50 newest other runs of the last 90 days whose shards have all finished, and of the same kind as this one: a CI run is compared with a CI run, a local run with a local one. A re-run is never picked here: it holds only the shards that ran again, not the whole suite. Among the rest, Squally prefers, in this order:

1. the baseline: a run on the same branch that started before this one - the previous run on that branch;
2. a run that was not cancelled;
3. a run with every shard reported;
4. a run with at least a quarter as many tests as this one;
5. the newest.

A notice says when the pick has far fewer tests or is missing shards.

Choose run beside “Compared with” opens the picker. A sentence at its top says which runs it lists and lets you change each part, for example “Showing CI runs on main with any result · last 90 days”:

• the kind of run: CI, local or all runs. It opens on this run’s own kind;
• the branch: it opens on this run’s branch; all branches lists every branch;
• the result: any result, or only passed, failed, cancelled or incomplete runs.

Two shortcuts sit above the list: the preselected run - named Previous run on main when it is that, The attempt it re-ran for a re-run - and Last passing run on main, the newest run on this run’s branch that passed. Each row shows the run’s status and number, its result bar with the number of tests, and its branch, commit and shard count, with the runs list’s chips Local and Re-run of #N. The selected run is marked with a check. A search finds a run by number or branch. You can pick a re-run there yourself. A run still running is not offered. A link to a run of another project, or to one that does not exist, is not compared - the page says the run is not available and compares with the preselected run instead.

The two cards

The older run is on the left, the newer on the right, whichever run you opened the page from. The run you came from is titled “This run”, the other “Compared with”.

Each card shows its run’s status, number and chips - Local, the job key (which job of its CI run it was, for example firefox) and Re-run of #N, a link to the attempt it re-ran - then its facts: Shards (for a sharded run), Branch, Commit, Author, Started and Duration.

The newer run’s card marks what changed against the older one. Branch and Commit get a changed badge when they differ; a changed commit also links to View changes since the older commit on GitHub, when the project is linked to one repository. Duration shows the difference, for example “+55s”, and how many times slower the run got from 1.1 times.

While a run’s counts can still change - a shard is still running, or one is missing while the run is not yet overdue - its card says “Partial — these counts cover only part of the run.”, by the same rule as the run page. A run that is over never says it - passed, failed, cancelled with every shard reported, or settled incomplete; a settled incomplete run’s card only names the shards that sent no results. When the preselected partner is incomplete, the notice says how many of its shards reported - and, while its counts can still change, that they are partial.

Under the cards, Between the runs names the Playwright version, the number of workers and the number of shards - only where they differ. The line is left out when there is nothing to say.

Result

Result draws both runs’ result bars on one scale, the older above the newer, so a run with fewer tests has the shorter bar. Under the bars it names only what differs, for example “+26 tests · +23 passed · +1 failed”; when the counts agree, it names them once and says “same counts in both runs”.

The counts are counted like the runs list and the run page: every test once, however many shards reported it, by its final attempt, and the tests a cancel cut off not at all. A cancelled run therefore has the same total as its row in the runs list.

The groups

Every test that ran in either run is sorted into one group:

Group	Test
Newly failing	failed now, passed, flaky or skipped before - or new and failed
Newly flaky	flaky now, not before - or new and flaky
Still failing	failed in both
Still flaky	flaky in both
Removed	ran in the older run, not in the newer
Now skipped	ran before, skipped now
Fixed	failed or flaky before, passed now
Added	new in the newer run, and passed or skipped
No longer skipped	skipped before, passed now
Unchanged	passed in both, or skipped in both

A test that is new and fails is Newly failing, not Added. A sharded run is compared as a whole - all of its shards, not one. Tests are paired by name, file and browser, so a renamed test shows as one removed and one added. A Still failing row whose error differs from the older run’s gets a changed badge; one without the badge failed the same way. The sentence at the top says so too, for example “2 tests still fail, 1 with a different error”.

Which tests are shown

Above the table is one chip per group the comparison has, in the order of the table above, each with its count. Turn any of them on or off; the table shows the groups that are on.

The page opens on the groups that need attention: Newly failing, Newly flaky, Still failing, Still flaky, Removed and Now skipped. When none of them has a test, it opens on the changes - Fixed, Added and No longer skipped - and when nothing changed at all, on the unchanged tests.

Each row carries a small square in its group’s colour; the chips are its legend. The rows stay in group order. The table shows 20 rows; Show 20 more adds the next ones.

The views Needs attention, All changes and All tests were replaced by the chips on 29.10.2026; an old link that names one of them opens on the default groups.

Δ time

Each row has a status for each run and Δ time: the newer run’s time on the test minus the older run’s, with all attempts added up - what the run spent on the test. It is marked when the test got at least 50% and 2 s slower, with how many times slower it got. A test that is new, removed or skipped on either side has no Δ time. Click the Δ time heading to sort by it, largest increase first; the groups then mix, and each row keeps its square.

One test, two runs

A row whose test ran in both runs opens it across both. Added and Removed rows are not links: there is nothing to compare. The page starts with the test’s name, its file and browser, View code changes or “Same commit”, and one sentence of facts.

The table compares the test in each run, one column per run:

Field	Shows
Status	the test’s result in that run, as its icon
Branch	the run’s branch - never marked as a difference
Shard	which shard ran it - only when a side is sharded
Attempts	how many attempts it took
Browser	the browser (Playwright project) it ran in - only when a side names one; the same on both sides, since tests are paired by it
Error	the category of the last failed attempt
Duration, all attempts	all attempts added up

Differences (the default) shows only the rows that differ; All fields shows every row. A row that differs gets a changed badge on its name, and the newer run’s value is drawn in bold. A duration counts as changed from half a second. When nothing differs, the page says “No differences. Same status, same attempts, same error, same shard.”

Evidence shows one card per run, older on the left. Each card opens on the last attempt, or on the failed one when the test passed on retry; switch attempts on either card. Each card shows:

• Error: the attempt’s error message, or that it had none;
• Screenshot, when one was recorded;
• Console lines only in this run: the lines of the browser console that the other run’s attempt does not have, at most eight. Numbers and whitespace are ignored when comparing, so “504 in 2.4 s” and “504 in 2.6 s” count as the same line.

The table describes the test in each run; a card describes one attempt. The run number on a card opens that run’s own page for the test.