GitHub

What it does

• A check on every commit. When a CI run finishes, Squally posts a check named Squally on the commit the run reported, in the linked repository the tests ran in - see Which repository to link - Squally · chromium, Squally · api and so on for a run with a job key. It succeeds unless a test failed - a test that passed after a retry does not fail it. With several repositories linked, a run posts only to the one it ran in; a run that does not name its repository posts to all of them.
• One comment per pull request. For CI runs on a pull request, Squally writes one comment with the results and rewrites that same comment on every later run of the pull request - with one block per job when the workflow has several jobs. A comment is only created once something failed or was flaky; once it exists, it is updated on every run, a green one included. Pull-request comments are off until an admin turns them on for the project.
• Only for CI runs. A local run - one no shard of which reported a CI provider - posts no check and no comment, even when its commit is on GitHub. Its run page says “Not posted · local run”. The pages count CI and local runs by default, but a check speaks for a commit to everyone on the pull request, and a run from someone’s laptop - a single test, or one failing on purpose while it is written - should not pass or fail that commit. Every check says it at the end: “CI runs only: local runs never post a check.”

Runs report pull requests under their head commit, so the check and the comment land on the pull request - see GitHub Actions.

The check

The check’s title is the run’s number and its four counts, always in this order and zeros included:

Run #128 · 64 passed · 2 flaky · 1 failed · 0 skipped

Passed is a test that passed on the first try, flaky one that passed only after a retry, and the four add up to the run’s tests - each test counted once, however many shards reported it, by its final attempt across all of them; the run page shows the same numbers (see Run summary). The check lists the failed tests and the flaky ones by name and file path - with the Playwright project in brackets where the test ran under one, for example “applies a coupon code (firefox) — tests/checkout.spec.ts” - and links to the run in Squally. It carries no error messages or stack traces: those stay behind Squally’s sign-in. Its duration on GitHub runs from the run’s first start to the moment the check was posted.

Check posted on the run’s page opens the check itself on GitHub - see Run summary. For a run older than that link (06.10.2026), and for a check posted to several linked repositories, it opens the commit’s list of checks.

One check per job

A run with a job key - SQUALLY_CI_JOB_KEY, squally-reporter 0.11.0, see Matrix jobs - gets a check named after its key: Squally · chromium, Squally · firefox. A run without a key keeps the name Squally, exactly as before.

GitHub shows the newest check of each name on a commit. With one name for every job, the job that finished last replaced the others’ checks - a green chromium job could hide a red firefox one. With a key, each job keeps its own check beside the others. A re-run of a job posts its check under the same name, so it replaces that job’s check - and only that job’s.

Required checks

A branch protection rule or ruleset that requires a check named Squally no longer matches once a job has a key: that job’s check is now Squally · chromium, and the required Squally never arrives. Require each job’s check instead - Squally · chromium, Squally · firefox. Renaming a key renames its check, so update the rule when you change a key.

A re-run

A re-run on GitHub Actions is a run of its own in Squally, and its counts cover only the shards that ran again. When the attempt it re-ran reported to Squally too, the check says so. Its title names that attempt between the run’s number and the counts:

Run #28 · re-run of #27 · 43 passed · 0 flaky · 3 failed · 1 skipped

and the first line of its text says which shards ran again and which did not:

Re-run of #27: shards 1 and 2 re-ran; shard 3 passed in #27 and was not re-run.

Each run number in that line links to the run in Squally. The other wordings follow the same pattern:

• an unsharded re-run: “Re-run of #27.”;
• Re-run all jobs, where every shard ran again: “Re-run of #27: shards 1, 2 and 3 re-ran.”;
• a re-run of a re-run, whose shards passed in different attempts: “Re-run of #28: shard 1 re-ran; shard 2 passed in #28 and shard 3 passed in #27; neither was re-run.”;
• a shard whose run is no longer in Squally: “… passed earlier and was not re-run.”

The rest of the text - the failed and flaky tests, the link - is the same as for any run.

Re-run on the Squally check

GitHub shows a Re-run button on Squally’s check, as it does on every app’s check, and confirms a click. Squally does nothing with it: it cannot start tests, so the check stays as it is. To test again, re-run your CI workflow - on GitHub Actions, Re-run jobs on the workflow run, or push a commit. The new run posts its own check. Every check’s text says so, above its last lines: “To test again, re-run your CI workflow.”

A run that did not complete

When a run ends incomplete - a shard timed out, never reported, or was stopped by the monthly result limit - Squally posts one check when the run settles, once the run timeout has passed:

• its title is the run’s line with “incomplete” added, for example Run #128 · 40 passed · 0 flaky · 1 failed · 0 skipped · incomplete - the results its shards reported;
• it says which shards timed out or never reported, and lists the failed tests. For a re-run, its first line is the re-run line, and a shard it re-ran that never reported reads “Shard 2 never reported (re-run of #27).”;
• its conclusion is timed out - or neutral when the monthly result limit stopped the run, because the limit is not a test problem and should not block a pull request like a failure.

It is posted once, whatever the cause. A shard that still reports after the run settled joins the run’s counts in Squally, but nothing is posted again.

A cancelled run

A cancelled run gets the check Run cancelled, with the conclusion cancelled. It counts only the tests that finished before the stop - the ones the cancel cut off are not in it - lists those that failed, and names the shards that sent nothing. A cancelled re-run’s text starts with the re-run line. A cancelled run does not update the comment, so a comment from an earlier, finished run is not replaced by a fragment. When another job’s run rebuilds the comment, a cancelled job’s block says “cancelled”.

The comment

A pull request has one Squally comment. Each time one of the project’s CI runs on it finishes, Squally rebuilds the comment from the project’s CI runs on the pull request’s current head commit: the latest run of each job - a re-run replaces the attempt it re-ran. Runs on older commits of the pull request and local runs are not in it. Two jobs that finish in the same second no longer overwrite each other: Squally rebuilds one pull request’s comment at a time, so the last rebuild always includes every job.

The rule for creating it is unchanged: a comment is only created when some job’s latest run failed or was flaky. Once it exists, every rebuild updates it in place - so it turns green after a successful re-run.

One job

A workflow with one job and no job key gets the comment it always got. Its heading is the same line as the check’s title, for example Squally · Run #128 · 8 passed · 2 flaky · 0 failed · 0 skipped. Below it, the Failed tests and the Flaky (passed on retry) ones, each with its Playwright project and file, and a link to the run. A test that passed after a retry is counted once, as flaky - not as passed and flaky. The comment carries no error messages or stack traces either: it lands in the inbox of everyone watching the repository.

For a re-run, the heading is the check’s re-run title, and under it comes the check’s re-run line with one sentence added:

Squally · Run #28 · re-run of #27 · 43 passed · 0 flaky · 3 failed · 1 skipped

Re-run of #27: shards 1 and 2 re-ran; shard 3 passed in #27 and was not re-run. The counts are for this re-run only.

A pull request still has one comment: the re-run rewrites the one the earlier attempt wrote, so its counts now cover only the shards that ran again - which is what the added sentence says.

Several jobs, one block each

When the pull request’s head commit has runs of more than one job - a browser matrix with job keys, or a job with a key beside one without - the comment’s heading is just Squally, and each job gets a block of its own: the job without a key first, then the keys in alphabetical order. Each block’s heading starts with a mark and the job key, then the run’s line as in the check’s title:

### Squally

#### ❌ Run #48 · 63 passed · 3 flaky · 2 failed · 2 skipped
#### ✅ chromium · Run #49 · 18 passed · 0 flaky · 0 failed · 1 skipped
#### ✅ firefox · Run #51 · re-run of #50 · 19 passed · 0 flaky · 0 failed · 1 skipped
#### ⏳ webkit · Run #52 · running

• ✅ - the run passed, flaky tests included; ❌ - it failed, was cancelled or ended incomplete (the heading then ends in “· cancelled” or “· incomplete”); ⏳ - the job is still running, and its block is that one line.
• A job without a key has no label: its heading starts with the run’s number.
• A re-run’s block replaces the block of the attempt it re-ran, and the line under its heading says so: “Re-run of #50. The counts are for this re-run only.” - with the shards, as in the check, for a sharded job.
• Under that, each block lists its Failed and Flaky (passed on retry) tests, up to 15 each and then “+N more in Squally” - or, with neither, “All tests passed on the first attempt.” (“No test failed.” when some were skipped) - and links to its run: “View run #51 in Squally”.

GitHub refuses a comment over 65,536 characters. When the comment would pass 60,000, the lists of the largest block shrink first - to 10, 5, 3, 1 and then no tests, the rest counted in “+N more in Squally” - until the comment fits.

When posting fails

A request to GitHub that does not answer is given up: 5 seconds for the token Squally needs first, 10 seconds for the check or the comment itself. Nothing is retried. The run page then says “Posting to GitHub timed out — check the integration” (or “failed”, with GitHub’s status code), and Squally’s own error tracking is told - with the run’s ids and the failure code only, see Hosting and data.

Which repository to link

Link the repository the tests run in - the one your CI checks out and runs the reporter in. The run reports that repository’s commit, and the check and the comment go on it.

When the application lives in one repository and the Playwright tests and their CI in another, link the test repository. If the project is linked to the application repository instead, Squally recognises that the run came from another repository and posts nothing to GitHub - no check, no comment: GitHub would refuse a check on a commit that is not in the repository. The run’s page says Not posted to GitHub and, under the result card, “Not posted to GitHub — tests ran in acme/shop-e2e, but this project is linked to acme/shop.” An organization admin gets a link to the project’s GitHub settings (“Link acme/shop-e2e in project settings”); a member reads “Ask an admin to link acme/shop-e2e in project settings.” The link opens the settings with acme/shop-e2e already chosen in the repository picker - linking is still your click on Link - and, when another project already links that repository, with the note naming the check this project would get. The commit and file links on the run point to the repository the tests ran in. To record which application version was tested, see Tested revision.

Squally can only tell when the run names its repository - the reporter sends it since 0.16.0, unless SQUALLY_SEND_REPOSITORY=false - and the repository is on github.com.

A project that does not run on GitHub. While a project links no repository, the GitHub card on its Integrations tab reads “No repository linked” in amber. When every one of the project’s recent runs names a repository on another host - GitLab, Azure Repos - the card says so in grey instead: “Runs come from gitlab.com - GitHub checks do not apply”. One run from github.com, or one that names no repository, keeps the amber line.

A renamed repository. When a linked repository is renamed on GitHub, its links follow at once: GitHub tells Squally about the rename, and every project linked to the repository gets the new name - also a project with several linked repositories. A project that already links the new name keeps its old link as it is. Should that message not arrive, the next run from the repository is still delivered when the project has one linked repository: Squally asks GitHub for the repository’s current name and updates the link then. After a transfer to another GitHub account, link the repository again.

Connecting

The Squally GitHub App can be installed on any GitHub account, on the repositories you select. Everything happens from a project - the organization has no Integrations page of its own. There are two ways in:

• From a project: an organization admin clicks Set up on the GitHub card of the project’s Integrations tab, or Install the GitHub App on its GitHub page. GitHub installs the app on the account you choose and brings you back to that project’s GitHub page to pick its repository. The installation still belongs to the organization; every project can use it.
• From the app’s page on GitHub (github.com/apps/squally-checks): install it there. GitHub sends you to Squally afterwards - sign in if you are not - and the installation is connected to the organization that is active for you in Squally at that moment.

Either way, GitHub asks you to authorize the app, and Squally connects the installation only when you are an admin on both sides: an admin of the Squally organization, and an admin of the GitHub account - an owner of the GitHub organization, or the account itself for a personal account. Squally asks GitHub this once, with a token it uses for nothing else and does not keep. Admins then link one or more of the repositories to each project. After an install from a project, you land on that project’s GitHub page with “GitHub connected. Pick the repository for this project below.”; after one from GitHub’s own page, on the GitHub page of the organization’s first project.

If you can only request the installation - you are not an owner of the GitHub organization - Squally says the request was sent. An owner of that GitHub organization who is also an admin in Squally then connects the app.

One Squally organization per installation. The first organization that connects an installation keeps it. To move it to another organization, uninstall the app on GitHub and install it again from there.

Manage connection. On a project’s GitHub page, admins see the organization’s GitHub accounts under Manage connection:

• Manage on GitHub opens the app’s settings on GitHub - choose which repositories it can see there, or uninstall it. GitHub shows that page only to the account itself, or for a GitHub organization to its owners, and answers everyone else with “404”. So next to the button Squally says whom to sign in to GitHub as: “Sign in to GitHub as dana to open this page.” or “Sign in to GitHub as an owner of acme to open this page.”
• Connect another GitHub account installs the app on one more account.

When Squally can’t reach GitHub for the settings link, the account is listed without Manage on GitHub. Open the app’s settings on GitHub directly instead: Settings › GitHub Apps of a GitHub organization, or Settings › Applications › Installed GitHub Apps of a personal account.

Disconnecting works on two levels:

• One project lets go of a repository with Unlink on its row. Only that project’s link goes; other projects linked to the same repository keep posting, and the app stays installed.
• Every project is disconnected by uninstalling the app on GitHub - Manage on GitHub, then Uninstall. That detaches the account from all projects of the organization at once, as described next.

Uninstalling removes the links. When the app is uninstalled on GitHub, Squally deletes the installation and every repository link that went through it. After installing the app again, an admin links the repositories again in each project. The next comment on a pull request is then a new one; the old one stays on GitHub, no longer updated. When GitHub cannot confirm an installation - it is unknown, connected elsewhere, or not yours - Squally answers with the same sentence every time: “We could not connect that GitHub installation.”

Several GitHub accounts. An organization can connect the app on more than one account - a GitHub organization and a personal account, say. Manage connection lists every account and offers Connect another GitHub account; a project’s repository picker lists the repositories of all of them, and each link goes through the account that owns the repository. A repository another project already links reads “linked in Demo Shop” in the list.

A suspended app. When GitHub suspends the app on an account, Squally posts nothing to that account’s repositories and says so - on the project’s GitHub page, beside the account and the repository, and on each run (Not posted to GitHub, with “The GitHub app is suspended on the repository’s account” on hover). The links stay; once the app is unsuspended, checks and comments go out again.

Permissions

• Repository metadata: read.
• Checks: read and write.
• Pull requests: read and write - for the comment.
• Organization members: read - only to confirm, while connecting, that the person installing the app is an owner of the GitHub organization.

The app has no access to your code.