GitHub Actions

The workflow

This workflow is based on the one of the public demo project, squally-demo: three shards, two browsers, one Squally run per workflow run. For the single-job workflow npm init playwright writes, see Reporter setup.

name: Tests

on:
  push:
    branches: [master]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      # One failing shard must not cancel the others.
      fail-fast: false
      matrix:
        shard: [1, 2, 3]
    env:
      SQUALLY_INGEST_KEY: ${{ secrets.SQUALLY_INGEST_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium firefox
      - name: Run Playwright tests
        run: npx playwright test --shard=${{ matrix.shard }}/3
      - name: Mark Squally run cancelled
        if: ${{ cancelled() }}
        run: npx squally-reporter cancel

Store the ingest key as a repository secret named SQUALLY_INGEST_KEY, under Settings → Secrets and variables → Actions → New repository secret. It is set on the job, not on the test step, because the cancel step reads it too. The reporter sends to https://app.squally.dev; add SQUALLY_INGEST_URL only for another host - it is not a secret. The demo also sets SQUALLY_CI_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }} - the value the reporter uses on GitHub Actions anyway, so you can leave it out.

In playwright.config.ts, replace the reporter line and check that retries are on in CI - Playwright’s template already has that line:

import { defineConfig } from "@playwright/test";

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: [["html"], ["squally-reporter"]],
  use: {
    screenshot: "only-on-failure",
    trace: "on-first-retry",
  },
});

The trace is recorded only when a test is retried - one per flaky or failing test, which keeps stored artifacts small.

Replace, don’t add: with a second reporter: key the later one wins without a warning. Keep any other reporters in the list. Without retries Squally cannot tell a flaky test from a broken one - see Playwright flaky tests.

Cancelled workflows

A cancelled workflow sends SIGINT to the test step’s shell, not to Playwright, so the reporter never reports the end. The optional last step marks the run cancelled instead:

- name: Mark Squally run cancelled
  if: ${{ cancelled() }}
  run: npx squally-reporter cancel

It runs only when the workflow was cancelled, finds the run by the same id the reporter sent from this job, and marks it Cancelled - “Cancelled via the CLI” on the run page. It never fails the step. See Reporter CLI.

exec: let Playwright report the cancel itself

GitHub sends the SIGINT to the process at the top of the test step. With run: npx playwright test that is the step’s shell, which does not pass the signal on: Playwright is killed without noticing, and the cancel step marks the run - source cli, as above.

Started with exec, Playwright replaces the shell and is the top process itself:

- name: Run Playwright tests
  run: exec ./node_modules/.bin/playwright test --shard=${{ matrix.shard }}/3

Then a shard that is running tests when the workflow is cancelled stops the way Ctrl+C stops it: the reporter sends the tests that finished, reports the interrupted ones as skipped and tells Squally the run was cancelled. The run page says Cancelled via the reporter, the read API’s cancellation.source is reporter. The cancel step still runs afterwards and marks only what nobody reported - shards that had not started yet - printing (0 shards); the source stays reporter.

Either way the run ends as Cancelled; exec only decides who reports it first. Keep the cancel step with exec too: it covers the shards that were still installing or queued. exec has to be the step’s last command - nothing after it runs. See Reporter CLI.

• The key on the job. The step needs SQUALLY_INGEST_KEY as well, so set it in the job’s env:, as above.
• SQUALLY_CI_JOB_KEY on the job. With a job key the step has to build the same id, so the key belongs in the job’s env:, not only on the test step.
• fail-fast: false. With GitHub’s default fail-fast: true, one failing shard cancels the other shards, their cancel steps run, and the whole run shows as Cancelled instead of Failed.

Without the step a cancelled workflow shows as Incomplete: the run stays Running until Squally’s hourly check ends it. You can also cancel a run from its page in Squally. See Cancelled runs.

Shards and re-runs

All shards of one workflow run share GITHUB_RUN_ID and become one run in Squally - see Sharded runs. The reporter adds GITHUB_RUN_ATTEMPT, so a re-run of the workflow is a new run in Squally. Since squally-reporter 0.11.0, each run’s page links to the page of exactly its attempt on GitHub, …/actions/runs/<id>/attempts/<n>, as View in CI.

• Re-run failed jobs runs only the shards that did not pass and reuses the others without running them. Squally’s run for that attempt waits only for the re-run shards: it completes once they have reported, and passes or fails on the tests they ran. The reused shards are neither waited for nor counted. A shard whose tests passed on retry passed - GitHub does not re-run it either. A re-run shard that never reports still leaves the run incomplete once the run timeout has passed.
• Re-run all jobs runs every shard again; every shard that reports counts.
• In Squally, the re-run links to the attempt it re-ran, when that attempt reported to Squally: its row in the runs list and its page carry a Re-run of #N link, and the earlier run’s page a Re-run: #M link pointing forward. The re-run’s Shards panel lists the shards it did not re-run as “passed in #N · not re-run”, and the Compare tab opens on the earlier attempt - see Run comparison.
• What a re-run posts. Its check, pull request comment and Slack message count only the shards that ran again, and say so. The check’s title reads “Run #28 · re-run of #27 · 43 passed · 0 flaky · 3 failed · 1 skipped”, and its first line “Re-run of #27: shards 1 and 2 re-ran; shard 3 passed in #27 and was not re-run.” The comment has the same line and adds “The counts are for this re-run only.” - the pull request’s one comment now describes the re-run. Slack adds “Re-run of #27 (shards 1, 2)” under the counts. A re-run shard that never reports reads “Shard 2 never reported (re-run of #27).” in the run’s incomplete check.
• A re-run started before the earlier attempt finished supersedes it: nothing more is posted for the earlier attempt - no late “did not complete” check over the re-run’s.
• A matrix without --shard - for example one job per browser - shares the run ID too. Give each job a key with SQUALLY_CI_JOB_KEY - see Matrix jobs.

Matrix jobs

Jobs of one workflow run that are not shards of each other - one job per browser, an api and a ui suite - need a job key each (squally-reporter 0.11.0), shared only by that job’s own shards:

strategy:
  fail-fast: false
  matrix:
    browser: [chromium, firefox]
env:
  SQUALLY_INGEST_KEY: ${{ secrets.SQUALLY_INGEST_KEY }}
  SQUALLY_CI_JOB_KEY: ${{ matrix.browser }}
steps:
  # …checkout, setup-node, npm ci, playwright install as above
  - name: Run Playwright tests
    run: npx playwright test --project=${{ matrix.browser }}
  - name: Mark Squally run cancelled
    if: ${{ cancelled() }}
    run: npx squally-reporter cancel

The reporter builds the run ID <GITHUB_RUN_ID>-<key>-<GITHUB_RUN_ATTEMPT>, so each job is a run of its own in Squally:

• A chip with the key beside the run’s number - in the runs list, on the run page, in the run report and on the comparison’s cards.
• A check of its own, named “Squally · chromium”, “Squally · firefox” - side by side on the commit instead of the last job’s hiding the other’s. If branch protection requires a check named “Squally”, it has to require each job’s check instead - “Squally · chromium” and “Squally · firefox”; see GitHub.
• Its own re-runs. “Re-run failed jobs” re-runs only the failed job; its new attempt is linked to that job’s own earlier attempt - never to another job’s - and its check replaces that job’s check. In the Demo Shop, firefox run #50 failed beside chromium #49; the re-run of the firefox job became #51, “Re-run of #50”, and chromium was not run again.
• One block in the pull request’s comment. On a pull request the comment has one block per job, the latest run of each - a re-run replaces its job’s block.

Without a key both jobs would share one run ID, and Squally refuses the second job at its start (ci_run_collision) - see Troubleshooting. The hand-built SQUALLY_CI_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.browser }} these docs suggested before 0.11.0 still groups each job, but its re-runs are never linked to the attempt before them; switch to the key.

A suite that is split into jobs and sharded takes one key per suite - SQUALLY_CI_JOB_KEY: ${{ matrix.suite }} beside --shard=${{ matrix.shard }}/2. See Sharded runs.

One run per commit

A workflow that runs on both push and pull_request without a branch filter runs twice for every commit pushed to a pull request’s branch: once for the push, once for the pull request. Squally stores each as a run of its own, with its own check, so the commit counts twice - in the run list, in the trends and in the monthly run limit. Two ways to keep it to one run:

• Filter push to the default branch, as the workflow above does. Pull requests run through pull_request, merged commits through push:

  on:
    push:
      branches: [main]
    pull_request:

• Let the newer run cancel the older one with concurrency. The group is the workflow and the branch - github.head_ref on a pull request, github.ref_name on a push - so the push and the pull request share it:

  concurrency:
    group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }}
    cancel-in-progress: true

  A run cancelled after it started reporting shows as cancelled in Squally - see Cancelled workflows.

Squally does not merge such pairs into one run: two runs of one commit can also be deliberate, such as two configurations.

Pull requests

On pull_request and pull_request_target, the run is reported under the pull request’s own branch and its head commit - not the merge commit GitHub checks out. That is the commit Squally’s GitHub check and pull-request comment attach to. The commit message and author are the head commit’s.

actions/checkout fetches one commit by default, and the head commit is not part of it. The reporter then fetches that single commit from origin, for at most 3 seconds and only in a shallow clone. If that fails, message and author stay empty and the reporter says so once; the run is unaffected. fetch-depth: 2 on the checkout step makes the fetch unnecessary; SQUALLY_DISABLE_HEAD_COMMIT_FETCH=true turns it off.

Outside pull requests, the run is reported under GITHUB_REF_NAME and GITHUB_SHA. The author is the commit’s author from git log in the checkout (after CI_COMMIT_AUTHOR, which only GitLab sets); GITHUB_ACTOR - whoever started the workflow - is used only when git cannot answer.