Cancelled runs

What Cancelled means

A run that was stopped from outside - a cancelled CI job, a Ctrl+C - is marked Cancelled in Squally: a small grey stop square in place of a passed or failed mark.

• Its own status. The result menu in the runs list’s filter sentence (any result) has Cancelled, and a cancelled run matches it and no other status. The Overview’s Branches pass rate and median duration leave it out, every shard of a cancelled CI run included; its Runs chart counts it as Cancelled or incomplete.
• Its finished tests still count. The tests that finished before the stop count toward each test’s metrics like any other result.
• The tests the cancel cut off count nowhere. Playwright reports them as skipped. They are not in the run’s counts - the runs list’s counts, the run page’s result card, the read API’s counts - nor in any test’s metrics. The run’s Tests tab still lists them, as skipped.
• A sharded run is cancelled when any of its shards recorded a cancel. It ends at the latest shard that reported, even if a shard stopped by the same cancel never did. While another of its shards is still running, the run is not finished yet and counts nowhere; it counts as cancelled once that shard ends or the run timeout has passed.

GitHub and Slack

A cancelled CI run gets one GitHub check, Run cancelled, with the conclusion cancelled: it counts only the tests that finished before the stop and names the shards that sent nothing. It does not update the pull-request comment and sends no Slack message - most cancels are a superseded push. When another job’s run rebuilds the comment, the cancelled job’s block says “cancelled” - see GitHub. A cancelled local run posts nothing at all.

In the read API, a cancelled run’s status is cancelled, and cancellation says how and when (source, at, byUserId); ?status=cancelled lists cancelled runs. See List runs and Get a run. Until read API 1.0.0-beta.14 a cancelled run kept the outcome of what it reported as its status, with a cancelled boolean beside it (API changelog).

How it happens

From Squally. An admin can cancel a running run from its page, with Cancel run… in the run menu. Squally marks the run cancelled at once - every shard still running - and discards results sent after that; the tests that finished stay in the run. It does not stop your CI job.

From CI. A step that runs when the job is cancelled calls npx squally-reporter cancel (squally-reporter 0.14.0). It marks the run the same way - every shard still running. The run ends as cancelled either way; the source it shows depends on who reported first:

• If a shard was running tests and Playwright itself noticed the cancel, the reporter reported it: Cancelled via the reporter, cancellation.source reporter. The cancel step then only marks the rest of the run (shards that had not started), and the source stays reporter.
• If no shard was running tests - or the cancel never reached Playwright - the cancel step marks the run: Cancelled via the CLI, cancellation.source cli.

See Reporter CLI.

Before every shard has started. Both ways also work when shards of a sharded run have not begun yet - matrix jobs still queued, say. Squally then marks the whole run cancelled right away: it is finished and shows as cancelled, and its GitHub check (with a linked repository) says so. A shard that starts later is refused (HTTP 409, run_cancelled): the reporter prints “This run was cancelled in Squally” and sends nothing from that job. Your test run and its exit code are not affected.

From the test run. Playwright has to be stopped with SIGINT - Ctrl+C, or a CI job cancelled with that signal - and Playwright itself has to receive it. Playwright then ends the run; the reporter (0.10.0 or newer) sends what finished and tells Squally the run was interrupted. Your exit code stays Playwright’s own, 130.

Two cases are not a cancel:

• A run stopped before its first test starts sends nothing, like any run that never ran a test.
• A run stopped by globalTimeout is simply failed.

GitHub Actions: add the cancel step

A cancelled workflow sends SIGINT to the test step’s shell, not to Playwright, so the reporter does not get to report the end. Add a step after the tests that marks the run cancelled:

- name: Mark Squally run cancelled
  if: ${{ cancelled() }}
  run: npx squally-reporter cancel

Set SQUALLY_INGEST_KEY - and SQUALLY_CI_JOB_KEY, if you use one - in the job’s env:, so this step sees them too, and use fail-fast: false on a sharded job: otherwise one failing shard cancels the others and the whole run shows as cancelled instead of failed. See GitHub Actions.

Without the step, the run is handled as below and shows as Incomplete. Other CI systems deliver a cancel in their own way; when Playwright itself does not receive it, the run ends the same way, unless your CI runs the cancel command in a step of its own.

A hard kill ends as incomplete

When the cancel does not reach Playwright - or after any hard kill (SIGKILL) - the reporter never gets to report the end. The run stays running until Squally’s hourly check marks it incomplete, once the run timeout has passed - 60 minutes by default (set per project), counted from the last time the run sent anything. An incomplete run is not a cancelled one, but it counts the same way: the tests that finished count, the Overview’s Runs chart counts the run as Cancelled or incomplete, and it stays out of a branch’s pass rate and median duration on the Overview and out of Analytics’ pass rate and run duration. See Timeouts.