Sharded runs

One run, however many shards

Run Playwright with --shard=1/3, --shard=2/3 and so on, as you would anyway. Every shard sends its results with the same run ID, its shard number and the shard total, and Squally joins them into one run: one entry in the runs list, one run number, and a row per shard on the run’s page that shows how each shard did.

The run is complete when every shard has reported its end. Until then it is Running - also while it waits for a shard that has not started yet, until the run timeout has passed since the run’s last activity. An empty shard still reports, so a shard that got no tests does not hold the run open. A shard that never reports leaves the run Incomplete once that time has passed - see Incomplete: waiting or settled.

How a sharded run is counted

Every count of the run - the runs list’s, the run page’s, the read API’s - takes each test once. When two shards both reported a test, usually a --shard misconfiguration, it is still one test of the run: its final attempt across both decides - the most retries first, then the lower shard, then the first repetition. The run page’s Tests tab lists it once, naming the shard its final attempt ran in.

Where the run ID comes from

The reporter reads it from the CI provider - an ID that all jobs of one CI run share, such as GITHUB_RUN_ID or CI_PIPELINE_ID. The full list is on Other providers. Set SQUALLY_CI_RUN_ID to the same value in every shard:

• on TeamCity and on a CI the reporter does not recognise, which have no such ID;
• when you shard locally, across several separate playwright test commands.

SQUALLY_CI_RUN_ID wins over the provider’s ID everywhere. Jobs that are not shards of each other need a job key instead - below.

Jobs that are not shards

Everything in one CI run shares the provider’s ID - also jobs that are not shards of each other: a matrix over browsers, an api and a ui suite. Since squally-reporter 0.11.0, give each job its own SQUALLY_CI_JOB_KEY. The reporter adds it to the run ID - <GITHUB_RUN_ID>-<key>-<GITHUB_RUN_ATTEMPT> on GitHub Actions, <provider ID>-<key> elsewhere - so each job becomes a run of its own, with its own shards.

A suite that is split into jobs and sharded takes one key per suite, shared by that suite’s shards:

strategy:
  fail-fast: false
  matrix:
    suite: [api, ui]
    shard: [1, 2]
env:
  SQUALLY_INGEST_KEY: ${{ secrets.SQUALLY_INGEST_KEY }}
  SQUALLY_CI_JOB_KEY: ${{ matrix.suite }}
steps:
  - run: npx playwright test tests/${{ matrix.suite }} --shard=${{ matrix.shard }}/2
  - name: Mark Squally run cancelled
    if: ${{ cancelled() }}
    run: npx squally-reporter cancel

The job key sits on the job, so the optional cancel step builds the same run id as the reporter - see Reporter CLI. fail-fast: false keeps one failing shard from cancelling the others: with the cancel step, those cancels would mark the whole run Cancelled instead of Failed.

That is two runs in Squally, api and ui, of two shards each. A re-run of failed jobs that re-runs one ui shard links to the ui run of the attempt before and waits only for that shard.

Without keys, Squally refuses a job of reporter 0.11.0 that would share a run with another one: a second job on the same shard, or a second unsharded job (ci_run_collision), and an unsharded job beside a sharded one or shards of different totals (ci_run_mixed). See Troubleshooting and Reporter setup.

Uploads are paced

Results go to Squally while the suite runs, in batches. Each shard sends a batch when 25 results are ready, or else on a timer: every 2 seconds, or 0.4 seconds × the number of shards when that is longer, at most 30 seconds. All shards share one ingest key and its rate limit, so from six shards on each one waits a little longer; up to five, each sends as often as an unsharded run.

Retries that keep the run ID

Some providers keep the run ID when you retry a job - a GitLab job retry, an Azure Pipelines re-run, a Bitbucket re-run of failed steps, a Buildkite retry, a Travis restart. The retried job then reports to the run its first attempt created, and a job key does not change that. When both attempts ran squally-reporter 0.11.0, Squally refuses the retry at its start (ci_run_reused once the first attempt’s run has ended, ci_run_collision while it still runs). Otherwise the retry’s results go to that run and are refused once it is complete; the reporter says once that Squally has already completed the run, suggests a SQUALLY_CI_JOB_KEY unique per job (a reporter older than 0.11.0 says SQUALLY_CI_RUN_ID), and sends nothing more. The job’s exit code is not affected either way.

On GitHub Actions a re-run is a new run, because the reporter adds the attempt number. Re-run failed jobs runs only the shards that did not pass and reuses the others without running them. The new run waits only for the shards that re-ran: it completes once they have reported, and passes or fails on the tests they ran - the reused shards are neither waited for nor counted. A re-run shard that never reports leaves the run Incomplete, as above. Re-run all jobs runs every shard again, and every shard that reports counts.

The re-run carries a Re-run of #N chip that links to the attempt it re-ran, and its shard chip counts what it waits for: a re-run of two failed shards out of three reads “shards 2/2”, and the third shard’s row in its Shards panel says “passed in #N · not re-run”. When more shards report than the run waited for - Re-run all jobs after only one shard failed - the chip shows the count alone, “shards 4”.

The re-run’s GitHub check, pull request comment and Slack message count only its own shards, too, and say so: “Run #28 · re-run of #27 · 43 passed · 0 flaky · 3 failed · 1 skipped”, then “Re-run of #27: shards 1 and 2 re-ran; shard 3 passed in #27 and was not re-run.” See GitHub Actions.

Not merge-reports

Do not point the reporter at the output of playwright merge-reports. The merge job reads branch and commit from its own machine, not from the shards’, and loses them without a checkout. Sharded runs join on their own.