Azure Pipelines

The pipeline

azure-pipelines.yml

trigger:
  - main

pool:
  vmImage: ubuntu-latest

jobs:
  - job: playwright
    strategy:
      parallel: 3
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: "22.x"
      - script: npm ci
        displayName: Install dependencies
      - script: npx playwright install --with-deps
        displayName: Install browsers
      - script: npx playwright test --shard=$(System.JobPositionInPhase)/$(System.TotalJobsInPhase)
        displayName: Run Playwright
        env:
          # Secret variables reach a script only through env: - see below.
          SQUALLY_INGEST_KEY: $(SQUALLY_INGEST_KEY)

For a single job, drop strategy: and --shard=…. The reporter needs Node.js 20.18.1 or later.

In playwright.config.ts, replace the reporter line and check that retries are on in CI - Playwright’s template already has that line:

import { defineConfig } from "@playwright/test";

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: [["html"], ["squally-reporter"]],
});

Replace, don’t add: with a second reporter: key the later one wins without a warning. Keep any other reporters in the list.

The ingest key needs an env: mapping

Add SQUALLY_INGEST_KEY as a pipeline variable (Pipelines › your pipeline › Edit › Variables) and select Keep this value secret - or put it in a variable group.

Azure Pipelines does not hand secret variables to scripts as environment variables, unlike most CIs. A script sees one only when the step maps it in env:, as in the pipeline above. Without that line the reporter finds no key, prints no ingest key configured, skipping report and sends nothing. Your tests and the job’s exit code are not affected. Map it on the step that runs npx playwright test; $(SQUALLY_INGEST_KEY) on the command line would also work, but it puts the key into the command, which Microsoft advises against.

Already have an env: block on that step? Add the SQUALLY_INGEST_KEY line to it - a second env: key on the same step is not valid YAML.

SQUALLY_INGEST_URL is not needed: the reporter sends to https://app.squally.dev by default.

Parallel jobs

The jobs of a strategy: parallel share Build.BuildId, and the reporter uses it as the run ID: the shards become one run in Squally, each shard numbered by Playwright’s --shard. See Sharded runs.

Every other job of the same pipeline run shares Build.BuildId too - a browser matrix, an api and a ui job, a job in another stage. Give each job that is not a shard of another its own SQUALLY_CI_JOB_KEY (in the step’s env:, for example SQUALLY_CI_JOB_KEY: ui): the run ID becomes <Build.BuildId>-<key>, and each job is a run of its own. See Sharded runs.

Branch, commit and pull requests

Build	Branch	Commit
Branch	Build.SourceBranch, without refs/heads/	Build.SourceVersion
Pull request	System.PullRequest.SourceBranch, without refs/heads/	System.PullRequest.SourceCommitId - the pull request’s head commit; where Azure does not set it, Build.SourceVersion
Tag	none	Build.SourceVersion

On a pull request Azure checks out the merge commit (refs/pull/N/merge) as Build.SourceVersion. Azure sets System.PullRequest.SourceCommitId only for a pull request build that a branch policy triggers; without it the run is reported under the merge commit. The commit message and author come from git for the reported commit; in a shallow checkout the reporter fetches that one commit, as on GitHub Actions.

The repository comes from Build.Repository.Uri - an Azure Repos repository or a GitHub repository - and with the commit it makes the file links on a test’s page work. With SAME_REPO_AS_APP=true, the tested revision is Build.SourceVersion: on a pull request, the merge commit.

No link to the build

The reporter sends no CI link for Azure Pipelines, so a run from Azure has no View in CI. Its Metadata tab shows the provider and the build ID.

Re-running failed jobs

Rerun failed jobs keeps Build.BuildId - only System.JobAttempt counts up - so the re-run reports under the first attempt’s run ID. Squally refuses it at its start: ci_run_reused once the first attempt’s run has ended, ci_run_collision while it still runs. The reporter says so once in the job log and sends nothing more; the job’s exit code is not affected, and the first attempt’s results stay as they were. To get a re-run into Squally, queue a new run of the pipeline: it gets a new Build.BuildId. See Troubleshooting.

Checks and pull-request comments

Checks and pull-request comments go to GitHub repositories only, through the Squally GitHub App. A repository in Azure Repos gets no status and no pull-request comment from Squally.

When your pipeline builds a GitHub repository and that repository is linked to the project, a check is posted on the reported commit as for a run from GitHub Actions - on a pull request only when the run is reported under its head commit, since the merge commit belongs to no pull request.

Cancelled builds

Squally shows a run as Cancelled when Playwright itself is interrupted - by SIGINT, as with Ctrl+C - and can still report the end. Whether Azure’s cancel reaches Playwright that way has not been verified. If Playwright is stopped without reporting, the run stays Running until Squally’s hourly check marks it Incomplete. See Cancelled runs.

In practice a cancelled pipeline shows as Incomplete after 60 minutes, the default run timeout. There is no cancel step for Azure Pipelines as there is for GitHub Actions.