Changelog

AllAppAIShardsReporterMCPAPI

1. 8 Nov 2026

   App

   A new Overview, Analytics, and error titles in your own words

   NewChanged

   Three releases, on November 6 and 8.

   Overview

   • A sentence above the tiles sums up the period: how many runs passed, how many tests are failing in a row, and whether one error is behind most failures.
   • Failing in a row, right under the tiles: the tests that failed in each of their latest runs - at least three - with the run where it began. It only appears when a test is failing in a row. See Failing in a row.
   • Most failing and Most flaky - each test once across Playwright projects, by its share of runs - with Top errors and Where the time goes below them.
   • Runs shows runs per day by result: failed, flaky, passed, cancelled or incomplete.
   • Branches is a short list: each branch with its runs, median duration, pass rate and last run. A row opens the runs on that branch.
   • The Last run line is gone - the sidebar’s Recent runs show it. Slowest tests moved to Analytics.

   Analytics

   • A new page in the sidebar: how a project’s tests develop over 7 to 90 days, by day or by week, against the period before, narrowed by Playwright project and tag. See Analytics.
   • One page in three groups, with links to jump between them: Reliability - pass rate over time, and tests failing more often, flaky more often, and better than before. Speed - run duration over time, the slowest tests and the tests getting slower. Where - one table by spec file with failures and test time, sortable by every column, then by error category, by tag and by project.
   • The tabs of the first version (Reliability, Performance, Breakdown) are gone, and so is the Failures by kind chart: By error category shows the failures per category, against the period before, each linked to the Error explorer.
   • New tests stay out of Failing / Flaky more often, where they hid the tests that really got worse: a line under the list counts them and links to Most failing / Most flaky on the Overview.
   • Better than before says what a test was - “was failed”, “was flaky” - and shows its share now as “↓ 8.4%”. Test time in the tables reads “34min 3s · 24%”, without a bar.

   Errors

   • Your own assertion message is the title. When a test writes one - expect(locator, "Checkout button never appeared") - that is the error’s title in the Error explorer, on the error’s page, in Top errors, on the test’s page and in the run; the kind of failure moves to the line under it. Without one, the title stays the kind and what it was about, such as “Timeout · test limit 30s”. See The title.
   • One title everywhere, and never with a leading “Error:”.
   • The read API’s title field is unchanged.
2. 6 Nov 2026

   API
   1.0.0-beta.18additive

   A 429 now says how long to wait and has the error body every other answer has

   New

   A 429 now says how long to wait and has the error body every other answer has.

   Added

   • Every 429 carries Retry-After: 60, the seconds to wait before retrying.
   • Every 429 has the usual error body: {"error": "Too many requests with this read key. Wait 60 seconds, then retry.", "code": "rate_limited"}.

   Changed

   • The limit is unchanged, 60 requests per 60 seconds per read key; it is now counted per key rather than per Authorization header.
   • Rate limit says a caller far beyond the limit can still be stopped by the edge firewall with a bare 429, without body or Retry-After: wait 60 seconds just the same.
   • x-error-codes lists rate_limited with hasBody: true.
3. 6 Nov 2026

   API
   1.0.0-beta.18document only

   Nothing the API sends or accepts changed

   Improved

   Nothing the API sends or accepts changed; the introduction of v1.json says more about using it.

   Changed

   • Authentication names the base URL, https://app.squally.dev/api/v1, and shows a first request: curl with Authorization: Bearer $SQUALLY_API_KEY against /projects.
   • Expiry warning says what follows the warning: 401 key_expired once the key has expired, no extension, and how to rotate without a gap.
   • Rate limit says the 429 has no JSON body and no Retry-After header.
   • New sections: Counts (what a run’s total, passed, failed, flaky and skipped count), Run numbers (runNumber) and Paging (which lists page, and that /projects and /branches do not).
4. 5 Nov 2026

   API
   1.0.0-beta.18document only

   Nothing the API sends or accepts changed

   Improved

   Nothing the API sends or accepts changed; one description in v1.json now names the default source as it is.

   Changed

   • testMetrics.tests on the overview said “CI runs by default”. It counts the runs of the source parameter’s kind, and that default is all since 1.0.0-beta.18 (ci before it). The number has always followed the parameter; only the description lagged behind.
5. 4 Nov 2026

   API
   1.0.0-beta.18document only

   Nothing the API sends or accepts changed

   Improved

   Nothing the API sends or accepts changed; one description in v1.json now names the check as it is posted.

   Changed

   • jobKey on a run: the job’s GitHub check is “ · ”, the check name of the linked repository - “Squally” for the first project that links a repository - where it said “Squally · ”.
6. 1 Nov 2026

   Reporter
   0.19.2

   Released from GitHub Actions

   Changed

   • Released from GitHub Actions. From this version on, each release is built and tested from its tag in the reporter’s CI and reaches npm through npm’s trusted publishing - no long-lived npm token - and only after a maintainer approves it. Until 0.19.0 releases were published by hand.
   • Nothing else changed: the reporter sends the same as 0.19.0.
   • 0.19.1 was tagged but never published, so there is no 0.19.1 on npm.
7. 1 Nov 2026

   Reporter
   0.19.0

   Annotations, the test's line, worker and start time

   New

   • Annotations. Every attempt sends its annotations - test.info().annotations, the { annotation } option, and the ones test.skip(), test.fixme(), test.slow() and test.fail() add - with type, description and where it was added. Descriptions are masked like error texts and cut at 1,000 characters; at most 20 per attempt, and Squally is told how many were left out.
   • The test’s line and column, for Open in repository.
   • Where in the repository the run started (e2e/ in a monorepo), once per run, so that link points at the right file. Off with SQUALLY_SEND_REPOSITORY=false, like the repository. See What the reporter sends.
   • Each attempt’s worker, its slot and its start time, stored for a later timeline of the run.
   • Nothing to configure. The exit code of your test run is never changed.
8. 1 Nov 2026

   MCP
   0.8.4

   Annotations, the test's line, worker and start time

   New

   • squally-debug-failure: every attempt carries the test’s annotations - an issue link or a skip or fixme reason is often the quickest answer to “why” - its line, the worker that ran it and when it started. Its description says so.
   • squally-get-run: every test row carries its final attempt’s annotation types and the test’s line.
   • Needs the read API as served since 01.11.2026; app.squally.dev serves it.
9. 1 Nov 2026

   App

   Annotations, the test's line and Open in repository

   New

   • Annotations - issue, slow, fixme, skip and your own - show where you look for a test: up to two per row on the run’s Tests tab, then “+N”; all of them with their descriptions on the test’s page in the run, a passed retry included; and the latest run’s on the Test detail. An issue whose description is a link opens it.
   • The test’s line beside its file, and Open in repository, which opens that line at the run’s commit on GitHub, GitLab, Bitbucket Cloud or Azure DevOps - in a monorepo too.
   • The read API serves them as well: each attempt’s annotations, the test’s line, and the worker and start time Squally now stores for a later timeline.
   • Needs squally-reporter 0.19.0 or newer; earlier runs show none of it.
10. 1 Nov 2026

    API
    1.0.0-beta.18text only

    One line of the Copy-for-AI prompt changed

    Improved

    One line of the Copy-for-AI prompt changed; no path, parameter, field, type, code or value changed.

    Changed

    • copyPrompt on an attempt of GET /projects/{projectId}/runs/{runId}/tests/{testName}: the line that named the attempt’s Playwright project reads “Project: mobile (webkit)” where it read “Browser: mobile”. The browser in brackets is the one the run’s Playwright config names for that project, left out when the config names none or the project is named like it (“Project: chromium”). browserName on the attempt is unchanged.
11. 1 Nov 2026

    API
    1.0.0-beta.18additive

    A test’s annotations, its line, and each attempt’s worker and start time

    New

    A test’s annotations, its line, and each attempt’s worker and start time; nothing was removed and no existing value changed.

    Added

    • On every attempt of GET /projects/{projectId}/runs/{runId}/tests/{testName}: annotations (Playwright’s test.info().annotations, at most 20, each with type, description - secrets masked like error text - and location), annotationsOmitted, location ({ line, column } of the test’s declaration), workerIndex, parallelIndex and startedAt.
    • On every test of GET /projects/{projectId}/runs/{runId}: annotations, the final attempt’s annotation types only, and location.
    • Squally stores these from squally-reporter 0.19.0 on. A result from an older reporter has no annotations, and null for the other fields.
12. 31 Oct 2026

    Reporter
    0.18.0

    Artifacts during the run, the run's link at the end, proxy support

    NewImproved

    • Artifacts are uploaded while your suite runs. A screenshot, trace or browser console goes up as soon as Squally has the result it belongs to, instead of after the last test. A job that is cancelled or killed partway keeps the artifacts of every result that already arrived.
    • Artifact uploads are retried like results since 0.17.0 - a network error, an HTTP 5xx or 429 - and the end of a run waits at most 3 minutes for the last ones. What did not arrive is one line at the end.
    • More than 250 artifacts per shard are no longer cut off.
    • A last line with the run’s link when every result arrived: Reported to Squally: run #128 · 412 results, 9 artifacts · https://....
    • SQUALLY_OUTPUT_FILE writes the run’s link and counts as JSON for a later CI step. See The end of a run.
    • HTTPS_PROXY, HTTP_PROXY and NO_PROXY are honoured for every request, npx squally-reporter cancel included. A runner that reaches the internet only through a proxy sent nothing until now. See Behind a proxy.
    • A failed quarantine rules request is no longer a warning in your CI log.
    • The exit code of your test run is never changed.
13. 30 Oct 2026

    Reporter
    0.17.0

    Short outages are waited out, one line about what did not arrive

    ChangedFixed

    • Short outages are waited out instead of costing results. A network error, an HTTP 5xx, HTTP 429 and an HTTP 403 from Vercel’s firewall in front of Squally (x-vercel-mitigated: challenge or deny) are retried for up to 90 seconds per request. The start of the run gets the same 90 seconds while your tests keep running. Until 0.16.0 a firewall challenge on the start of a run lost the whole run.
    • The end of a run waits longer only after such a failure: up to 60 seconds for the last results and for marking the run complete, instead of 5.
    • One line at the end about what did not arrive, instead of one line per dropped batch - and, if the run could not be marked complete, that it will show as incomplete once the project’s run timeout has passed.
    • A firewall challenge is no longer reported as a login or proxy problem, also in the quarantine rules warning, the artifact upload warning and npx squally-reporter cancel.
    • The exit code of your test run is never changed. See When Squally cannot be reached for a while.
14. 30 Oct 2026

    MCP
    0.8.3

    A firewall challenge is not reported as a key problem

    Fixed

    • When Vercel’s firewall in front of Squally stops a request itself - HTTP 403 with x-vercel-mitigated: challenge or deny - the tool result said only “The Squally API answered HTTP 403.”, which reads like a refused key. It now says that the key was not checked, and to wait a few minutes and retry once, not in a loop. The API’s own 403s keep their sentence and action. No tool or schema changed.
15. 30 Oct 2026

    MCP
    0.8.2

    Run counts for errors, corrected tool texts

    NewFixed

    • squally-list-errors returns failedRunCount and flakyRunCount for each error: the runs in which a test the error hit failed, and the runs in which none failed and one passed only after a retry. 0.8.2 needs a Squally that sends them; app.squally.dev does.
    • The description of squally-list-errors said failedCount and flakyCount say how the runs ended. They count (run, test) pairs - a run in which the error failed three tests counts three times. It now points to the run counts.
    • squally-get-run and the server instructions now say that each test row is the test’s final attempt: a test that passed only after a retry reads as passed with attempts above 1, and the errors of its failed attempts come from squally-debug-failure. See MCP server.
16. 30 Oct 2026

    API
    1.0.0-beta.18additive

    Two fields on an error signature

    New

    Two fields on an error signature; nothing was removed and no existing value changed.

    Added

    • failedRunCount and flakyRunCount on every signature of GET /projects/{projectId}/errors and on signature of GET /projects/{projectId}/errors/{fingerprint}: how the error’s RUNS ended. A run is failed when a test the error hit ended failed in it, and flaky when none did and one passed after a retry; each run counts once. Their sum is at most runCount. failedCount and flakyCount are unchanged: they count (run, test) pairs, so a run in which the error hit two tests counts twice there.
17. 29 Oct 2026

    Reporter
    0.16.0

    The run's Playwright config and repository

    New

    • The run’s Playwright config, for the run’s Metadata tab: workers, fullyParallel, forbidOnly, globalTimeout, maxFailures, reporter names and, per project, its name, browser, channel, viewport, trace/video/screenshot mode, retries and timeout. Read field by field - headers, credentials, storageState, proxy, config.metadata, the command line and file paths are never sent.
    • The project’s base URL, only if you ask for it: SQUALLY_SEND_BASE_URL=true sends each project’s use.baseURL - scheme, host and path only.
    • The repository as host and path (github.com/acme/shop), from your CI’s variables or the git remote, with credentials removed - Squally shows it when the project links no GitHub repository. SQUALLY_SEND_REPOSITORY=false turns it off. See What the reporter sends.
18. 29 Oct 2026

    Reporter
    0.15.0

    Where to watch the run, as soon as it starts

    New

    • Once Squally has created the run, the reporter prints one line with its number and link: [squally-reporter] Reporting to Squally: run #128 · https://app.squally.dev/o/…/runs/…, under --shard run #128, shard 2 of 3 · …. Every shard of a CI run shows the same number.
    • The line goes to stderr, so --reporter=json without an output file still writes clean JSON to stdout.
19. 27 Oct 2026

    API
    1.0.0-beta.18text only

    One error message and one field description changed

    Improved

    One error message and one field description changed; no path, parameter, field, type, code or value changed.

    Changed

    • invalid_period for a period beyond the plan’s retention names the periods the plan can ask for, “Use one of 7, 14, 30.” on a 60-day plan. It said “Use 60 or less.”, and 60 is not a period the API accepts.
    • copyPrompt on an attempt: the description names where the same text is copied today, Copy prompt under Fix in your agent, Without MCP, on the attempt page. The text itself is unchanged.
20. 27 Oct 2026

    API
    1.0.0-beta.18behaviour change

    The project’s run timeout counts from a run’s last activity

    Changed

    No field was added or removed; when a run counts as overdue changed.

    Changed

    • The project’s run timeout counts from a run’s last activity - its begin, its latest batch of results or its completion - instead of from its start. A long run that keeps sending results stays running and settled: false; a run that goes silent becomes incomplete one timeout after its last activity. The descriptions of status and settled say so. Runs that began before this change count from their start as before.
    • The timeout itself (30 to 1440 minutes, 60 by default) can now be set in Project settings → General; until now it was always 60.
21. 27 Oct 2026

    API
    1.0.0-beta.18document only

    Nothing the API sends or accepts changed

    Improved

    Nothing the API sends or accepts changed; v1.json now says what a cancel has meant since 2026-10-22.

    Changed

    • A cancel can mark a whole sharded run, not only the shards that are running: cancelled from the dashboard or with npx squally-reporter cancel while shards it expects have not begun, the run is cancelled at once and a shard that begins later is refused. Its cancellation is set as for any cancel - source is user or cli, at is when the run was marked. The descriptions of status and cancellation.at say so; the behaviour itself shipped on 2026-10-22.
22. 25 Oct 2026

    Reporter
    0.14.1

    Terminal output beside html, clearer connection errors

    FixedChanged

    • Playwright’s terminal output is back beside the html reporter (reporter: [["html"], ["squally-reporter"]]): progress, “2 passed” and the list of failures show again.
    • One clear line when something other than Squally answers - a login or proxy in front of the URL: “did not answer as Squally (HTTP 401, not Squally’s JSON). A login or proxy in front of it? Check SQUALLY_INGEST_URL.” Squally’s own 401 says the ingest key was refused. The same for npx squally-reporter cancel.
    • Redirects are not followed. A 3xx is reported with where it pointed. If your SQUALLY_INGEST_URL starts with http:// and only reached Squally through a redirect, change it to https:// - or leave it unset.
    • The quarantine rules warning waits for the run to start, and an error line names the HTTP status once.
23. 24 Oct 2026

    MCP
    0.8.1

    Security updates of dependencies

    Fixed

    • @modelcontextprotocol/sdk 1.32.2 and zod 4.6.7: zod before 4.6.6 could exhaust memory on crafted input, and the server validates every tool call’s arguments with zod. No tool, input or behaviour changed.
    • A call against a deleted organization is explained (organization_deleted) instead of reported as an unknown code.
24. 24 Oct 2026

    API
    1.0.0-beta.18document only

    Nothing the API sends or accepts changed

    Improved

    Nothing the API sends or accepts changed; only how v1.json writes a value that may be null.

    Changed

    • A field that is a string or null (and likewise for numbers and other types) reads "type": ["string", "null"] instead of an anyOf of the two types. Both mean the same in OpenAPI 3.1; a client generated from the document accepts the same responses. The schema library that writes the document changed its output in a security update.
25. 23 Oct 2026

    MCP
    0.8.0

    CI and local runs by default

    Changed

    • Follows read API 1.0.0-beta.18, which counts CI and local runs when source is left out. squally-get-test-metrics, squally-list-tests and squally-list-errors follow it; send source=ci for CI runs only. A call that sends source gets what it got before. 0.8.0 works against beta.17 as well. See MCP server.
26. 23 Oct 2026

    API
    1.0.0-beta.18additive

    One error code added

    New

    One error code added; nothing removed or renamed.

    Added

    • organization_deleted (403) on every endpoint: the key’s organization was deleted and can still be restored. The sentence names the date until which Squally support can restore it; the same key works again after a restore. It is checked after the key and before tier_too_low.
27. 23 Oct 2026

    API
    1.0.0-beta.18

    Every endpoint with a source parameter counts all runs, CI and local, when source is left out

    Changed

    Every endpoint with a source parameter counts all runs, CI and local, when source is left out; it counted CI runs only. GET /runs already listed all runs by default and is unchanged. A request that sends source gets what it got before.

    Changed

    • source on GET /projects/{projectId}/overview, /tests, /tests/{testName}, /tests/{testName}/metrics, /errors and /errors/{fingerprint}: default all, was ci. The allowed values are unchanged. To keep the old figures, send source=ci.
    • population in those responses names the source used, so a request without source now reads an all_ value where it read a ci_ one.
    • copyPrompt on GET /projects/{projectId}/runs/{runId}/tests/{testName} is unchanged: its evidence line still counts CI runs only.
28. 22 Oct 2026

    Reporter
    0.14.0

    A cancel step for CI

    New

    • npx squally-reporter cancel marks the CI run as cancelled in Squally. Put it in a step with if: ${{ cancelled() }} after the tests: on GitHub Actions a cancelled workflow often never reaches Playwright, and without the step the run showed as incomplete after the run timeout. It finds the run by the same CI run ID the reporter sent, prints one line and always exits 0. See Reporter CLI and GitHub Actions.
    • Set SQUALLY_CI_JOB_KEY at job level, and use fail-fast: false on sharded jobs - with fail-fast one failing shard cancels the others and the whole run is marked cancelled.
29. 22 Oct 2026

    Reporter
    0.13.1

    Runs on Node.js 20.18.1 and later

    ChangedFixed

    • Node.js 20.18.1 or newer is enough (was 22.19.0), so CI images with Node 20 or an older Node 22 run the reporter.
    • Fixed: with 0.13.0 the reporter crashed while loading on Node below 22.19 (createZstdDecompress missing), and Playwright ended a green run with exit code 1.
30. 21 Oct 2026

    MCP
    0.7.0

    No stable branch

    Breaking

    • Breaking: follows read API 1.0.0-beta.17, which drops the project’s stable branch. stableBranch is gone from squally-list-projects and squally-find-run. 0.6.1 rejects beta.17’s responses in a client that validates structured output; 0.7.0 works against beta.16 and beta.17.
31. 21 Oct 2026

    App

    No stable branch - every branch by default, and the branch you pick is remembered

    Changed

    • The Overview counts every branch unless you pick one. The branch switch lists every branch that ran in the period, and Squally remembers the branch you pick for each project in your browser - the next visit opens on it again. See Overview.
    • The stable branch setting is gone: a new project needs only a name, the Getting started list has three steps, and the project cards, the Branches section and the GitHub page no longer name one.
    • Run comparison picks the previous run on the run’s own branch. See Run comparison.
    • Email reports count all branches or one branch by name. Reports that followed the stable branch now name that branch.
32. 21 Oct 2026

    API
    1.0.0-beta.17

    The project’s stable branch setting is gone

    Breaking

    The project’s stable branch setting is gone: no field names it any more, and GET /projects/{projectId}/overview counts every branch unless branch names one.

    Changed

    • branch on GET /projects/{projectId}/overview: an exact branch name, as on /runs and /tests. Absent, or all, means every branch - the default, which was the stable branch. branch=stable is a 400 invalid_parameter that says the value was removed.
    • GET /projects/{projectId}/overview: branch - the branch every figure counts, or null for every branch - replaces branchScope and stableBranch.
    • lastRun on GET /projects/{projectId}/overview: the newest run on the branch, or null when it has none. It no longer falls back to another branch.

    Removed

    • stableBranch from GET /projects, GET /projects/{projectId}/runs and GET /projects/{projectId}/settings.
    • branchScope and stableBranch from GET /projects/{projectId}/overview, and isFallbackBranch from its lastRun.
33. 20 Oct 2026

    Reporter
    0.13.0

    Progress of a running run

    New

    • How far a running run got. At the start of each run (each shard under --shard) the reporter tells Squally how many tests it is about to run. While the run is going, the run page and the runs list show “38/47” and the tests still to run. Only shards that have started count. See Live results.
    • Tests of a setup (dependency) project run in every shard and count in each shard’s plan, while Squally counts them once - the bar can keep a small remainder until the run ends.
34. 20 Oct 2026

    API
    1.0.0-beta.16behaviour change

    A run that waits for a shard is running until it is overdue

    Changed

    A run that waits for a shard is running until it is overdue; no path, parameter, field or type changed.

    Changed

    • A run’s status, on GET /projects/{projectId}/runs and on GET /projects/{projectId}/runs/{runId}: running now also covers a run whose started shards have all finished while it still waits for a shard that has not started yet - one a CI queue holds back, say. It stays running until it is overdue by the project’s run timeout, counted from its latest shard’s start; only then is it incomplete. Before, such a run was incomplete at once and went back to running when the shard started.
    • settled stays false while the run waits for a missing shard.
35. 19 Oct 2026

    Reporter
    0.12.0

    Steps, a key alone is enough

    NewChangedFixed

    • Steps. For an attempt that failed or needed a retry, the reporter sends the steps Playwright recorded - hooks, fixtures, actions, assertions, test.steps - with duration and file:line; the attempt page shows them with the failing step marked. Typed values never leave the runner (Fill "•••"), at most 300 steps and 200 characters per title. See Test in a run.
    • The ingest URL defaults to https://app.squally.dev. SQUALLY_INGEST_KEY alone is enough; set SQUALLY_INGEST_URL only for another host. See Keys.
    • Pacing: a batch every 2 seconds, or 0.4 seconds × the number of shards when that is longer, at most 30 seconds - up to five shards send as often as an unsharded run. Batches are also bounded by size (3 MB).
    • The author is the commit’s, not whoever started the GitHub Actions workflow; GITHUB_ACTOR only when git cannot read the checkout.
    • A read key is refused from every source - SQUALLY_INGEST_KEY and the apiKey option too - before anything is sent.
    • A run cancelled in Squally says “This run was cancelled in Squally” instead of advice about job keys. Skipped artifacts name their real limit (4 MB for a browser console, 50 MB for screenshots and traces, 250 per shard).
36. 17 Oct 2026

    API
    1.0.0-beta.16text only

    One field description changed

    Improved

    One field description changed; no path, parameter, field, type or value changed.

    Changed

    • copyPrompt on an attempt: the description names the button that now copies this text, Copy the full context in the Debug with AI panel of the attempt page. The text itself is unchanged.
37. 10 Oct 2026

    API
    1.0.0-beta.16

    GET /projects/{projectId}/errors is paged in the database, and signatures with equal counts are ordered by their fingerprint

    Changed

    GET /projects/{projectId}/errors is paged in the database, and signatures with equal counts are ordered by their fingerprint. No path, parameter, field or type changed.

    Changed

    • GET /projects/{projectId}/errors: the order is occurrences, then runs, then tests, all descending, then fingerprint ascending. Signatures with equal counts were ordered by their message before; they now follow the fingerprint, so many of them move - in a project where most errors happened once, most positions change. The order is total: the same data gives the same pages in every request.
    • The page is cut in the database: only the requested page’s signatures are built and returned. Every count still covers the whole period, and page.total and page.pageCount are unchanged. A page past the last still returns the last page.
38. 6 Oct 2026

    Reporter
    0.11.0

    One run per job, a link to the CI run

    NewImproved

    • SQUALLY_CI_JOB_KEY: jobs of one CI run that are not shards of each other - a browser matrix, an api and a ui suite - each report as a run of their own. The key goes into the run ID, <run_id>-<key>-<attempt> on GitHub Actions; the shards of one job share it. It replaces the hand-built SQUALLY_CI_RUN_ID for matrix jobs. See Reporter setup.
    • Re-runs of a keyed job on GitHub Actions link to that job’s own earlier attempt.
    • A link to the CI run on GitHub Actions (the attempt’s page), GitLab CI, Jenkins, Buildkite, Travis CI and Drone - View in CI on the run page. See The link to the CI run.
    • A refused job says so at its start. When Squally refuses a job that would share a run with another one (ci_run_collision, ci_run_mixed), the reporter prints the reason and the fix and sends nothing more for the job; the exit code is untouched. See Troubleshooting.
    • The “already completed” warning suggests SQUALLY_CI_JOB_KEY.
39. 6 Oct 2026

    MCP
    0.6.0

    Job key, attempt, re-run link, CI link, settled

    BreakingNew

    • Breaking: needs read API 1.0.0-beta.15. 0.5.0 keeps working against beta.15 - its output schemas allow the new fields.
    • Five new fields on every run of squally-find-run and on squally-get-run’s run: settled (status and counts are final), jobKey (the reporter’s SQUALLY_CI_JOB_KEY - each job with a key is its own run), attempt, rerunOf and ciRunUrl (the run page’s View in CI link). See MCP server.
    • The key’s location in the README and in the server’s messages is Organization › Read keys, and the README says the server reads CI and local runs.
40. 6 Oct 2026

    App

    Every link names its organization and project

    Improved

    • App addresses carry the organization and the project: /o/{organizationId}/p/{projectId}/runs, /tests, /errors, /settings, and /o/{organizationId} for the organization’s own pages. A link to a run, a test or an error opens that project, whichever one you had open, and two tabs can show two projects. See Project settings.
    • Links Squally sends - the GitHub check, the pull-request comment, Slack and the email reports - use the new addresses. Links sent before keep working: every old /dashboard/... address leads to the same page.
41. 6 Oct 2026

    App

    Matrix jobs - one run, one check and one comment block per job

    NewImprovedFixed

    • One run per job. With squally-reporter 0.11.0’s SQUALLY_CI_JOB_KEY, each job of a CI run - a browser matrix, an api and a ui suite - is a run of its own, with its key as a chip in the runs list, on the run page, in the run report and on the comparison’s cards. See Matrix jobs.
    • One GitHub check per job, “Squally · firefox” beside “Squally · chromium”, so the job that finishes last no longer hides the other’s result. A run without a key keeps “Squally”. Branch protection that requires “Squally” has to require each job’s check instead. See GitHub.
    • One pull-request comment, one block per job. The comment is rebuilt from the latest run of each job on the pull request’s head commit - a re-run replaces its job’s block, a running job reads “running” - and two jobs that finish together no longer overwrite each other. A single job without a key gets the same comment as before. See GitHub.
    • View in CI on the run page and in the run report opens the CI run - on GitHub Actions the page of exactly its attempt. See Run summary.
    • Check posted opens the GitHub check itself, not the commit’s list of checks.
    • Two jobs under one run ID are refused at the second one’s start, with squally-reporter 0.11.0: ci_run_collision for two jobs on the same slot, ci_run_mixed for an unsharded job beside a sharded one or shards of different totals. They used to be merged into one run that lost the slower job’s results. See Troubleshooting.
    • A run’s status is written once, by the first job that completes it: a second job that reported under the same run ID can no longer turn a failed run into a passed one.
    • Read API 1.0.0-beta.15: runs say settled, jobKey, attempt, rerunOf and ciRunUrl, and the overview’s testsWithRetries and ciTests are now flakyTests and tests. See the API changelog.
42. 6 Oct 2026

    API
    1.0.0-beta.15

    Runs say which job and which attempt of their CI run they are, which run they re-ran, where the CI run is and whether they are settled

    Breaking

    Runs say which job and which attempt of their CI run they are, which run they re-ran, where the CI run is and whether they are settled; two /overview fields get the names the dashboard uses.

    Added

    • settled on every run of GET /runs and on GET /runs/{runId} run: true when the run’s status and counts are final and it counts in the dashboard’s figures. An incomplete run that still waits for a shard within the run time limit is not settled; the hourly check settles it.
    • jobKey on the same runs: the job key the reporter sent (SQUALLY_CI_JOB_KEY, squally-reporter 0.11.0), or null. Each job with a key is its own run.
    • attempt on the same runs: which attempt of its CI run this is, 2 after GitHub’s Re-run failed jobs; null when it is not known.
    • rerunOf on the same runs: { id, runNumber } of the earlier attempt this run re-ran, or null.
    • ciRunUrl on the same runs: the CI run’s page, the run page’s View in CI link - the URL squally-reporter 0.11.0 sends, else for GitHub Actions one built from the project’s one linked repository; null otherwise.

    Changed

    • GET /overview testMetrics.testsWithRetries is now flakyTests, and testMetrics.ciTests is now tests. Values unchanged; the old names are gone.
    • shards.expected on GET /runs and GET /runs/{runId}: described as what it has been since block 1c - the shards the run waits for, which for a re-run are the shards it re-ran. No value changed.
43. 4 Oct 2026

    MCP
    0.5.0

    One run status, local runs listed

    BreakingNew

    • Breaking: needs read API 1.0.0-beta.14, and 0.4.0 does not work with it. squally-find-run and squally-get-run no longer have cancelled; a cancelled run’s status is cancelled.
    • status on squally-find-run and squally-get-run is the run page’s word - passed, failed, cancelled, incomplete or running - and the status filter of squally-find-run takes all five.
    • source on squally-find-run: all by default, ci or local; every run says local. See Local runs.
    • Counts count every test once across a run’s shards; the tests a cancel cut off count nowhere. See MCP server.
44. 4 Oct 2026

    App

    Runs, the run page and the comparison - one status, local runs marked, every test once

    NewImproved

    • The runs list filters by every status - All, Passed, Failed, Cancelled, Incomplete and Running - and by where a run ran: CI · Local · All, All by default here, because the list counts nothing and a run you just started on your laptop should be there. Local runs are marked Local. A run is placed in the period and ordered by its first shard’s start, and every filter applies before a page is cut. See Run list.
    • One count per run. The runs list, the run page, the comparison’s cards and the read API count every test once, however many shards reported it - its final attempt across all of them decides - and the tests a cancel cut off count nowhere.
    • The run page marks a local run Local, and says “Partial” only while its counts can still change - not on a run that ended incomplete and has settled. See Run summary.
    • The Errors tab counts “X failed · Y flaky” by each test’s final attempt across the whole run, as the error pages do: a test skipped after a retry is neither, and one shard selected still shows the run’s counts. See Errors tab. The run report and the AI analysis count the same way.
    • The comparison preselects a partner of the same kind - CI with CI, local with local - from the last 90 days, and its picker has CI · Local · All. Every run’s status there is the run’s own, incomplete runs included. See Run comparison.
    • Read API 1.0.0-beta.14: a run’s status is the same word as in the app - cancelled included - every run says local, and List runs takes source. See the API changelog.
45. 4 Oct 2026

    App

    Flaky tests by name, and the comparison counts like the runs list

    ImprovedFixed

    • “Flaky tests” is the dashboard tile that was called “Tests with retries”: the tests with at least one flaky run. A test’s page says “most flaky runs on main”, and the pull-request comment setting says a comment comes for failed or flaky tests. “Retried” keeps one meaning: a test that ran more than once. See Dashboard.
    • The comparison’s cards count like the runs list - the tests a cancel cut off count nowhere - and say “Partial” only while a run’s counts can still change. See Run comparison.
46. 4 Oct 2026

    App

    GitHub, Slack and the weekly report - CI runs only, one count, one way to say it

    ImprovedFixed

    • GitHub, Slack and the run page say the same numbers. The check’s title, the pull-request comment’s heading and the Slack message read “Run #128 · 64 passed · 2 flaky · 1 failed · 0 skipped” - the run’s number and all four counts, zeros included, each test counted once by its final attempt across all shards. The check names the flaky tests instead of “N retried”, the comment no longer counts a flaky test twice, and test names carry their Playwright project. See GitHub and Slack.
    • Local runs post nothing. No check, no pull-request comment and no Slack message for a run no shard of which reported a CI provider; its run page says “Not posted · local run”. See Local runs.
    • A run that did not complete gets one check, when it settles - for a timed-out shard, a shard that never started and the monthly result limit alike, with the results its shards reported. Its conclusion is timed out, or neutral for the monthly limit, and Slack gets one message with an “Incomplete” line. A cancelled run’s check counts only the tests that finished. See Timeouts.
    • The weekly report counts what the app counts: finished CI runs by the same stored rule, cancelled and incomplete runs as runs but in no rate, unfinished runs only as a count (“1 cancelled · 2 incomplete · 1 unfinished”). Mass failures stay out of the test lists, errors say “X failed · Y flaky” per run, “New this week” is first seen this week, and slow tests rank by the median. See Email reports.
    • The run report says “Partial” only while the run’s counts can still change and marks a local run Local, like the run page. See Run report.
    • Delivery that hangs is given up: 5 seconds for a Slack webhook and for GitHub’s token, and a failed delivery is recorded with ids and a failure code only. See Hosting and data.
47. 4 Oct 2026

    API
    1.0.0-beta.14document only

    Four descriptions said something the API does not do

    Improved

    No path, parameter, field or type changed, and no response changed. Four descriptions said something the API does not do.

    Changed

    • GET /runs/{runId}/tests/{testName} attempts[].shard: the shard this attempt ran in. The description was the test row’s, which names the shard of the test’s final attempt - wrong for every other attempt.
    • GET /tests: counts finished runs of the source (it said completed; an overdue run with a dead shard counts too, since 1.0.0-beta.10), and its cost is aggregate queries over the period’s results of the source (it said two queries over the period’s CI results).
    • GET /overview: its cost is aggregate queries over the period’s finished runs, no result rows loaded (it said one pass over the period’s runs).
    • GET /overview testMetrics and testMetrics.testsWithRetries: the dashboard’s tile is called Flaky tests. The field keeps its name.
48. 4 Oct 2026

    API
    1.0.0-beta.14

    A run’s status is the dashboard’s own word

    Breaking

    A run’s status is the dashboard’s own word - the four end states plus running - and the cancelled flag is gone; runs say whether they ran locally, and the runs list filters by where they ran.

    Added

    • local on every run of GET /runs and on GET /runs/{runId} run: true when no shard of the run reported a CI provider.
    • source on GET /runs: all (the default), ci or local - which runs are listed by where they ran. The default lists every run, as before; it is all and not ci because the runs list counts nothing, it answers what ran. Anything else is 400 invalid_parameter.

    Changed

    • status on GET /runs and GET /runs/{runId}: passed, failed, cancelled, incomplete or running, the first that applies in that order - the dashboard’s status. A cancelled run is cancelled; it reported the outcome of what it ran before the stop. Never null.
    • ?status= on GET /runs takes all five values; a cancelled run matches cancelled only. It took passed and failed, and a cancelled run matched neither. Anything else is 400 invalid_parameter.
    • GET /runs/{runId} run.shards.received[].status: the shard in the same words - cancelled when it recorded a cancel, else running or incomplete by its state, else failed or passed. Never null; it was the shard’s own outcome, null until the shard finished.
    • GET /overview lastRun.status: passed, failed, cancelled or incomplete - a cancelled run is cancelled, as on GET /runs. Never null.

    Removed

    • cancelled on GET /runs runs and GET /runs/{runId} run: it repeated status === "cancelled". cancellation stays - where the cancel came from, when, and who - and is set exactly when status is cancelled.
49. 3 Oct 2026

    MCP
    0.4.0

    Local runs on request, errors counted per run

    BreakingNew

    • Breaking: needs read API 1.0.0-beta.9, and 0.3.0 does not work with it. squally-list-errors no longer has isNew; compare firstSeen with window.from instead.
    • source on squally-list-tests, squally-get-test-metrics and squally-list-errors: ci (the default, what they counted before), local or all. population names the runs counted. See Local runs.
    • squally-list-errors counts runs: runCount counts a sharded run once, over finished runs only. Each error gains failedCount and flakyCount - how the runs ended for the tests it hit - and massFailureRunCount; the answer gains unfinishedRuns.
50. 3 Oct 2026

    AppShards

    Runs that will not finish count once their timeout has passed

    Improved

    • The run timeout is checked every hour, no longer once a day.
    • A run whose shard died or never started now counts once the run timeout has passed since its last shard started: it ends incomplete, and the tests its shards finished count in the test metrics and the errors, like a cancelled run’s. Until now such a run counted nowhere. See Timeouts.
    • Incomplete runs stay out of the run rates, like cancelled ones: the dashboard’s Branches shares and median duration and its Runs chart. An incomplete Last run is shown in grey.
    • A shard that reports late still counts, and nothing about the run is sent to GitHub or Slack a second time.
51. 3 Oct 2026

    App

    A run stopped by the monthly result limit counts like a timed-out one

    Improved

    • A run stopped by the monthly result limit now counts once its run timeout has passed, like a run whose shard timed out: it stays incomplete, the results it stored before the limit count in the test metrics and the errors, like a cancelled run’s, and it stays out of the run rates. Until now such a run counted nowhere. See Timeouts.
52. 3 Oct 2026

    AppShards

    Local runs get their own numbers, errors count runs

    NewImproved

    • CI · Local · All on the dashboard, the Tests page, a test’s page and the errors pages. CI stays the default; Local shows the runs from your machine on their own, All both. Until now local runs were left out of every number but the errors, where they were mixed in with CI runs. See Local runs.
    • Errors count runs, each once however many shards it had, and only once all its shards have finished. Each error says how those runs ended for the tests it hit: “3 failed · 1 flaky”.
    • Mass failures are marked in the errors, and count there as one run.
    • The New filter is gone from the errors. Sort by First seen instead.
53. 3 Oct 2026

    API
    1.0.0-beta.13

    A cancelled run’s counts leave out the tests the cancel interrupted, and the runs list places and orders a run by its first shard’s start

    Changed

    A cancelled run’s counts leave out the tests the cancel interrupted, and the runs list places and orders a run by its first shard’s start. No field and no value changed.

    Changed

    • counts on GET /runs and GET /runs/{runId}: in a cancelled run the tests the cancel interrupted - squally-reporter reports them as skipped - count nowhere, as in every other figure. skipped and total of such a run drop by that many; GET /runs/{runId} tests still lists them.
    • GET /runs: a run is in the period by the start of its first shard and ordered by it, newest first. It was listed when any shard started in the period, ordered by its latest shard. The cursor is still a run id; pages no longer skip or repeat a run whose shards started apart, and a status filter never leaves a page short while more runs follow.
54. 3 Oct 2026

    API
    1.0.0-beta.12

    A run’s counts take every test once, however many shards reported it

    Changed

    A run’s counts take every test once, however many shards reported it. No field and no value changed.

    Changed

    • counts on GET /runs and GET /runs/{runId}: over the union of the run’s shards - every test once, its final attempt chosen across all of them (highest retry, then the lowest shard, then the first repetition). They summed each shard’s own counts, so a test reported from two shards counted twice. A run whose tests each ran in one shard is unchanged.
    • GET /runs/{runId} tests: one row per test, shard naming the shard its final attempt ran in and attempts counting every shard’s attempts. A test reported from two shards was one row per shard.
55. 3 Oct 2026

    API
    1.0.0-beta.11

    A run stopped by the monthly result limit now finishes like a run the run timeout ended

    Changed

    A run stopped by the monthly result limit now finishes like a run the run timeout ended: once it is overdue, it counts like a cancelled run. No field and no value changed.

    Changed

    • Every figure that counts finished runs - the test metrics, the error figures, /overview and a test’s history (counted): a shard stopped by the monthly result limit is treated like one that timed out. Once the run is overdue by the project’s run timeout, counted from its latest shard’s start, it finishes and counts with the results its shards reported. Until now such a run never finished.
    • unfinishedRuns on GET /overview and GET /errors counts a limit-stopped run only until it is overdue.
    • GET /overview lastRun.status can be incomplete for such a run too.
56. 3 Oct 2026

    API
    1.0.0-beta.10

    A run that will not finish on its own now finishes by the run timeout and counts like a cancelled run

    Changed

    A run that will not finish on its own now finishes by the run timeout and counts like a cancelled run. No field and no value changed.

    Changed

    • Every figure that counts finished runs - the test metrics, the error figures, /overview and a test’s history (counted): a run also finishes when it is overdue by the project’s run timeout, counted from its latest shard’s start, and every shard that started has completed or timed out. A shard that died or never started no longer keeps the run out of every number. Such a run counts with the results its shards reported, like a cancelled run. The population values are unchanged; their description says what finished means.
    • unfinishedRuns on GET /overview and GET /errors: only runs with a shard still running, or with one timed out or missing while the run is not yet overdue. An overdue one is counted instead.
    • GET /overview runsInWindow includes these incomplete runs; runsSeries leaves them out, as it leaves out cancelled runs.
    • GET /overview lastRun.status can be incomplete: the newest run may be one the run timeout finished.
57. 3 Oct 2026

    API
    1.0.0-beta.9

    The error endpoints count like everything else

    Breaking

    The error endpoints count like everything else: over finished runs, all shards of a run one run, of the source you ask for. isNew is removed, and each error gains how its runs ended.

    Added

    • source (ci, local, all; default ci) on GET /errors and GET /errors/{fingerprint}. Any other value answers 400 invalid_parameter.
    • population on both responses: ci_groups_finished (the default), local_groups_finished or all_groups_finished. Runs that were a mass failure as a whole count here, unlike in the per-test metrics.
    • On every error: failedCount and flakyCount - of the runs and tests the error hit, how many ended failed and how many passed after a retry, by the test’s final attempt across the run’s shards - and massFailureRunCount, the mass-failure runs among them.
    • GET /errors: unfinishedRuns, the period’s runs of the source that have not finished. They are in no figure.
    • GET /errors/{fingerprint}: runCount, failedCount and flakyCount on each of tests, and massFailure on each of runs.

    Changed

    • runCount: runs, each counted once however many shards it had. It counted every shard the error appeared in.
    • Every count: only finished runs of the source. Runs still running or timed out, and local runs unless source asks for them, no longer count.
    • firstSeen: the earliest such run still stored.
    • unfingerprintedCount: over the window only. It read the period before it too.
    • GET /errors/{fingerprint} runs: one row per run and test, runId the run’s id as /runs gives it, finalStatus the test’s final attempt across all the run’s shards. It was one row per shard, with the shard’s id and that shard’s final attempt.

    Removed

    • isNew on every error. Compare firstSeen with window.from instead.
58. 3 Oct 2026

    API
    1.0.0-beta.8

    Runs on a developer’s machine get numbers of their own

    Breaking

    Runs on a developer’s machine get numbers of their own: a new source parameter picks CI runs (the default), local runs or both, and population becomes an enum that names the one counted. Without source every response counts what it did, but a client that pins population to its one value fails on the two new ones.

    Added

    • source (ci, local, all; default ci) on GET /overview, GET /tests, GET /tests/{testName} and GET /tests/{testName}/metrics. A local run is one for which no shard reported a CI provider. Any other value answers 400 invalid_parameter.

    Changed

    • population on /tests, /tests/{testName}/metrics and /overview’s testMetrics: an enum of ci_groups_finished_not_mass_failure (the default, unchanged), local_groups_finished_not_mass_failure and all_groups_finished_not_mass_failure. It was that first value alone.
    • /overview: every figure, lastRun, unfinishedRuns and runsSeries included, counts the runs of source.
    • /tests/{testName}: occurrences[].counted follows source; every occurrence is still listed, and local still marks local runs.
59. 2 Oct 2026

    MCP
    0.3.0

    A sharded CI run counts once

    Breaking

    • Breaking: needs read API 1.0.0-beta.6, and 0.2.x does not work with it. In squally-list-tests and squally-get-test-metrics, a CI run counts once, however many shards it had, and only when all its shards have finished; population is "ci_groups_finished_not_mass_failure", was "ci_completed_not_excluded".
    • A test’s recent runs carry the run id squally-find-run gives - one per run, no longer one per shard.
60. 2 Oct 2026

    App

    Weekly email reports, run reports and new settings

    NewImproved

    • Weekly email reports of how a project’s tests did, on Standard and Pro.
    • A report for every run: filter it, add your notes, print it or copy it as Markdown. Notes stay in your browser.
    • Settings are split into your organization’s and each project’s, with a page that lists all projects.
    • Plan and usage in one place: what your organization uses, a plan change without leaving the page, and the billing portal for invoices.
    • Seats per plan: 1 on Free, 10 on Standard, 30 on Pro, pending invitations included.
    • Organization and project switchers sit in the top bar.
    • Every page opens on the last 30 days.
    • Calmer colours: colour marks results and real problems only, labels look the same everywhere, and a flaky result has its own zigzag mark - in the app and in emails.
61. 2 Oct 2026

    API
    1.0.0-beta.7

    GET /projects/{projectId}/overview counts like the dashboard

    Breaking

    GET /projects/{projectId}/overview counts like the dashboard: every figure is over the window’s CI runs whose shards all finished, each run counted once however many shards it had. Field names and types stay; their meaning changes, and two fields are added.

    Added

    • unfinishedRuns: CI runs of the window on branchScope that have not finished. They are in no other figure.
    • lastRun.flaky and lastRun.skipped: with passed (now first-try passes) and failed they sum to lastRun.total.

    Changed

    • lastRun: the newest finished CI run on branchScope, whatever the window, its counts over all its shards and its id as /runs gives it. It was the newest shard row, finished or not, with that shard’s counts.
    • runsInWindow: finished CI runs, mass-failure and cancelled runs included. It counted every completed shard without mass failures.
    • windowStability: over the same runs, mass-failure runs included.
    • suiteSize: the largest run, every test once over all its shards.
    • timeLost.wallClockMs: each shard’s own duration, summed over the window’s finished CI runs; local and unfinished runs no longer count.
    • failingTests: over the window’s finished CI runs; local runs no longer count.
    • testMetrics: on branchScope. It counted every branch.
    • slowestTests and timeSinks: up to 5 tests, was 3 and 4. Skipped results are left out of the median.
    • runsSeries: the last 30 finished CI runs on branchScope, whatever the window, one point per run, durationMs from its first shard’s start to its last shard’s finish. It was the last 20 shards.
62. 1 Oct 2026

    MCP
    0.2.3

    Test metrics tools

    BreakingNew

    • Breaking: squally-list-tests and squally-get-test-metrics replace squally-list-flaky-tests and squally-get-test-status - plain stability, flaky rate and failure rate per test. Needs read API 1.0.0-beta.2.
    • 0.2.1: a test’s last runs include skipped ones.
    • 0.2.2: runs say whether they were cancelled, and attempts whether they timed out. Needs read API 1.0.0-beta.4.
    • 0.2.3: days defaults to 30.
63. 1 Oct 2026

    API
    1.0.0-beta.6

    A run of a test is a CI run as a whole now: all its shards count as one run, and only once every shard has finished

    Breaking

    A run of a test is a CI run as a whole now: all its shards count as one run, and only once every shard has finished. Field names and types stay; their meaning changes, and the population of the per-test metrics has a new value. Runs on a developer’s machine still never count.

    Added

    • occurrences[].local and occurrences[].counted on GET /projects/{projectId}/tests/{testName}: whether a run is local, and whether it is in the metrics’ population.

    Changed

    • population on listTests, getTestMetrics and the overview’s testMetrics: "ci_groups_finished_not_mass_failure", was "ci_completed_not_excluded". Counted are CI runs whose shards all finished, by the start of their first shard; a run that was a mass failure as a whole is left out. It was every completed shard on its own.
    • runs, stableRuns, flakyRuns, failedRuns and the rates: per run, however many shards it had. The final attempt is the one with the most retries, then from the lowest shard, then the first repeatEach repetition. Unchanged for a test that ran in one shard per run.
    • recentResults[].runId: the run’s id as GET /projects/{projectId}/runs gives it, one per run. It was the id of the shard the test ran in. A test a cancellation interrupted no longer appears as skipped.
    • occurrences on GET /projects/{projectId}/tests/{testName}: one per run, its runId as /runs gives it, its status the final attempt across all shards. It was one per shard.
    • The overview’s testMetrics (ciTests, failingTests, testsWithRetries) counts over the same runs. Its other fields are unchanged in this version.
64. 1 Oct 2026

    API
    1.0.0-beta.5

    GET /projects/{projectId}/tests and GET /projects/{projectId}/tests/{testName}/metrics default to 30 days when days is left out

    Changed

    GET /projects/{projectId}/tests and GET /projects/{projectId}/tests/{testName}/metrics default to 30 days when days is left out; they defaulted to 14. Every read endpoint has the same default now - the period the app’s pages open on, the tests list and a test’s page included. A request that sends days gets what it got before.

    Changed

    • days on listTests and getTestMetrics: default 30, was 14. The allowed values (7, 14, 30, 90) and the retention limit are unchanged, and the response’s days, since and until name the window used, as before. To keep the old window, send days=14.
    • copyPrompt on GET /projects/{projectId}/runs/{runId}/tests/{testName}: its evidence line is the test’s per-test metrics over the default period, so it reads “Last 30 days on CI: …” where it read “Last 14 days on CI: …”.
    • The examples of listTests and getTestMetrics show a 30-day window.
65. 30 Sep 2026

    API
    1.0.0-beta.4text only

    Two error sentences name the page where read keys live since the settings restructure

    Improved

    No path, parameter, field or type changed. Two error sentences name the page where read keys live since the settings restructure.

    Changed

    • 401 key_expired: “… Create a new one in Organization › Read keys.” (was “… in Settings > API keys.”)
    • 401 wrong_key_type: “… Read routes need an organization read key - Organization › Read keys.” (was “- Settings > API keys.”)
    • The document’s own hints name the same page: the fix hints of invalid_key, key_revoked and key_expired, the info description and the security scheme’s description (were “Settings → API keys”).
66. 28 Sep 2026

    API
    1.0.0-beta.4document only

    The document now calls an error an error, as the app does

    Improved

    No path, parameter, field or type changed, and no response changed. The document now calls an error an error, as the app does.

    Changed

    • GET /projects/{projectId}/errors: summary “List error groups” is now “List errors”; the description says “the project’s errors in a period - failures grouped by fingerprint”.
    • GET /projects/{projectId}/errors/{fingerprint}: summary “Get an error group” is now “Get an error”; the description says “one error”.
    • The Errors tag and the fingerprint path parameter describe errors, not error signatures.
    • Unchanged, because they are names rather than wording: the schema ErrorSignature, the field signature, the category values, and the 404 message error_not_found sends (“No error signature with this fingerprint in this project.”).
67. 27 Sep 2026

    Reporter
    0.10.2

    Eight more CI providers, cancelled runs and timeouts

    NewImprovedFixed

    • Jenkins, Azure Pipelines, Bitbucket Pipelines, Buildkite, TeamCity, AWS CodeBuild, Travis CI and Drone are recognised, shards included. Runs on Jenkins, Azure Pipelines, TeamCity and CodeBuild now count as CI runs.
    • Pull requests are reported under their own branch and head commit, so GitHub checks and comments land on the pull request. The head commit is fetched when a shallow checkout lacks it.
    • Cancelled runs show as Cancelled - on GitHub Actions, start Playwright with exec.
    • Each result says whether it timed out, and what its limit was.
    • SAME_REPO_AS_APP=true works on every provider, and TESTED_REVISION wins when both are set. Only CI=true or CI=1 marks a run as CI.
    • A test Playwright interrupts is reported as skipped; a test.fail() test that times out counts as a failure; the branch is never reported as HEAD.
    • 0.10.1: a head-commit fetch that times out no longer leaves git processes behind.
    • 0.10.2: secrets are masked in error text, console output and the commit message before upload.
68. 27 Sep 2026

    AppAIShardsReporter

    Run comparison, the Tests page and AI for a single attempt

    NewImprovedBreaking

    • Compare two runs, test by test: what newly fails, what passes again, what still fails, and what ran in only one of them. A sharded run is compared as one.
    • The Tests page replaces the flaky list. Every test with its stability, flaky rate and failure rate over your CI runs - plain numbers instead of a label. A test that passed only after a retry counts as flaky.
    • Analyse a single failed attempt with AI, or copy a ready-made prompt for your coding agent with Copy for AI. One analysis covers every shard of a run.
    • Cancelled runs show as Cancelled, and an attempt that ran into its time limit says so, with the limit. Needs squally-reporter 0.10.
    • Secrets in error text are masked when results arrive.
    • Only admins change settings; members see them.
    • The shards panel is a card on the Summary tab of a run’s page.
    • Breaking: squally-reporter versions before 0.5.0 can no longer send results. Squally answers them with a message to upgrade.
69. 27 Sep 2026

    API
    1.0.0-beta.4behaviour change

    Secrets are now masked in the error text Squally stores, so the read API returns that text masked

    Changed

    No field, type or description changed. Secrets are now masked in the error text Squally stores, so the read API returns that text masked.

    Changed

    • For results received from 2026-09-27 on, error text is masked before it is stored: the error message and stack, every error’s message, stack and snippet, and the page snapshot. It reaches the read API in tests[].errorMessage on GET /projects/{projectId}/runs/{runId}, attempts[].errors (message, stack, snippet) and attempts[].copyPrompt on GET /projects/{projectId}/runs/{runId}/tests/{testName}, and message and title on GET /projects/{projectId}/errors and GET /projects/{projectId}/errors/{fingerprint}. Console output was masked already; it now uses the same, wider list.
    • A secret becomes [redacted]. Where it follows a scheme or a URL’s user name, those stay: Authorization: Bearer [redacted], https://x-access-token:[redacted]@github.com/…. Masked are GitHub tokens of every type (ghp_, gho_, ghu_, ghs_ - the long stateless installation tokens included - ghr_, github_pat_), Squally keys (sqly_), sk_ keys, AWS access key ids, Bearer and Basic Authorization headers, and the password in any scheme://user:password@host URL.
    • An error’s fingerprint is computed from the masked text: two failures that differ only in a secret are one error group. Results stored before keep their text and their fingerprint.

    Removed

    • Outside the read API: ingest v1 (POST /api/ingest) is removed and answers 410 with code: "ingest_v1_removed". squally-reporter 0.5.0 and later use ingest v2 and are not affected.
70. 27 Sep 2026

    API
    1.0.0-beta.4

    Runs say whether they were cancelled, and tests say whether an attempt ran into its time limit

    New

    Runs say whether they were cancelled, and tests say whether an attempt ran into its time limit. Every new field is always present. Some listed behaviour already applied from 2026-09-27, before this release; it is documented here. Two descriptions are corrected as well: commitSha is described for the first time - the commit the run is reported under, the pushed commit or for a pull request its head - and testedRevision is re-described - the application revision the run was told it tested, opt-in and null unless the CI job sets TESTED_REVISION, or SAME_REPO_AS_APP=true for the commit CI built; it is never the pull request’s head. The ?sha= description said testedRevision was “a pull request’s head behind a merge commit”, which no reporter has ever sent; ?sha= still matches either field.

    Added

    • cancelled (boolean) and cancellation (source, at and byUserId, or null) on each run of GET /projects/{projectId}/runs and on run of GET /projects/{projectId}/runs/{runId}. A sharded run is cancelled when any shard recorded a cancel; at is the first one.
    • cancellation.source is reporter (Playwright was interrupted - a cancelled CI job, a Ctrl-C - and the reporter said so), cli or user. It is a plain string, not a closed set: new sources may be added.
    • A run’s status never says cancelled. A cancelled run keeps the outcome of what it reported: passed or failed, or running / incomplete while a shard stopped by the same cancel is still open.
    • timedOut (boolean or null) and timeoutMs (integer or null) on each test of GET /projects/{projectId}/runs/{runId} (the final attempt’s), on each attempt of GET /projects/{projectId}/runs/{runId}/tests/{testName}, and on each occurrence of GET /projects/{projectId}/tests/{testName} (the final attempt’s). null means the reporter did not say (squally-reporter before 0.10.0). timeoutMs 0 means no limit.

    Changed

    • A cancelled run’s finishedAt and durationMs (runs list and run) no longer stay null while a shard stopped by the same cancel never reported: the run ends at the latest shard that finished.
    • ?status=passed and ?status=failed on GET /projects/{projectId}/runs match no cancelled run - only the unfiltered list has it, and total counts the same way.
    • runsSeries on GET /projects/{projectId}/overview leaves out cancelled runs, every shard of a cancelled CI run included.
    • copyPrompt on GET /projects/{projectId}/runs/{runId}/tests/{testName}: an attempt that ran into its time limit gets the line “Timed out at its N ms limit” after its duration, and its “Failure category” line says Timeout.
    • Errors on GET /projects/{projectId}/errors and GET /projects/{projectId}/errors/{fingerprint}: when the example occurrence of an error ran into its time limit, category is Timeout Issues and title starts with “Timeout”, whatever the message says. This applies to results from squally-reporter 0.10.0 on; existing errors keep their category. The category values themselves are unchanged.
71. 26 Sep 2026

    API
    1.0.0-beta.3

    A test’s recentResults now include the runs in which its final attempt was skipped, with result skipped

    New

    A test’s recentResults now include the runs in which its final attempt was skipped, with result skipped - on GET /projects/{projectId}/tests (each item) and GET /projects/{projectId}/tests/{testName}/metrics. Until now those runs were left out. They are shown and still counted nowhere: not in runs, stableRuns, flakyRuns or failedRuns, not in any rate, and not in timeLostMs, branches or topBranch. The list is still the last 20 runs, oldest first. A client that reads result as a closed set of stable, flaky and failed has to accept the fourth value.

    Changed

    • The Copy-for-AI prompt (copyPrompt on GET /projects/{projectId}/runs/{runId}/tests/{testName}) says a test that passed on a retry had “a flaky run”, where it said “the test is flaky rather than consistently broken”. Text only - no field or type changed.

    Removed

    • GET /projects/{projectId}/flaky (listFlakyTests), as announced when it was deprecated in 1.0.0-beta.2: the 301 to /projects/{projectId}/tests is gone. Call /projects/{projectId}/tests. The path now matches no route and gets the same 404 as any unknown path under /api/v1 - the server’s generic page, HTML, not the {error, code} JSON body - with or without a key, and with no Location header.
72. 25 Sep 2026

    MCP
    0.1.4

    Squally in your coding agent

    NewFixed

    • First release of squally-mcp: read-only tools for your coding agent - projects, runs, a failing test’s attempts, test status, flaky tests and errors - with an organization read key.
    • 0.1.1 to 0.1.4: the operating system’s certificate store is used, a failed request names its cause, and flaky tests carry the number of runs they appeared in.
73. 25 Sep 2026

    API
    1.0.0-beta.2

    The flakiness engine behind /flaky and the test status route is gone, and with it every verdict

    Breaking

    The flakiness engine behind /flaky and the test status route is gone, and with it every verdict: no test is called flaky, broken or healthy any more. In their place are plain per-test metrics, computed from the results on every request. Each CI run of a test ends stable (passed on the first try), flaky (passed after a retry) or failed, and the metrics count those - over completed CI runs on every branch, or on one branch when a request names it. Every field’s description in v1.json states its rule.

    Added

    • GET /projects/{projectId}/tests (listTests): every test with a CI run in the period and its metrics, ranked. Query: days (7, 14, 30 or 90, default 14), search, branch, browser, sort (stability - the default, lowest first - or flakyRate, failureRate, runs, timeLost, highest first), page, perPage (1 to 100, default 50). Response: project, population, days, since, until, branch, browser, sort, page, perPage, total, pageCount and items. Each item: testName, filePath, browser, runs, stableRuns, flakyRuns, failedRuns, stability, flakyRate, failureRate, timeLostMs, recentResults (the last 20 runs, oldest first: runId, runNumber, startedAt, branch, result - stable, flaky or failed), branches and topBranch (branch, kind - failures or retries - and count, or null). Rates are fractions from 0 to 1, null when there is nothing to divide by.
    • GET /projects/{projectId}/tests/{testName}/metrics (getTestMetrics): one test’s metrics - the fields of a list item, flat, beside project, population, days, since, until and branch. Query: days, branch and filePath. The name resolves like on the other test routes: ?filePath= when it exists in several files (400 ambiguous_test names them), 404 test_not_found for a test with no result at all. A test with results but no CI run in the period answers 200 with zero counts and null rates.
    • population is always "ci_completed_not_excluded": completed CI runs that started in the period, without runs excluded as a mass failure. Local runs never count.
    • browser is the Playwright project the stored test name starts with (chromium in chromium > cart.spec.ts > adds an item); null when the name has none.
    • days=14 is accepted by every endpoint that takes days.

    Changed

    • /projects/{projectId}/overview: flakyOrBrokenCount, flakyWindowDays and failingTests.withVerdict are removed. New: testMetrics with population, days (the response’s period), testsWithRetries, failingTests and ciTests - the dashboard’s Tests with retries and Failing tests tiles and the denominator both print, counted like /tests: CI runs only, every branch, not narrowed by branch. windowStability and runsInWindow count that population too - completed CI runs without mass failures, on the branch scope - where they counted every run in the window. windowStability is still the share of test results that passed on the first try; its description said “share of runs”, which it never was.
    • /projects/{projectId}/runs and /projects/{projectId}/runs/{runId}: counts.recovered is now counts.flaky - tests whose final attempt passed after a retry in that run, as before. They are not in passed: total is passed + failed + skipped + flaky. No per-test or per-attempt status anywhere was ever recovered; those still read passed, failed or skipped.
    • /projects/{projectId}/runs/{runId}/tests/{testName}: the copyPrompt’s record line reads “Last 14 days on CI: flaky rate …, failure rate …” - two rates over every CI run of the test, on every branch.

    Deprecated

    • GET /projects/{projectId}/flaky (listFlakyTests) answers 301 Moved Permanently to /projects/{projectId}/tests for this release and is removed in the next. The query is carried over: sort=rate becomes sort=flakyRate, sort=timeLost stays, sort=impact and status are dropped (the list’s default order, stability, applies), everything else passes through. What the redirect leads to has the new shape, so a client that parsed the old list must move to /tests. The 301 carries Deprecation (RFC 9745) and Link with rel="successor-version".

    Removed

    • GET /projects/{projectId}/tests/{testName}/status (getTestStatus), with no redirect: use /tests/{testName}/metrics. The path now matches no route and gets the server’s generic 404 page - HTML, not the {error, code} JSON body.
    • The Flakiness tag. The two new operations and the deprecated one are under Tests.
    • The schemas FlakyResponse and V4StatusResponse. New: TestsResponse and TestMetricsResponse.
74. 25 Sep 2026

    API
    1.0.0-betaadditive

    One field added

    New

    One field added; nothing removed or renamed.

    Added

    • /projects/{projectId}/flaky: every test carries runsInWindow - the runs in the window that contain the test, on all branches, with no branch or revision filter (completed runs only; a run in which the test was skipped does not count). It sits beside runsConsidered, which counts only the runs flipRate is computed over and can be 0 for a test that runs every day; runsInWindow is never smaller.

    Changed

    • runsConsidered’s description now names runsInWindow as the plain count.
75. 25 Sep 2026

    API
    1.0.0-betadocument only

    runsConsidered has a description

    Improved

    No change to any endpoint or response.

    Changed

    • /projects/{projectId}/flaky: runsConsidered has a description. It counts the runs flipRate and flakeScore are computed over - every run on the project’s stable branch, plus the runs on other branches whose testedRevision matches the test’s previous run - and not the runs the test appeared in. A test that runs every day, but only on a feature branch whose runs carry no testedRevision, shows 0 there, and its verdict can still come from retry_rate, which counts every run.
76. 24 Sep 2026

    Reporter
    0.9.0

    SQUALLY_INGEST_KEY and SQUALLY_INGEST_URL

    Improved

    • The variables are renamed to SQUALLY_INGEST_KEY and SQUALLY_INGEST_URL. Same values, same meaning.
    • SQUALLY_API_KEY and SQUALLY_API_URL still work for now, with a warning in the CI log - those names belong to the read API’s keys. A read key (sqly_ro_) in them is never sent.
77. 24 Sep 2026

    API
    1.0.0-betabehaviour change

    timeLostMs is now computed for every listed test

    Changed

    No change to any endpoint, and timeLostMs keeps its type (number | null) - but what the API puts in it changed, so the description did too.

    Changed

    • /projects/{projectId}/flaky: timeLostMs is now computed for every listed test. It used to be computed only for tests that had both passed and failed in the window, so a test that never passed - and a test that passes in every run but only on the retry - came back as null. Both now carry a number, and the two are usually the largest on the list.
    • The field description no longer says null means “never passed in the window - the worst case”. It describes the arithmetic (wasted attempts x the final attempt duration, where a failed run wastes every attempt and a retry-rescued run wastes the ones before the retry), states that 0 is a real measurement, and says null is a guard that should not occur on this response.
    • Sorting by timeLost puts null last instead of first, for the same reason: it no longer marks the worst case.
78. 24 Sep 2026

    API
    1.0.0-betadocument only

    Four revisions on one day, newest first below: the document carries more of what a client needs, and reads better

    Improved

    No change to any endpoint or response, and no operationId changed. Four revisions on one day, newest first below: the document carries more of what a client needs, and reads better.

    Changed

    • x-error-codes: the action of test_not_found now reads “Check the test name and filePath exactly as Squally shows them; do not retry.” It used to tell the caller to URL-encode the name - right for an HTTP request, wrong for an agent passing the name as a tool argument, which then gets encoded twice. The encoding rule for HTTP callers is unchanged, in the testName parameter description.
    • Every operation has a short summary (“List runs”, “Get test status”, …). The longer sentence each had before is now the first sentence of its description.
    • Operations are grouped by tags, each with a one-line description, listed on the document in this order: Projects, Runs, Tests, Flakiness, Errors, Project.
    • x-error-codes on the document: every error code the API can answer with, one entry each - code, status, when (the sentence the API sends), action (what to do) and hasBody (false only for 429, which the edge answers without a JSON body).
    • Retry rules in the description, in one place: 429 - wait 60 seconds, then retry; 5xx - exponential backoff from 1 second, capped at 30 seconds; 400, 401, 403, 404 - never retry. This replaces “back off for at least 60 s”.
    • An example body for every 200 response. The data is invented.
    • This changelog is served at /openapi/CHANGELOG.md.
    • Error responses carry their schema inline, with code limited to the slugs that route answers with at that status, and one full example (error and code) per slug. The shared components/schemas/ApiError is gone; the shape it described is unchanged.
    • Integers no longer carry the ±9007199254740991 bounds, and date-time fields no longer carry a regex next to format: date-time.
79. 23 Sep 2026

    API
    1.0.0-beta

    First published version

    New

    First published version.

    Added

    • Twelve read endpoints under /api/v1: projects, runs (list, one run, a test’s attempts with the Copy-for-AI prompt), branches, flaky tests, a test’s stored status and history, error signatures (list and one), the dashboard overview, and project settings.
80. 19 Sep 2026

    Reporter
    0.8.1

    The browser console

    NewFixed

    • The browser console of a failed attempt is uploaded, read out of the trace - turn tracing on to get it. SQUALLY_DEBUG=1 shows what was found.
    • Your test’s own output is now called Test output, so it is not mistaken for the browser console.
    • 0.8.1: Squally keys (sqly_) are masked in console output and in the uploaded browser console, and the reporter warns when an apiKey option would be written into a json or blob report.
81. 19 Sep 2026

    AppShards

    A new dashboard, expected failures and the browser console

    NewImprovedFixed

    • A new dashboard with tiles, charts of runs and run duration, and cards for the slowest tests and the ones that keep failing. On a run’s page, each outcome tile opens the run’s tests, filtered to that outcome.
    • The errors list and the error page are rebuilt.
    • Tests marked with test.fail() show as expected failures. One that starts passing is flagged, so you can remove the marker. Needs squally-reporter 0.7.
    • The browser console of a failed attempt has its own section, next to the screenshot and the trace. Needs squally-reporter 0.8.
    • A run says when results went missing on the way - fewer arrived than the reporter produced. Needs squally-reporter 0.6.
    • A sharded run’s result covers all of its shards.
    • Fixed: GitHub checks, pull-request comments and Slack messages were not sent for runs that report live.
82. 15 Sep 2026

    Reporter
    0.7.0

    Expected failures, and screenshots and traces again

    NewFixed

    • Tests marked with test.fail() are reported as such, so Squally shows them as expected failures - and flags one that starts passing.
    • Fixed: screenshots and traces are uploaded again. Between 10 and 15 September 2026 they were dropped without a warning.
    • Fixed: terminal hyperlinks and window-title sequences no longer end up in stored console output.
83. 14 Sep 2026

    Reporter
    0.6.0

    Squally can tell when results went missing

    NewImproved

    • The reporter tells Squally how many results it produced, so a run shows when fewer arrived instead of quietly listing fewer tests.
    • Sharded runs pace their uploads by the number of shards, so a large sharded suite no longer runs into Squally’s rate limit.
84. 13 Sep 2026

    AppShardsAI

    Live results, sharded runs as one, and an Errors view

    NewImproved

    • Results appear while the run is going, test by test, instead of all at once at the end. Needs squally-reporter 0.5 or newer.
    • A sharded CI run is one run: one entry and one run number in the runs list, and a shards panel that shows how each shard did. Filter a run’s tests and errors by shard.
    • The run page has three tabs - Summary, Tests and Errors.
    • Errors is a new view per project: failures grouped by their error.
    • Every test in a run has its own page, with an Overview of its attempts and its History on the branch.
    • A run that never finishes is marked incomplete.
    • Runs in which nearly every test fails at once - an app under test that was down, for example - are kept out of the test metrics.
    • AI analysis now answers in three parts and sits inside each error group.
85. 10 Sep 2026

    Reporter
    0.5.2

    Results stream live, and shards become one run

    NewFixed

    • Results are sent while the suite runs, in small batches, so the run fills in live in Squally.
    • --shard runs of one CI run are joined into one run automatically.
    • 0.5.1: a run with failing tests exits with code 1 again; 0.5.0 could exit 0 and turn CI green.
    • 0.5.2: commands that run no test, such as --list, no longer create an empty run.

    Squally accepts 0.5.0 and newer only.

86. 31 Aug 2026

    AppAIReporter

    Foundations — August 2026

    New

    Everything Squally did before September, in one place.

    • Runs and tests. A runs list, and a page for every run and every test with its attempts, commit, author and console output. Screenshots and traces of failed attempts, with a link into Playwright’s trace viewer.
    • Flaky tests. Squally spots tests that pass only some of the time, and you choose the stable branch it measures them on.
    • Organizations and projects. Sign in with email or Google, set up your organization and its first project, invite your team, and add more projects. Runs are numbered per project, and times show in your own time zone.
    • GitHub and Slack. GitHub checks and one comment per pull request; Slack messages about your runs.
    • Plans. Free, Standard and Pro, with project, seat and monthly result limits, and results kept for 30, 60 or 90 days.
    • AI failure analysis explains why a run failed. An error that was analysed once is not analysed again.
    • squally-reporter 0.1 to 0.4. The Playwright reporter sends every result with its commit message, author and console output, and uploads screenshots and traces. A missing key is a warning, never a failed build.
    • A new look - warm surfaces, a teal accent and a new logo.