Email reports

What a report is

A weekly email about one project. It covers the seven full days before the day it is sent, in the report’s time zone, and the CI runs of that week - on all branches, or on one branch the report names. Local runs are never in it.

It contains:

• the week’s news. The subject ends with it in short - for example “5 new failures” - and the mail starts with it in bold: “5 tests failed, none of them the week before. 3 of them failed in their latest run.” Then the passed runs and the time lost to retries against the week before;
• four tiles: Passed runs, Stability (“first-try passes · this week’s CI runs”), Time lost and New failures, each but Stability compared with the week before;
• Needs attention - up to five tests that failed this week and not the week before, each with a line like “failed 9 of 42 · flaky 1 · first failed Mon”, its latest error, and Failed, Flaky (passed after a retry) or Passed for how its latest run ended. A test that passes in its latest run after an earlier failure says since when: “no failure since Tue 16:05”;
• Flaky - the three tests with the most flaky runs that are not already under Needs attention. The count beside the title is every flaky test of the week, and a note says how many are listed under Needs attention;
• Got slower - up to three tests whose median duration grew by 25 % or more;
• The week in numbers - runs, failed runs, flaky runs (no failures), average run duration, test results, the share that failed and that passed on retry, distinct errors and error occurrences, this week and the week before. A line under it gives the failed runs by day and how many errors were new this week.

Which runs it counts

The report counts CI runs only, and its header says so: “CI runs only”. The pages count CI and local runs by default - so a first run from a laptop shows up right away - but a report goes to people who pick no filter. A teammate’s local runs - single tests, tests failing on purpose while they are written, usually without retries - would change the numbers the whole team is sent about CI. So the report stays on CI, whatever the pages default to.

It counts the same runs as the Overview with its source on CI runs - the stored rule every figure in Squally reads, not a rule of its own:

• A finished run counts, in the week its first shard started, as one run however many shards it had. A run with a flaky test is a passed run.
• A cancelled or incomplete run is a run, but not a result. It is in the run count, and the tests that finished count, but it is in no rate, no failed or flaky run count, no bar and no average duration. An incomplete run counts once it has settled - see Incomplete: waiting or settled.
• An unfinished run - one with a shard still running, or a cancelled run whose shard is still running - is in no figure. The report only counts it.

The header names the runs outside the rates in the app’s words, for example 1 cancelled · 2 incomplete · 1 unfinished.

A mass failure - a run in which most of the suite failed at once, usually an outage - counts in the run and error figures, but is left out of the test lists (failing, flaky and slow tests, and the “newly failing” count), so one broken runner does not fill them.

How the lists read

• Tests are named with their Playwright project when they ran under one, for example “applies a coupon code (firefox)”, so the chromium and the firefox variant read apart.
• Got slower compares the median duration of a test’s final attempts in the two weeks, among tests timed in at least three runs of each week - a test that timed out once does not land on the list. Each Playwright project’s variant is its own test here; Analytics’ Getting slower counts one test across projects. The test that slowed down most comes first.
• New this week is an error first seen this week. First seen is the first counted CI run, on the report’s branch scope, that had the error - the Error explorer’s First seen with the same filters.

A week without a report

A week without a counted run sends one short mail instead of the report: “No runs on main this week.” - or, when runs started but none has finished, “No finished runs on main this week. 2 unfinished.” Then the report stays silent until finished runs arrive again.

When it is sent

Each report has a weekday, a full hour and a time zone, for example every Monday at 08:00 Europe/Berlin. Squally checks every hour for reports that are due and sends them. A week that was missed is skipped, not sent late.

Who receives it

• Organization admins create, change, pause and delete reports. A project has up to 10 reports, each with up to 50 recipients.
• Members an admin adds receive the report from the next send.
• Addresses outside the organization receive a confirmation mail first, and nothing else until they confirm. The link is valid 7 days; it can be sent again at most once an hour. After confirming, the page names the date and time of the first report.
• Every member can preview a report and subscribe or unsubscribe themselves.

Reports and confirmation mails come from reports@mail.squally.dev. Adding it to your safe senders keeps them out of spam.

Unsubscribing

Every report mail has an unsubscribe link, and mail clients that support it offer one-click unsubscribe. A member who unsubscribed can only be subscribed again by themselves - not by an admin. An outside address comes back only through a new confirmation.

Plans

Weekly reports come with the Standard and Pro plans. On Free, a report is kept but paused, and recipients cannot be added; it resumes when the organization upgrades.