Expected failures
How Squally reads Playwright's test.fail() - counted as a pass while it fails, flagged when it starts passing.
Playwright’s test.fail() marks a test as expected to fail: it guards a bug
you know about, and Playwright runs the test to check that it still fails.
Squally reads it the same way. It needs squally-reporter 0.7.0 or newer; both
test.fail() and a testInfo.fail() call inside the test are recognised.
How Squally counts it
Section titled “How Squally counts it”| The marked test… | Squally shows | Counts as |
|---|---|---|
| fails, as declared | an expected failure chip | a pass |
| passes | passed — remove test.fail() | a failure |
| runs out of time | a failure | a failure |
The second row is the one to act on: the bug is fixed, and the marker should go. The third follows Playwright, which does not count a timeout as the expected failure either (squally-reporter 0.10.0 or newer).
What it affects
Section titled “What it affects”- Test metrics - stability, flaky rate and failure rate use the counting above. See How Squally measures tests.
- The GitHub check and pull-request comment - an expected failure does not fail them.
- The mass-failure check - a run of at least 20 tests in which more than 80 % failed is left out of the metrics (see How Squally measures tests). Expected failures do not count as failures for that check; a marked test that unexpectedly passes does.
- Errors and AI - an expected failure creates no error entry and is not offered for Fix with Squally.
test.skip() and test.fixme() say something else and are not affected.