Live results

Results while the suite runs

The reporter sends results in small batches as tests finish: when 25 are ready, or else on a timer: every 2 seconds, or every 0.4 seconds × the number of shards when that is longer, and never less often than every 30 seconds. The run appears in Squally as soon as its first test starts, and its results fill in while the suite is still running. This needs squally-reporter 0.5.0 or newer - older versions are refused.

Seeing new results

While a run is in progress, its page, the runs list and the project overview update by themselves every 10 seconds - the run’s header and the runs list say Live meanwhile. When nothing is running, the runs list and the overview still check once a minute, so a run that starts while they are open appears without a reload; after five minutes without mouse or keyboard input they pause until you are back. A tab in the background checks nothing.

Live progress

While a run is in progress, its page shows Progress instead of the result: a bar with the tests that finished, a hatched remainder for the ones still to run, and the count as finished/planned - “38/47”. A sharded run has a row per shard, each with its own count, and a shard that has not started yet says “Waiting for shard 3”. Squally never estimates a remaining time.

• squally-reporter 0.13.0 or newer sends the planned count when each run or shard starts. With an older reporter, the page shows the count of finished tests only, without the “/47” and the remainder.
• The plan counts each test once however often it runs (--repeat-each), and each Playwright project’s tests on their own. Tests of a setup project run in every shard and are planned in each, so the remainder can stay a little above zero until the run ends.
• Local sharded runs: start every shard with the same SQUALLY_CI_RUN_ID, or each invocation becomes a run of its own - see Local runs.

Running

Running means the run has started and has not reported its end yet. A sharded run is running while any shard is - and while it waits for a shard that has not started yet, one a CI queue holds back, say: it stays running until the run timeout has passed since its last activity, and only then is it incomplete. Its counts cover only what has arrived. The runs list marks running runs with a spinning circle, groups them under Running now at the top, and shows them under Running in the result menu of its filter sentence (any result).

Incomplete

Incomplete means the run will not finish:

• it never reported its end - its job was killed, or the reporter stopped - and Squally’s hourly check found it silent for longer than the run timeout (60 minutes by default), or a shard had still not started once the run timeout passed (see Timeouts); or
• the organization’s monthly result limit was reached during the run.

Until the run timeout has passed since its last activity, a run with a timed-out or limit-stopped shard is waiting: a late shard can still report. After that it is settled: its counts are final, the note goes, and it counts like a cancelled one - the results its shards reported count, and the run stays out of the run rates. A CI run that settles gets one GitHub check and one Slack message then. See Incomplete: waiting or settled. The runs list marks both with a half-filled yellow circle and shows them under Incomplete in the result menu of its filter sentence. A run that was stopped with Ctrl+C or a cancelled CI job and could still report its end is Cancelled, not incomplete - see Cancelled runs. Running runs count nowhere until they finish.

Results that went missing

The reporter also says how many results it produced (squally-reporter 0.6.0 or newer). When fewer arrived, the run shows “N of M results did not reach Squally.” See Troubleshooting.