Verdicts

Outcome	Meaning
Passed	Connect played what the test expected, and the call ended the way it predicted.
Passed on re-run	It failed once and passed on a second run: most likely Connect missed a prompt the first time.
Failed	Connect played something else, or the call went elsewhere. Look at the step the test stopped at.
Failed twice	It failed on its re-run as well, so it is unlikely to be chance.
Not a verdict	Connect could not run the test (capacity, throttling, a service error). It is neither a pass nor a fail.
Checked, not run	A dry run: the test was converted to Connect’s format and checked, but nothing ran.
Skipped / Stopped	Not run (estimated past Connect’s time limit), or stopped before it finished.

Inconclusive results (not a verdict, skipped, stopped) are reported separately and never count towards a suite’s pass rate.

Dry runs and live runs

A dry run converts every test and validates it, without creating anything in Connect. Do one first. A live run creates the test cases in Connect and executes them; Amazon bills these executions. Live runs need a key with the PUBLISH_LIVE scope.

Interrupted runs

If the server restarts while an evolution run or a Connect run is in progress, that run is marked Interrupted: it did not finish and was not lost silently. Start it again.