Clarify retry semantics: what etryAttempts[] contains, how etries is counted, and how to report a run where only failed tests are re-executed

[Evangelink]

Hi! We produce CTRF reports from Microsoft.Testing.Platform (the Microsoft.Testing.Extensions.CtrfReport extension) and we hit three points where the written spec and the published example disagree, or where we could not find an answer at all. All three affect the same feature — retries — so I'm grouping them in one issue. Happy to split if you prefer.

Context: our platform has two very different retry mechanisms, and we want both to produce spec-correct CTRF.

1. In-process retries — the test framework re-executes a test inside a single run and publishes each attempt. One CTRF document, retryAttempts[] on the test.
2. Orchestrated retries (--retry-failed-tests N) — the runner re-launches the whole test host process, and each re-launch executes only the tests that failed in the previous attempt. Every attempt is a separate process that writes its own CTRF document.

---

1. Does retryAttempts[] contain every attempt, or only the re-executions?

The spec text and the example point in opposite directions.

Section 11 (Retry Attempt Object) intro:

Each retry attempt represents a single re-execution of a test after an initial failure or according to a retry policy.

Terminology:

Retry / Retry Attempt: Any attempt with an attempt number greater than 1.

Both of those read as "retryAttempts[] holds attempts 2..N".

But §11.1 says:

Attempt number 1 represents the first execution; higher numbers represent retries.

…which only makes sense if attempt 1 can appear in the array. And examples/with-retries.json does exactly that — a test that ran three times lists all three attempts, including the first (attempt: 1, failed) and the final passing one (attempt: 3, passed), while the test object's own status is passed:

{
  "name": "flaky test",
  "status": "passed",
  "duration": 300,
  "flaky": true,
  "retries": 2,
  "retryAttempts": [
    { "attempt": 1, "status": "failed", "duration": 120, "message": "Connection timeout" },
    { "attempt": 2, "status": "failed", "duration": 130, "message": "Connection timeout" },
    { "attempt": 3, "status": "passed", "duration": 140 }
  ]
}

Question: is the example normative here — i.e. retryAttempts[] is the complete attempt history (1..N, including the attempt whose outcome is mirrored in the test object's status)? Or is it meant to hold only attempts 2..N?

This matters a lot for consumers: if a producer emits only 2..N, the diagnostics of the first failure (message/trace/stdout) have nowhere to live, because the test object itself carries the final outcome. Our current implementation emits attempts 1..N-1 (every attempt except the final one, whose data is on the test object), which is a third shape and satisfies §9.20 below but not the example — hence the question.

2. retries — count of retryAttempts[], or count minus one?

§9.20:

retries … SHOULD equal the count of entries in retryAttempts.

The example has "retries": 2 with three entries in retryAttempts. Under the "array holds every attempt" reading, retries is attempts - 1 (the number of re-executions), which contradicts §9.20 as written.

Question: should §9.20 be amended to retries SHOULD equal retryAttempts.length - 1 (when the array holds every attempt), or should the example be corrected? A consumer computing "how many times did this test run" needs one unambiguous rule.

3. Reporting a logical run when only the failed subset is re-executed, in separate processes

This is the case that has no worked example. Concretely, a 4-test suite with --retry-failed-tests 3:

attempt	tests executed	outcome
1	4	2 passed, 2 failed
2	2	1 passed, 1 failed
3	1	1 failed
4	1	1 failed

Each attempt writes its own CTRF document, so attempt 4's document legitimately reports summary.tests: 1. Reading §2.1 ("A logical run MAY consist of multiple physical executions … Results from such executions MAY be merged into a single CTRF document") together with §2.11 (a merge produces a new document with a different reportId, leaving the inputs immutable), our understanding is:

• the per-attempt documents stay as they are (immutable artifacts, one per physical execution);
• the runner additionally emits one merged document for the logical run, in which each logical test appears once — status = the final attempt's outcome, earlier attempts folded into retryAttempts[], flaky: true where a failure was followed by a pass;
• summary.tests therefore counts logical tests (4 here), not executions (8), and summary.flaky counts the recovered ones — consistent with §8.1 ("SHOULD equal the length of the tests array") and §4.9.1 ("Producers SHOULD NOT emit duplicate identity values within a single CTRF document when those duplicates would make correlation ambiguous").

Questions:

• Is that the intended representation? A short note or an example in the spec would help — today "shards" is the only merge scenario discussed, and shards have disjoint tests, whereas retry attempts overlap, which is precisely the case where naive concatenation inflates summary.tests.
• Should the merged document and the per-attempt documents share a runId (§5.4), so a consumer can tell they describe the same logical run? The spec frames runId around shards/workers; retry attempts of the same suite feel like the same concept, but it's not spelled out.
• Is there any expectation about the test object's duration in the merged document — the final attempt's duration, or the sum across attempts? (The example's 300 matches neither its own attempt durations nor their sum, so we assume it's illustrative only.)

---

Thanks a lot for the spec — and for the quick turnaround on #53, which we followed for labels. Whatever you confirm here, we'll implement in Microsoft.Testing.Extensions.CtrfReport and reference back from microsoft/testfx#10293.

[Ma11hewThomas]

Thanks for raising these. You’re right about the inconsistencies. We’ll correct and align them.

The intended model matches your implementation:

retryAttempts[] is the attempt history preceding the final attempt. It contains attempts 1..N-1, including the initial execution and any earlier retries.

The final attempt is excluded from retryAttempts[] because its outcome and associated information are represented by the test object itself.

retries is the number of re-executions after the initial execution. It equals retryAttempts.length, and the final attempt number is retries + 1.

This matches the approach currently taken by your implementation. We’ll update the specification, schema descriptions, and examples so they express this consistently.

[Ma11hewThomas]

Regarding retries orchestrated across separate processes, your suggested approach seems sensible and aligns with the retry history model. We'll consider adding a short note or an example in the spec for this use case.

The physical-execution documents and merged document represent the same logical run, so they should share a runId. Each remains a distinct immutable artifact and should have its own reportId.

In this case, the final attempt's duration should be used for the test object duration in the merged report.

Thanks again for sharing these use cases, which are really important to help shape CTRF.