Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- Clarified namespace guidance for `extra` extension keys and examples ([#56](https://github.com/ctrf-io/ctrf/pull/56)).
- Clarified immutability guidance for emitted CTRF report artifacts ([#55](https://github.com/ctrf-io/ctrf/pull/55)).
- Clarified `tags` as simple keyless classifications and `labels` as structured key-value test metadata ([#54](https://github.com/ctrf-io/ctrf/pull/54)).
- Allowed non-empty arrays of strings, numbers, and booleans as label values ([#54](https://github.com/ctrf-io/ctrf/pull/54)).
26 changes: 18 additions & 8 deletions examples/comprehensive.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,19 @@
"timestamp": "2025-11-24T12:00:00Z",
"generatedBy": "example-ci",
"extra": {
"pipelineStage": "e2e",
"trigger": "pull_request"
"myorg.ci": {
"pipelineStage": "e2e",
"trigger": "pull_request"
}
},
"results": {
"tool": {
"name": "example-runner",
"version": "3.5.0",
"extra": {
"plugins": ["retry", "screenshots", "video"]
"myorg.ci": {
"plugins": ["retry", "screenshots", "video"]
}
}
},
"summary": {
Expand Down Expand Up @@ -216,13 +220,17 @@
"testEnvironment": "staging",
"healthy": false,
"extra": {
"nodeVersion": "18.0.0",
"region": "us-west-2"
"myorg.ci": {
"nodeVersion": "18.0.0",
"region": "us-west-2"
}
}
},
"extra": {
"shardIndex": 1,
"totalShards": 4
"myorg.ci": {
"shardIndex": 1,
"totalShards": 4
}
}
},
"insights": {
Expand Down Expand Up @@ -267,7 +275,9 @@
"buildUrl": "https://ci.example.com/builds/442",
"commit": "f6e5d4c3b2a1",
"extra": {
"branch": "main"
"myorg.ci": {
"branch": "main"
}
}
}
}
162 changes: 149 additions & 13 deletions spec/ctrf.md
Original file line number Diff line number Diff line change
Expand Up @@ -333,8 +333,6 @@ A CTRF document MUST include:
- `specVersion` - MUST follow Semantic Versioning 2.0.0.
- `results` - MUST contain the results of a single test execution run.

Additional top-level fields MUST be included within an `extra` object.

### 4.3. Schema Conformance

The document MUST validate against the normative JSON Schema provided in Appendix A.
Expand All @@ -353,10 +351,50 @@ Consumers MUST reject any document containing unknown properties outside of `ext

### 4.6. Extension Mechanism

- Objects named `extra` MAY contain arbitrary data and are the only supported extensibility points in CTRF.
- Producers MAY include tool-specific or domain-specific metadata within an `extra` object.
- Consumers MAY require particular fields within an `extra` object for their platform or workflow; such fields MUST NOT appear outside `extra`.
- Consumers SHOULD ignore unrecognized fields inside any `extra` object unless they explicitly depend on them.
Objects named `extra` are the only supported extensibility points in CTRF.

Producers MAY include tool-specific or domain-specific metadata within an `extra` object.
Consumers MUST ignore unrecognized fields within any `extra` object and MUST NOT reject documents solely due to their presence.

The `extra` object defines a mapping of extension keys to values.

#### 4.6.1. Extension Keys

Each property within an `extra` object represents a distinct extension.

Extension keys:

- MUST be strings
- SHOULD be namespaced to avoid collisions
- SHOULD use a stable, recognizable prefix identifying the extension owner (e.g. `myorg.myproject`)
- SHOULD include a delimiter (e.g. `.` or `/`) separating the namespace from the identifier

The `ctrf.` and `ctrf/` namespace prefixes are reserved for CTRF-defined extensions.
Producers other than CTRF MUST NOT use extension keys beginning with `ctrf.` or `ctrf/`.

Extension keys are opaque identifiers.
Consumers MUST NOT interpret extension keys as hierarchical paths.

A producer SHOULD use the same namespace prefix consistently across all `extra` objects within a single document.
If a producer contributes extensions at multiple levels (for example, at the top level, within `results`, `summary`, or individual tests), the same namespace prefix SHOULD be used for all of that producer's extension keys throughout the document.

#### 4.6.2. Extension Values

Each extension key maps to a value.

- The value MAY be any valid JSON type.
- Extensions containing multiple related fields SHOULD use an object value.
- Extensions with a single value MAY use a scalar.

Producers SHOULD group related extension data under a single extension key.

#### 4.6.3. Extension Semantics

The structure and semantics of each extension value are defined by the extension producer.

- CTRF does not define or validate extension value structure.
- Consumers MUST treat values for unrecognized extension keys as opaque.
- Consumers MUST NOT assume the presence or meaning of any extension unless explicitly supported.

### 4.7. Versioning and Forward Compatibility

Expand Down Expand Up @@ -2985,15 +3023,19 @@ It includes:
"timestamp": "2025-11-24T12:00:00Z",
"generatedBy": "example-ci",
"extra": {
"pipelineStage": "e2e",
"trigger": "pull_request"
"myorg.ci": {
"pipelineStage": "e2e",
"trigger": "pull_request"
}
},
"results": {
"tool": {
"name": "example-runner",
"version": "5.0.0",
"extra": {
"parallel": true
"myorg.ci": {
"parallel": true
}
}
},
"summary": {
Expand All @@ -3009,7 +3051,9 @@ It includes:
"stop": 1700000312000,
"duration": 12000,
"extra": {
"reportedBy": "summary-module"
"myorg.ci": {
"reportedBy": "summary-module"
}
}
},
"tests": [
Expand Down Expand Up @@ -3111,11 +3155,15 @@ It includes:
"testEnvironment": "staging",
"healthy": false,
"extra": {
"container": "node:18"
"myorg.ci": {
"container": "node:18"
}
}
},
"extra": {
"runShard": 1
"myorg.ci": {
"runShard": 1
}
}
},
"insights": {
Expand All @@ -3139,7 +3187,95 @@ It includes:
"source": "main branch - last stable build",
"buildNumber": 9800,
"extra": {
"note": "Release candidate baseline"
"myorg.ci": {
"note": "Release candidate baseline"
}
}
}
}
```

### D.6. CTRF Document with Namespaced Extra Objects

This example demonstrates how producers SHOULD use namespaced extension keys within `extra` objects, grouping related fields under a single key as an object value, and using the same namespace prefix consistently across all `extra` objects in the document.

Two producers contribute extensions in this example:

- `myorg.ci` (the CI system) contributes extensions at the top level, `results`, `summary`, and per-test level — using the `myorg.ci` namespace key consistently throughout.
- `acme.test-management` (a test management integration) contributes extensions at the per-test level only — using the `acme.test-management` namespace key consistently.

The `ctrf.` and `ctrf/` namespace prefixes are reserved for CTRF-defined extensions and MUST NOT be used by other producers.

```json title="CTRF document with namespaced extra objects"
{
"reportFormat": "CTRF",
"specVersion": "0.0.0",
"extra": {
"myorg.ci": {
"pipeline": {
"stage": "e2e",
"triggerEvent": "pull_request",
"region": "us-east-1"
}
}
},
"results": {
"tool": {
"name": "example-runner",
"version": "2.0.0"
},
"summary": {
"tests": 2,
"passed": 1,
"failed": 1,
"skipped": 0,
"pending": 0,
"other": 0,
"start": 1700000000000,
"stop": 1700000005000,
"extra": {
"myorg.ci": {
"retryBudgetUsed": 2,
"retryBudgetTotal": 10
}
}
},
"tests": [
{
"name": "checkout flow completes",
"status": "passed",
"duration": 2000,
"extra": {
"myorg.ci": {
"featureFlag": "new-checkout-ui"
},
"acme.test-management": {
"issueKey": "SHOP-1234",
"testCycleName": "Sprint 42 Regression"
}
}
},
{
"name": "payment fails gracefully",
"status": "failed",
"duration": 3000,
"message": "Expected error banner to be visible",
"extra": {
"myorg.ci": {
"featureFlag": "new-payment-ui"
},
"acme.test-management": {
"issueKey": "SHOP-1235",
"testCycleName": "Sprint 42 Regression"
}
}
}
],
"extra": {
"myorg.ci": {
"deploymentId": "deploy-abc123",
"environment": "staging"
}
}
}
}
Expand Down
Loading