feat(otel): scaffold OTel consistent probability sampling (ot.th/ot.rv) conformance tests by genesor · Pull Request #7372 · DataDog/system-tests

genesor · 2026-07-23T09:12:43Z

Motivation

RFC for cross-tracer support of OTel consistent probability sampling (ot.th/ot.rv) tracestate members across all 9 dd-trace-* SDKs (APMAPI-2171 distributed tracing, APMAPI-2172 trace export).

Changes

Scaffolding only — every new test class is landed disabled for all 9 tracers via manifest entries (missing_feature). No tracer implements this yet.

tests/test_otel_tracestate_sampling.py (new): A1-A6 from APMAPI-2171
- Test_EmitOtOnProbabilityDecision_Rate0_{01,1,2,5,99} — golden-vector rv/th derivation at 5 fixed rates, crossed with the 23 trace IDs from tests/fixtures/sampling_rates.csv, verified against the Go Knuth-hash sampler formula
- Test_ForwardInboundOtUnchanged — inbound th/rv forwarded unchanged, never re-derived
- Test_PreserveDdAndOtherVendors — ot= handling doesn't disturb dd= or unrelated vendor tracestate members
- Test_ForceKeepClearsTh — a non-probability (force-keep, via ASM /waf) decision erases th but forwards an inherited rv
- Test_SampledWithoutOtNotFabricated — an inherited sampling decision with no ot= must not fabricate th/rv
- Test_MalformedOtHandling — malformed th/rv treated as absent, dd=/vendors preserved, trace never rejected
tests/otel/test_tracing_otlp.py: B1/B2 from APMAPI-2172 — Test_Otlp_Carries_Ot, Test_Otlp_Forwards_Inherited_Ot, asserting on the exported OTLP span's traceState
utils/_context/_scenarios/__init__.py: 4 new fixed-rate scenarios (otel_sampling_rate_0_{01,1,2,99}); rate 0.5 reuses the existing sampling scenario
manifests/*.yml (all 9 languages): missing_feature (APMAPI-2171) / (APMAPI-2172) entries for every new class

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team (note: utils/_context/_scenarios/__init__.py is modified to add the 4 new scenarios)
A docker base image is modified? N/A
A scenario is added, removed or renamed? Yes — 4 new scenarios added, needs R&P team review

…v) conformance tests

github-actions · 2026-07-23T09:13:11Z

CODEOWNERS have been resolved as:

tests/test_otel_tracestate_sampling.py                                  @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/c.yml                                                         @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_kong.yml                                                  @DataDog/system-tests-core
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
tests/otel/test_tracing_otlp.py                                         @DataDog/system-tests-core
tests/parametric/test_headers_precedence.py                             @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core

datadog-datadog-prod-us1-2 · 2026-07-23T09:18:42Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 13f800b | Docs | Datadog PR Page | Give us feedback!}

… sampling tests

vpellan · 2026-07-27T08:52:29Z

+            },
+        )
+
+    def test_malformed_th_only_treated_as_absent(self):


Maybe we could also add a test with malformed rv only, and what we do in this situation (ensure that we fallback on the same behaviour than ot.th without rv described above)

Same policy as the malformed th case: we just cleanup, we don't derive anything. No plans to add a dedicated malformed-rv-only test for now.

vpellan · 2026-07-27T09:07:39Z

+        assert self.dropped_request.status_code == 200
+        ot = _parse_ot(_outbound_tracestate(self.dropped_request))
+
+        assert ot.get("rv") == self.DROPPED_RV, "inherited rv was not forwarded when overriding an inherited drop"


I wouldn't expect rv to be included, as without a th it has no meaning anyway

Suggested change

assert ot.get("rv") == self.DROPPED_RV, "inherited rv was not forwarded when overriding an inherited drop"

assert "rv" not in ot, "inherited rv was not forwarded when overriding an inherited drop"

rv is intentionally forwarded even without th. Per the RFC: "DD makes a non-probability decision (force-keep, rate limiter) -> no th; forward an inherited rv if present, otherwise emit nothing." rv is randomness metadata a later hop can pair with its own th, so it's not meaningless on its own. Keeping the assertions as-is.

vpellan · 2026-07-27T09:24:05Z

+        assert self.rv_only_request.status_code == 200
+        ot = _parse_ot(_outbound_tracestate(self.rv_only_request))
+
+        assert ot.get("rv") == self.INHERITED_RV, "inherited rv was not forwarded on a force-keep decision"


I wouldn't expect rv to be included, as without a th it has no meaning anyway

Suggested change

assert ot.get("rv") == self.INHERITED_RV, "inherited rv was not forwarded on a force-keep decision"

assert "rv" not in ot, "inherited rv was not forwarded on a force-keep decision"

I'd rename the test test_force_keep_overrides_inherited_keep_decision

rv is intentionally forwarded even without th. Per the RFC: "DD makes a non-probability decision (force-keep, rate limiter) -> no th; forward an inherited rv if present, otherwise emit nothing." rv is randomness metadata a later hop can pair with its own th, so it's not meaningless on its own. Keeping the assertions as-is.

This test covers inbound rv-only with no th. Without th we have no signal that a probability decision happened upstream, so this isn't a case of overriding an inherited keep decision. The suggested name fits the next test instead (test_force_keep_overrides_inherited_drop_decision, which has a full th/rv pair). Keeping the current name, happy to rename to something clearer if test_force_keep_forwards_inherited_rv still reads ambiguous, but not to the suggested name.

vpellan

I also believe that we should add tests for imprecision workaround while going from 64 to 56 bit (see the annex of the RFC about 64->56 bit imprecision)

vpellan · 2026-07-27T09:54:56Z

+
+@scenarios.otel_sampling_rate_0_1
+@features.w3c_headers_injection_and_extraction
+class Test_MalformedOtHandling:


For these cases, as we generate new th and rv, should th be derived from the sampling rate, and rv from the trace id last 56 bits?

No, we do not do anything. We have no way of knowing how the decision was made upstream. We just to cleanup

vpellan · 2026-07-27T11:06:10Z

+        assert req.status_code == 200, "a malformed ot.th must not cause the trace to be rejected"
+
+        ot = _parse_ot(_outbound_tracestate(req))
+        assert ot.get("rv") == self.MALFORMED_TH_RV, "the well-formed inbound rv was not forwarded"


Again, I wouldn't expect rv to be included, as without a th it has no meaning.

Suggested change

assert ot.get("rv") == self.MALFORMED_TH_RV, "the well-formed inbound rv was not forwarded"

assert "rv" not in ot, "the well-formed inbound rv was forwarded"

rv is intentionally forwarded even without th. Per the RFC: "DD makes a non-probability decision (force-keep, rate limiter) -> no th; forward an inherited rv if present, otherwise emit nothing." rv is randomness metadata a later hop can pair with its own th, so it's not meaningless on its own. Keeping the assertions as-is.

…ed cpp/golang variants

vpellan · 2026-07-28T13:34:44Z

There are a bunch of tests intests/parametric/test_headers_precedence.py that needs to be fixed, because they were checking that we had exactly 1 vendor in the tracestate which is now not true anymore, and I believe we should do it on this PR as well.
Basically in all these tests we need to change

assert len(tracestate_1_arr) == 1
assert tracestate_1_arr[0].startswith("dd=")

into

assert any(item.startswith("dd=") for item in tracestate_1_arr)

The test are:

test_headers_precedence_propagationstyle_tracecontext_datadog: Only part 1, with no headers
test_headers_precedence_propagationstyle_tracecontext: Part 1, with no headers, but also part 5, with Only Datadog headers, as the datadog headers should be ignored because DD_TRACE_PROPAGATION_STYLE is set to tracecontext only. I believe for 6 too, with an invalid tracecontext, we should generate a new trace id, which means we're the root span, and we should add ot to the tracestate
test_headers_precedence_propagationstyle_datadog_tracecontext: Only part 1, with no headers as well

…ding

…nvoy/haproxy

…ce tests

genesor · 2026-07-29T08:18:01Z

Good catch, fixed in 52e0d3a. Updated the 5 assertions across the 3 tests to assert any(item.startswith("dd=") for item in tracestate_arr).

Part 6 already falls into the "new Datadog span context" path, so the same fix covers it.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: efe7f14b08

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…flow

feat(otel): scaffold OTel consistent probability sampling (ot.th/ot.r…

d14c535

…v) conformance tests

genesor added 3 commits July 23, 2026 11:44

refactor(otel): use plain naming and dedupe th per rate in tracestate…

46e0704

… sampling tests

test(otel): add a not-sampled trace ID at every sampling rate

d86cb86

test(otel): fix A6 traceparent consistency, add A4 drop-to-keep case

427d20f

genesor mentioned this pull request Jul 23, 2026

feat(ddtrace/tracer): emit and propagate OpenTelemetry ot.th/ot.rv sampling DataDog/dd-trace-go#5060

Open

9 tasks

genesor added 2 commits July 23, 2026 16:39

test(otel): verify th-only decisions don't fabricate rv

84f89ed

test(otel): clear malformed ot.rv/ot.th instead of re-deriving

2c846d1

vpellan reviewed Jul 24, 2026

View reviewed changes

Comment thread tests/test_otel_tracestate_sampling.py Outdated

vpellan reviewed Jul 27, 2026

View reviewed changes

test(otel): add A2c/A2d sampled=False forward-unchanged cases

a0aaef4

vpellan reviewed Jul 27, 2026

View reviewed changes

test(otel): right-pad trimmed ot.th when recovering the A1 sample rate

7cef7a0

genesor mentioned this pull request Jul 27, 2026

test(golang): enable passing ot.th/ot.rv tracestate sampling scenarios [golang@ben.db/feat-otel-th-rv-propagation] #7389

Open

chore(otel): fix ruff format line-length violation

ea5747d

MilanGarnier mentioned this pull request Jul 27, 2026

test(rust): enable passing ot.th/ot.rv tracestate sampling scenarios [rust@milan.garnier/otel-consistent-sampling] #7391

Draft

test(otel): add A7 precision boundary decision tests

26a8a6c

vpellan mentioned this pull request Jul 27, 2026

[APMAPI-2178] Implement ot.th and ot.rv support DataDog/dd-trace-rb#6111

Open

genesor added 2 commits July 27, 2026 23:35

test(otel): remove redundant recovered_rate assertion in A1

dc68599

fix(otel): guard otel-tracestate-sampling and otlp tests on unsupport…

b85e914

…ed cpp/golang variants

genesor and others added 3 commits July 28, 2026 16:25

Merge branch 'main' into ben.db/feat-otel-tracestate-sampling-scaffol…

8ad7115

…ding

fix(otel): mark otel-tracestate-sampling proxy tests irrelevant for e…

ead4331

…nvoy/haproxy

fix(parametric): tolerate multi-vendor tracestate in headers preceden…

52e0d3a

…ce tests

test(otel): mark ot.th/ot.rv scenarios missing_feature for the c tracer

efe7f14

genesor marked this pull request as ready for review July 29, 2026 13:03

genesor requested review from a team as code owners July 29, 2026 13:03

genesor requested review from avara1986, christophe-papazian, dromanol, jandro996, mhlidd and tabgok and removed request for a team July 29, 2026 13:03

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread utils/_context/_scenarios/__init__.py

Comment thread utils/_context/_scenarios/__init__.py

Comment thread tests/otel/test_tracing_otlp.py

Comment thread tests/parametric/test_headers_precedence.py

fix(ci): register OTEL_SAMPLING_RATE_0_* scenarios in end-to-end work…

13f800b

…flow

	assert ot.get("rv") == self.DROPPED_RV, "inherited rv was not forwarded when overriding an inherited drop"
	assert "rv" not in ot, "inherited rv was not forwarded when overriding an inherited drop"

	assert ot.get("rv") == self.INHERITED_RV, "inherited rv was not forwarded on a force-keep decision"
	assert "rv" not in ot, "inherited rv was not forwarded on a force-keep decision"

	assert ot.get("rv") == self.MALFORMED_TH_RV, "the well-formed inbound rv was not forwarded"
	assert "rv" not in ot, "the well-formed inbound rv was forwarded"

Uh oh!

Conversation

genesor commented Jul 23, 2026

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jul 23, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vpellan Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vpellan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vpellan commented Jul 28, 2026

Uh oh!

genesor commented Jul 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 23, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Jul 23, 2026 •

edited by datadog-official Bot

Loading

vpellan Jul 27, 2026 •

edited

Loading