test(otlp-trace-metrics): maintain parity with the OTEP spec by mabdinur · Pull Request #7363 · DataDog/system-tests

mabdinur · 2026-07-22T19:10:06Z

Summary

Follow-up to #6834 (already merged). This addresses the review threads on that PR that were left unresolved.

Resolved with concrete test changes:

Test_FR02_Mutual_Exclusion.test_fr02_3_otlp_suppresses_native_stats now also sets DD_TRACE_STATS_COMPUTATION_ENABLED=1, proving OTLP wins even when native stats computation is explicitly enabled too (not just left at default) — directly answers "what happens if both are true, which one wins?"
test_fr01_5_disabled_when_traces_exporter_not_otlp renamed to test_fr01_5_disabled_when_tracing_is_disabled, plus a new test_fr01_6_disabled_when_apm_tracing_disabled covering DD_APM_TRACING_ENABLED=false (APM standalone mode).
test_fr06_1_resource_span_name now also asserts datadog.resource.name is absent, matching the module docstring's claim that it's never emitted in either mode.
service.name is now a required data-point attribute. test_fr06_9_service_env_version and test_fr06_14_custom_service_on_data_point now assert service.name is present on every data point, matching the OTel Span Metrics Connector, which always attaches it as one of its default dimensions — regardless of whether it matches the resource's default service.
OTel-semantics mode is now restricted to the SMC's default attribute set. test_fr07_3_otel_attributes_present (which asserted extra OTel semantic-convention attributes like http.* WERE present) is replaced by test_fr07_3_only_smc_default_attributes, which asserts data points carry only service.name, span.name, span.kind, status.code — the Span Metrics Connector's real default dimensions — and nothing else.
span.kind and status.code values are narrowed to the SMC's actual output format, based on reading the SMC connector source and traceutil.go directly:
- status.code must be exactly "STATUS_CODE_ERROR" (a string) — the previously-tolerated bare "ERROR" and integer 2 don't correspond to any real SMC output for this key by default (those only appear under a different, feature-gated otel.status_code key).
- span.kind must be exactly "SPAN_KIND_SERVER" — the previously-tolerated "server"/"SERVER" casing doesn't match the SMC's own convention.

Verified against the real Python tracer (not just collected — actually run):

Ran the full file against TEST_LIBRARY=python. 4 tests fail against the current Python implementation and are now gated missing_feature in manifests/python.yml:
- test_fr01_6_disabled_when_apm_tracing_disabled — OTLP trace metrics are still emitted when DD_APM_TRACING_ENABLED=false.
- test_fr06_2_span_kind — emits the raw Datadog value ("server"), not SPAN_KIND_SERVER.
- test_fr06_8_status_code_error — emits status.code as the integer 2, not "STATUS_CODE_ERROR".
- test_fr09_2_error_count — depends on the same status.code format gap, so the error span isn't counted.
test_fr06_9/test_fr06_14/test_fr07_3_only_smc_default_attributes were already known-failing (service.name / SMC-only-attrs are genuinely new requirements) and were already gated.
Node.js was not re-run against a live tracer this round; manifests/nodejs.yml carries the same 3 known-certain gates (test_fr06_9, test_fr06_14, test_fr07_3_only_smc_default_attributes) as a starting point — CI will surface if it also needs the span.kind/status.code/fr01_6 gates.

Why draft

Marked draft on purpose. The service.name and OTel-semantics-mode design decisions above are now resolved and encoded in the tests, but this should still land after the OTEL spec (RFC SEMCON-1093) is finalized, since it's the authoritative source these tests are meant to track.

Test plan

ruff format --check / ruff check pass
mypy passes
./run.sh PARAMETRIC --collect-only --scenario-report -k test_otlp_trace_metrics collects all 52 tests cleanly
./run.sh PARAMETRIC -k test_otlp_trace_metrics run live against TEST_LIBRARY=python; all non-gated tests pass, gated tests xfail as expected
CI parametric jobs (to be monitored once opened)

🤖 Generated with Claude Code

Resolves the mechanically-actionable open threads from PR #6834: - FR02 mutual-exclusion test now also enables native stats computation explicitly, proving OTLP wins even when both mechanisms are turned on. - FR01 test renamed to test_fr01_5_disabled_when_tracing_is_disabled, plus a new test_fr01_6 covering DD_APM_TRACING_ENABLED=false. - span.kind casing is now tolerant of both "server" and "SERVER" pending spec finalization, via a new SPAN_KIND_SERVER_VALUES tuple. - FR06 resource/span-name test now also asserts datadog.resource.name is absent, matching the module's "in both modes" claim. Two open design questions (service.name repetition on data points, and the OTel-semantics-mode attribute set) are flagged with docstring notes rather than resolved, since they depend on the still-unfinalized OTEL spec (RFC SEMCON-1093). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

github-actions · 2026-07-22T19:10:34Z

CODEOWNERS have been resolved as:

manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
tests/parametric/test_otlp_trace_metrics.py                             @DataDog/system-tests-core @DataDog/apm-sdk-capabilities

datadog-datadog-us1-prod · 2026-07-22T19:38:55Z

Tests

⚠️ Warnings

🚦 4 Pipeline jobs failed

Testing the test | System Tests (nodejs, dev) / parametric / parametric (2)

🧪 4 Tests failed

tests.parametric.test_otlp_trace_metrics.Test_FR01_Enablement_Configuration.test_fr01_6_disabled_when_apm_tracing_disabled[library_env0, parametric-nodejs]

from system_tests_suite

AssertionError: OTLP trace metrics must be disabled when APM tracing is disabled, got: [&#39;traces.span.sdk.metrics.duration&#39;]
assert not [{&#39;attributes&#39;: [{&#39;key&#39;: &#39;span.name&#39;, &#39;value&#39;: {&#39;stringValue&#39;: &#39;web.request&#39;}}, {&#39;key&#39;: &#39;datadog.operation.name&#39;, &#39;val...ketCounts&#39;: [0, 0, 1, 0, 0, 0, ...], &#39;count&#39;: 1, &#39;explicitBounds&#39;: [0.002, 0.004, 0.006, 0.008, 0.01, 0.05, ...], ...}]
 &#43;  where [{&#39;attributes&#39;: [{&#39;key&#39;: &#39;span.name&#39;, &#39;value&#39;: {&#39;stringValue&#39;: &#39;web.request&#39;}}, {&#39;key&#39;: &#39;datadog.operation.name&#39;, &#39;val...ketCounts&#39;: [0, 0, 1, 0, 0, 0, ...], &#39;count&#39;: 1, &#39;explicitBounds&#39;: [0.002, 0.004, 0.006, 0.008, 0.01, 0.05, ...], ...}] = _duration_data_points([{&#39;resourceMetrics&#39;: [{&#39;resource&#39;: {&#39;attributes&#39;: [{...}, {...}, {...}, {...}, {...}, {...}, ...], &#39;droppedAttributesCount&#39;: 0}, &#39;scopeMetrics&#39;: [{&#39;metrics&#39;: [...]}]}]}])

self = &lt;tests.parametric.test_otlp_trace_metrics.Test_FR01_Enablement_Configuration object at 0x7f91309c0260&gt;
otlp_trace_metrics_library_env = {&#39;DD_APM_TRACING_ENABLED&#39;: &#39;false&#39;, &#39;DD_SERVICE&#39;: &#39;test-otlp-stats-svc&#39;, &#39;OTEL_EXPORTER_OTLP_METRICS_ENDPOINT&#39;: &#39;http:...pm-test-agent-worker-gw2-5740354900026072187:4318/v1/metrics&#39;, &#39;OTEL_EXPORTER_OTLP_METRICS_PROTOCOL&#39;: &#39;http/json&#39;, ...}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f912fe61c70&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f912f756ba0&gt;

    @pytest.mark.parametrize(
...

tests.parametric.test_otlp_trace_metrics.Test_FR06_Otel_Span_Attributes.test_fr06_2_span_kind[library_env0, parametric-nodejs]

from system_tests_suite

AssertionError: Expected span.kind in (&#39;SPAN_KIND_SERVER&#39;,), got attrs: {&#39;span.name&#39;: &#39;web.request&#39;, &#39;span.kind&#39;: &#39;server&#39;, &#39;datadog.operation.name&#39;: &#39;web.request&#39;, &#39;datadog.span.type&#39;: &#39;web&#39;, &#39;datadog.span.top_level&#39;: True}
assert &#39;server&#39; in (&#39;SPAN_KIND_SERVER&#39;,)
 &#43;  where &#39;server&#39; = &lt;built-in method get of dict object at 0x7fd69f6e4f40&gt;(&#39;span.kind&#39;)
 &#43;    where &lt;built-in method get of dict object at 0x7fd69f6e4f40&gt; = {&#39;datadog.operation.name&#39;: &#39;web.request&#39;, &#39;datadog.span.top_level&#39;: True, &#39;datadog.span.type&#39;: &#39;web&#39;, &#39;span.kind&#39;: &#39;server&#39;, ...}.get

self = &lt;tests.parametric.test_otlp_trace_metrics.Test_FR06_Otel_Span_Attributes object at 0x7fd6d017f6e0&gt;
otlp_trace_metrics_library_env = {&#39;DD_SERVICE&#39;: &#39;test-otlp-stats-svc&#39;, &#39;OTEL_EXPORTER_OTLP_METRICS_ENDPOINT&#39;: &#39;http://ddapm-test-agent-worker-gw3-57403...:4318/v1/metrics&#39;, &#39;OTEL_EXPORTER_OTLP_METRICS_PROTOCOL&#39;: &#39;http/json&#39;, &#39;OTEL_TRACES_SPAN_METRICS_ENABLED&#39;: &#39;true&#39;, ...}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fd69f407380&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fd69fba78c0&gt;

...

View all failed tests

Testing the test | System Tests (nodejs, prod) / parametric / parametric (1)

🧪 4 Tests failed

tests.parametric.test_otlp_trace_metrics.Test_FR01_Enablement_Configuration.test_fr01_6_disabled_when_apm_tracing_disabled[library_env0, parametric-nodejs]

from system_tests_suite

AssertionError: OTLP trace metrics must be disabled when APM tracing is disabled, got: [&#39;traces.span.sdk.metrics.duration&#39;]
assert not [{&#39;attributes&#39;: [{&#39;key&#39;: &#39;span.name&#39;, &#39;value&#39;: {&#39;stringValue&#39;: &#39;web.request&#39;}}, {&#39;key&#39;: &#39;datadog.operation.name&#39;, &#39;val...ketCounts&#39;: [0, 1, 0, 0, 0, 0, ...], &#39;count&#39;: 1, &#39;explicitBounds&#39;: [0.002, 0.004, 0.006, 0.008, 0.01, 0.05, ...], ...}]
 &#43;  where [{&#39;attributes&#39;: [{&#39;key&#39;: &#39;span.name&#39;, &#39;value&#39;: {&#39;stringValue&#39;: &#39;web.request&#39;}}, {&#39;key&#39;: &#39;datadog.operation.name&#39;, &#39;val...ketCounts&#39;: [0, 1, 0, 0, 0, 0, ...], &#39;count&#39;: 1, &#39;explicitBounds&#39;: [0.002, 0.004, 0.006, 0.008, 0.01, 0.05, ...], ...}] = _duration_data_points([{&#39;resourceMetrics&#39;: [{&#39;resource&#39;: {&#39;attributes&#39;: [{...}, {...}, {...}, {...}, {...}, {...}, ...], &#39;droppedAttributesCount&#39;: 0}, &#39;scopeMetrics&#39;: [{&#39;metrics&#39;: [...]}]}]}])

self = &lt;tests.parametric.test_otlp_trace_metrics.Test_FR01_Enablement_Configuration object at 0x7fe228d2df70&gt;
otlp_trace_metrics_library_env = {&#39;DD_APM_TRACING_ENABLED&#39;: &#39;false&#39;, &#39;DD_SERVICE&#39;: &#39;test-otlp-stats-svc&#39;, &#39;OTEL_EXPORTER_OTLP_METRICS_ENDPOINT&#39;: &#39;http:...pm-test-agent-worker-gw4-5740354900026072187:4318/v1/metrics&#39;, &#39;OTEL_EXPORTER_OTLP_METRICS_PROTOCOL&#39;: &#39;http/json&#39;, ...}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fe1f8472fc0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fe1f801d280&gt;

    @pytest.mark.parametrize(
...

tests.parametric.test_otlp_trace_metrics.Test_FR06_Otel_Span_Attributes.test_fr06_2_span_kind[library_env0, parametric-nodejs]

from system_tests_suite

AssertionError: Expected span.kind in (&#39;SPAN_KIND_SERVER&#39;,), got attrs: {&#39;span.name&#39;: &#39;web.request&#39;, &#39;span.kind&#39;: &#39;server&#39;, &#39;datadog.operation.name&#39;: &#39;web.request&#39;, &#39;datadog.span.type&#39;: &#39;web&#39;, &#39;datadog.span.top_level&#39;: True}
assert &#39;server&#39; in (&#39;SPAN_KIND_SERVER&#39;,)
 &#43;  where &#39;server&#39; = &lt;built-in method get of dict object at 0x7fe1f926be40&gt;(&#39;span.kind&#39;)
 &#43;    where &lt;built-in method get of dict object at 0x7fe1f926be40&gt; = {&#39;datadog.operation.name&#39;: &#39;web.request&#39;, &#39;datadog.span.top_level&#39;: True, &#39;datadog.span.type&#39;: &#39;web&#39;, &#39;span.kind&#39;: &#39;server&#39;, ...}.get

self = &lt;tests.parametric.test_otlp_trace_metrics.Test_FR06_Otel_Span_Attributes object at 0x7fe228df5970&gt;
otlp_trace_metrics_library_env = {&#39;DD_SERVICE&#39;: &#39;test-otlp-stats-svc&#39;, &#39;OTEL_EXPORTER_OTLP_METRICS_ENDPOINT&#39;: &#39;http://ddapm-test-agent-worker-gw4-57403...:4318/v1/metrics&#39;, &#39;OTEL_EXPORTER_OTLP_METRICS_PROTOCOL&#39;: &#39;http/json&#39;, &#39;OTEL_TRACES_SPAN_METRICS_ENABLED&#39;: &#39;true&#39;, ...}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fe1f8472fc0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fe1f843f680&gt;

...

View all failed tests

Testing the test | System Tests (java, dev) / parametric / parametric (2)

🧪 7 Tests failed

tests.parametric.test_otlp_trace_metrics.Test_FR01_Enablement_Configuration.test_fr01_6_disabled_when_apm_tracing_disabled[library_env0, parametric-java]

from system_tests_suite

KeyError: &#39;resourceMetrics&#39;

self = &lt;tests.parametric.test_otlp_trace_metrics.Test_FR01_Enablement_Configuration object at 0x7fb2ee391f40&gt;
otlp_trace_metrics_library_env = {&#39;DD_APM_TRACING_ENABLED&#39;: &#39;false&#39;, &#39;DD_SERVICE&#39;: &#39;test-otlp-stats-svc&#39;, &#39;OTEL_EXPORTER_OTLP_METRICS_ENDPOINT&#39;: &#39;http:...est-agent-worker-gw2-5740354900026072187:4318/v1/metrics&#39;, &#39;OTEL_EXPORTER_OTLP_METRICS_PROTOCOL&#39;: &#39;http/protobuf&#39;, ...}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fb2be73f230&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fb2bdaef260&gt;

    @pytest.mark.parametrize(
        &#34;library_env&#34;,
        [{**DEFAULT_ENVVARS, &#34;DD_APM_TRACING_ENABLED&#34;: &#34;false&#34;}],
...

tests.parametric.test_otlp_trace_metrics.Test_FR06_Otel_Resource_Attributes.test_fr06_14_custom_service_on_data_point[library_env0, parametric-java]

from system_tests_suite

AssertionError: Expected test-otlp-stats-svc service.name on its data point: {&#39;postgres&#39;}
assert &#39;test-otlp-stats-svc&#39; in {&#39;postgres&#39;}

self = &lt;tests.parametric.test_otlp_trace_metrics.Test_FR06_Otel_Resource_Attributes object at 0x7f150dd8be30&gt;
otlp_trace_metrics_library_env = {&#39;DD_SERVICE&#39;: &#39;test-otlp-stats-svc&#39;, &#39;OTEL_EXPORTER_OTLP_METRICS_ENDPOINT&#39;: &#39;http://ddapm-test-agent-worker-gw4-57403...8/v1/metrics&#39;, &#39;OTEL_EXPORTER_OTLP_METRICS_PROTOCOL&#39;: &#39;http/protobuf&#39;, &#39;OTEL_TRACES_SPAN_METRICS_ENABLED&#39;: &#39;true&#39;, ...}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f150cab9ac0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f150ca3e150&gt;

    @pytest.mark.parametrize(&#34;library_env&#34;, [{**DEFAULT_ENVVARS}])
    def test_fr06_14_custom_service_on_data_point(
...

View all failed tests

View all 4 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 0 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: b48223e | Docs | Datadog PR Page | Give us feedback!}

…ty assertions Resolves the two open design questions from PR #6834 review with concrete decisions: service.name is now required on every data point, and OTel- semantics mode is restricted to exactly the Span Metrics Connector's default attribute set (service.name, span.name, span.kind, status.code). Also narrows ERROR_STATUS_VALUES and SPAN_KIND_SERVER_VALUES to the SMC's actual output format (STATUS_CODE_ERROR / SPAN_KIND_SERVER), based on reading the SMC connector source and traceutil.go directly rather than guessing at tolerant casing. Verified live against TEST_LIBRARY=python: 4 tests fail against the current Python implementation and are gated missing_feature in manifests/python.yml (fr01_6, fr06_2, fr06_8, fr09_2), in addition to the 3 already-known gates for the service.name/SMC-attrs requirements.

mabdinur changed the title ~~test(otlp-trace-metrics): address open review comments from PR #6834~~ test(otlp-trace-metrics): maintain parity with the OTEP spec Jul 23, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(otlp-trace-metrics): maintain parity with the OTEP spec - #7363

test(otlp-trace-metrics): maintain parity with the OTEP spec#7363
mabdinur wants to merge 2 commits into
mainfrom
munir/otlp-trace-metrics-address-review-comments

mabdinur commented Jul 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

datadog-datadog-us1-prod Bot commented Jul 22, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

mabdinur commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why draft

Test plan

Uh oh!

github-actions Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-us1-prod Bot commented Jul 22, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mabdinur commented Jul 22, 2026 •

edited

Loading

github-actions Bot commented Jul 22, 2026 •

edited

Loading

datadog-datadog-us1-prod Bot commented Jul 22, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading