Mark all spans with apm billing disabled tag by IlyasShabi · Pull Request #7359 · DataDog/system-tests

IlyasShabi · 2026-07-22T13:18:11Z

Motivation

Adds system-test coverage for delayed trace chunks in standalone ASM, the test verifies that all spans have the tag _dd.apm.enabled: 0.

Changes

Add new test class

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-07-22T13:18:36Z

CODEOWNERS have been resolved as:

manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/appsec/test_asm_standalone.py                                     @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/utils.py                                                   @DataDog/asm-libraries @DataDog/system-tests-core

datadog-prod-us1-3 · 2026-07-22T13:20:00Z

Tests

⚠️ Warnings

🚦 2 Pipeline jobs failed

Testing the test | System Tests (java, dev) / End-to-end #1 / spring-boot-payara 1

Testing the test | all-jobs-are-green

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 6c984fd | Docs | Datadog PR Page | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 84c13538fc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-24T09:10:02Z

  : - component_version: "<3.11.0"
      declaration: missing_feature (APPSEC-57830 python tracer was using MANUAL_KEEP for 1 trace in 60 seconds to keep instead of AUTO_KEEP)
      weblog: *django
+  tests/appsec/test_asm_standalone.py::Test_AppSecStandalone_APMDisabledMarker: missing_feature


Enable the new marker test for supported libraries

This new test is added as missing_feature here, and the same blanket entry is added to every library manifest touched by the commit, so the APPSEC_STANDALONE CI matrix will never execute the new coverage. Several adjacent AppSec standalone classes are already enabled for released/dev versions in the same manifests, so this should use the appropriate version/weblog-specific activation instead of disabling the class everywhere.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-24T09:10:02Z

+    """Every span sent in standalone mode carries the APM-disabled billing marker."""
+
+    def setup_all_spans_have_apm_disabled_marker(self) -> None:
+        self.r = weblog.get("/")


Send an AppSec event before checking standalone spans

When this class is enabled for a tracer that honors DD_APM_TRACING_ENABLED=false, a plain GET / does not create a security or upstream-kept trace, so interfaces.library.get_traces(request=self.r) can be empty and the new assert spans fails without testing the marker. Use a request that actually generates an AppSec-retained trace, such as the Arachni/v1 attack header pattern already used in this file, because setup methods are the phase that must generate the captured traces for validation. .cursor/rules/end-to-end-testing.mdcL9-L9

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-24T09:10:02Z

+    def test_all_spans_have_apm_disabled_marker(self) -> None:
+        assert self.r.status_code == 200
+
+        spans = [span for _, trace in interfaces.library.get_traces(request=self.r) for span in trace]


Collect delayed chunks by trace ID

For delayed trace chunks, filtering with get_traces(request=self.r) only returns chunks where some span can be matched back to the request RID; delayed child/outbound chunks can share the trace ID but lack the request user-agent tag, so they are silently skipped here. After identifying the request trace, iterate all library traces with the same trace ID before asserting the billing marker, otherwise this can pass while the delayed chunk is missing _dd.apm.enabled.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-24T09:10:02Z

+        assert spans, "No spans were sent for the request"
+
+        for span in spans:
+            apm_enabled = span["metrics"].get("_dd.apm.enabled")


Assert the marker on chunk roots, not every span

When the retained request emits more than one span in a trace chunk, this asserts _dd.apm.enabled on child spans too, but the change is testing a trace-chunk billing marker and trace-level markers are attached to the chunk root/first span. This can make compliant tracers fail once the test uses a multi-span or delayed-chunk endpoint; iterate the chunks and validate trace[0] for each chunk instead.

Useful? React with 👍 / 👎.

avara1986 · 2026-07-24T09:14:37Z

+    def setup_all_spans_have_apm_disabled_marker(self) -> None:
+        self.r = weblog.get("/")
+
+    def test_all_spans_have_apm_disabled_marker(self) -> None:


This function should be moved to the top level and implemented in the standalone AI Guard tests as well

Done! I have a function that I can reuse on ai guard! I'll create a follow-up for ai guard once this merged

avara1986 · 2026-07-24T09:15:14Z

    - weblog_declaration:
        nextjs: missing_feature (endpoint not implemented)
  tests/appsec/test_asm_standalone.py::Test_APISecurityStandalone: *ref_5_52_0
+  tests/appsec/test_asm_standalone.py::Test_AppSecStandalone_APMDisabledMarker: missing_feature


Hasn't any tracer implemented this feature?

Probably Go, I'd like too keep it missing feature and let lib owners make the changes

avara1986 · 2026-07-24T09:55:31Z

+    """Every span sent in standalone mode carries the APM-disabled billing marker."""
+
+    def setup_all_spans_have_apm_disabled_marker(self) -> None:
+        self.r = weblog.get("/")


Maybe we should create a new endpoint that generates nested spans, or makes an HTTP call, to ensure there are multiple spans and that they are all marked correctly

My initial implementation attempted to add nested calls, but this could have different behaviour for async frameworks! well we can do it with the newest test version. probably on a follow-up

…led (#19242) APPSEC-69365 ## Summary The APM tracing disablement metric `_dd.apm.enabled:0` was only being set on the **local root span**, not on all service entry (top-level) spans. This mirrors a mistake previously made for AppSec event tagging. When APM tracing is disabled and a trace contains an inferred/child service — e.g. a sibling service entry span created for an outgoing HTTP client call — that span (`_dd.top_level:1`) was left untagged. As a result the intake metered those inferred services as having APM enabled. Reported in a customer support case (found in java and nodejs first). Also tested with DataDog/system-tests#7359 ## Change In `TraceSamplingProcessor.process_trace`, when `apm_opt_out` is set, tag every span in the trace chunk with `_dd.apm.enabled:0` instead of only the local root span. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: florentinl <florentin.labelle@datadoghq.com>

Julio-Guerra requested changes Jul 22, 2026

View reviewed changes

Comment thread docs/understand/weblogs/end-to-end_weblog.md Outdated

IlyasShabi force-pushed the ishabi/appsec-standalone-trace-chunk-billing branch 2 times, most recently from 7fbe28c to e674e2c Compare July 23, 2026 11:58

IlyasShabi requested a review from Julio-Guerra July 23, 2026 11:58

IlyasShabi force-pushed the ishabi/appsec-standalone-trace-chunk-billing branch from e674e2c to 42003de Compare July 23, 2026 12:00

IlyasShabi mentioned this pull request Jul 23, 2026

fix(standalone): stamp _dd.apm.enabled on every span DataDog/dd-trace-js#9506

Merged

christophe-papazian mentioned this pull request Jul 23, 2026

fix(tracing): tag all spans as APM-disabled when APM tracing is disabled DataDog/dd-trace-py#19242

Merged

christophe-papazian approved these changes Jul 23, 2026

View reviewed changes

IlyasShabi marked this pull request as ready for review July 24, 2026 09:01

IlyasShabi requested review from a team as code owners July 24, 2026 09:01

IlyasShabi requested review from P403n1x87 and vlad-scherbich and removed request for a team July 24, 2026 09:01

IlyasShabi requested review from Anilm3, avara1986, claponcet, dougqh and dromanol and removed request for a team July 24, 2026 09:01

claponcet approved these changes Jul 24, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 24, 2026

View reviewed changes

avara1986 reviewed Jul 24, 2026

View reviewed changes

nccatoni approved these changes Jul 24, 2026

View reviewed changes

dougqh approved these changes Jul 24, 2026

View reviewed changes

IlyasShabi added 5 commits July 27, 2026 11:13

Add trace chunk billing marker for appsec standalone

0a7a000

check at least one span

65682d7

remove uneccessary files

0213e71

fix linter

02bd2f7

make tests simpler

240f52a

IlyasShabi force-pushed the ishabi/appsec-standalone-trace-chunk-billing branch from 9fc2201 to 240f52a Compare July 27, 2026 09:14

Merge branch 'main' into ishabi/appsec-standalone-trace-chunk-billing

870a603

IlyasShabi changed the title ~~Add trace chunk billing marker for appsec standalone~~ Mark all spans with apm billing disabled tag Jul 28, 2026

IlyasShabi requested a review from avara1986 July 28, 2026 07:34

avara1986 approved these changes Jul 28, 2026

View reviewed changes

Merge branch 'main' into ishabi/appsec-standalone-trace-chunk-billing

6c984fd

Uh oh!

Conversation

IlyasShabi commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-prod-us1-3 Bot commented Jul 22, 2026 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

avara1986 Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

IlyasShabi Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

avara1986 Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

IlyasShabi Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

avara1986 Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

IlyasShabi Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

IlyasShabi commented Jul 22, 2026 •

edited

Loading

github-actions Bot commented Jul 22, 2026 •

edited

Loading

datadog-prod-us1-3 Bot commented Jul 22, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading