Verify standalone billing scenarios run infra-less - #7387
Conversation
Standalone billing scenarios assert APM is opted out, but nothing checked that the agent stopped reporting infrastructure data, so the host could still be billed as an infra host. Run APPSEC_STANDALONE, IAST_STANDALONE, SCA_STANDALONE and AI_GUARD_STANDALONE with DD_INFRASTRUCTURE_MODE=none on the agent, and add a shared test asserting no infra metric namespace reaches the backend. The same assertion is applied to the two existing APPSEC_APM_STANDALONE scenarios, which had no infra verification. APPSEC-69411 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
|
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 760acea | Docs | Datadog PR Page | Give us feedback! |
test_telemetry_smoke fails on every PHP weblog variant in both infra-less scenarios: the agent forwards generate-metrics telemetry, but no waf.* series is in it. Both scenarios already ran infra-less before this branch, so this is not a regression from the infra-disabled coverage. APPSEC-69425 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ea8327ffe5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| self.agent_has_flushed_metrics = interfaces.agent.wait_for( | ||
| _is_metrics_payload, timeout=AGENT_METRICS_FLUSH_TIMEOUT | ||
| ) |
There was a problem hiding this comment.
This setup_* method is attached to end-to-end scenarios and now performs an explicit interfaces.agent.wait_for(..., timeout=60) before pytest reaches the normal post-setup collection phase. When the agent does not flush promptly, every selected standalone scenario can spend up to a minute blocked in setup; the repository review guidance says longer waits should be expressed through scenario-level agent_interface_timeout/library_interface_timeout instead of explicit waits in setup methods. .cursor/rules/pr-review.mdcL32-L35
Useful? React with 👍 / 👎.
| """SCA standalone billing does not report infrastructure data.""" | ||
|
|
||
|
|
||
| @features.ai_guard_standalone |
There was a problem hiding this comment.
This class is tagged as ai_guard_standalone even though the test only observes the agent's metric flush and never exercises the AI Guard SDK; for libraries whose manifests mark the AI Guard standalone SDK tests as missing_feature (for example dotnet/php), this new nodeid has no matching manifest entry and can still pass, creating a false passing AI Guard standalone result. Either gate it like the existing product tests or mark the infra-only assertion as not reported. .cursor/rules/repository-structure.mdcL83-L86
Useful? React with 👍 / 👎.
2 participants
Motivation
Jira: APPSEC-69411 (epic APPSEC-61460)
The AI Guard / AppSec / IAST / SCA standalone scenarios assert that APM billing is opted out (
DD_APM_TRACING_ENABLED=false,_dd.apm.enabled=0, no client-computed stats). That is only half of the bill: the host can still be billed as an infra host if the agent keeps shipping infrastructure data.The agent side of that is APM Standalone Mode,
infrastructure_mode: "none"(spec — "This host should not be billed as an Infra Host"). Today it is set onAPPSEC_APM_STANDALONEandAPPSEC_STANDALONE_APM_STANDALONEonly, and nothing asserts the outcome:tests/appsec/smoke_tests/only checks that WAF / RASP / telemetry / RC / API security / user events still reach the backend with the flag on.Measured behaviour (
datadog/agent7.77.3)system.cpu.*,system.mem.*,system.disk.*,system.io.*,system.load.*,system.swap.*,system.fs.*,system.uptimeDD_INFRASTRUCTURE_MODE=nonesystem.*; onlydatadog.agent.*,datadog.dogstatsd.*,datadog.trace_agent.*Host metadata on
/intake/(including the processresourcessnapshot) is still sent in this mode, so the metric series are the right signal to assert on.Changes
utils/_context/_scenarios/__init__.py:agent_env={"DD_INFRASTRUCTURE_MODE": "none"}onAPPSEC_STANDALONE,IAST_STANDALONE,SCA_STANDALONE,AI_GUARD_STANDALONE. No new scenarios, so no extra CI cost.tests/test_infra_disabled.py(new):BaseInfraDisabledwaits for a real agent metrics flush (interfaces.agent.wait_for, so the check cannot pass vacuously) and asserts no series matchessystem./container./docker./kubernetes./process./ntp.. Bound to the four scenarios above plus the two existing*_APM_STANDALONEones.manifests/agent.yml: gated<7.77.0-0as irrelevant, matching the existing APM standalone smoke tests.Tests
./run.sh <SCENARIO> -k InfraDisabled), python/flask-poc.DD_INFRASTRUCTURE_MODE=nonetemporarily removed fromAI_GUARD_STANDALONE, the test fails with['ntp.offset', 'system.load.1', 'system.load.15', ...].IAST_STANDALONE,SCA_STANDALONE,AI_GUARD_STANDALONEsuites pass.APPSEC_STANDALONEhas 18 failures locally that are pre-existing — a baseline run without this change produces the identical 18 (localdd-trace-py4.12.0-rc1 weblog)../run.sh TEST_THE_TEST: 354 passed../format.sh: mypy and ruff clean.🤖 Generated with Claude Code