Cover OTLP export status in startup logs [dotnet@brian.marks/otlp-export-startup-log][nodejs@brian.marks/otlp-export-startup-log][python@brian.marks/otlp-export-startup-log][ruby@brian.marks/otlp-export-startup-log] - #7376
Draft
bm1549 wants to merge 2 commits into
Conversation
Extend Test_Startup_Logs to assert the three OTLP-export booleans every tracer now publishes in its startup diagnostic configuration: otlp_traces_export_enabled, otlp_metrics_export_enabled, otlp_logs_export_enabled. Add a cross-language extraction helper that locates the config line by field name and reads each boolean with a quote-style- and casing-tolerant regex. This tolerates the per-tracer serialization differences: JSON with a " - " marker (Node.js, .NET, Ruby "CORE" section), JSON with no separator (Go, Java), and Python's dict repr (True/False, single quotes) which is not valid JSON. Requiring an unquoted true/false/True/False keeps the assertion that each field is a real boolean. New tests: - test_startup_logs_otlp_export_fields: all three present, boolean, and false by default (no OTLP env vars). - test_startup_logs_otlp_traces_export_enabled: OTEL_TRACES_EXPORTER=otlp. - test_startup_logs_otlp_metrics_export_enabled: DD_METRICS_OTEL_ENABLED=true (+ DD_RUNTIME_METRICS_ENABLED for .NET). - test_startup_logs_otlp_logs_export_enabled: DD_LOGS_OTEL_ENABLED=true. Manifest gating uses per-language placeholder next-version floors (the tracer changes are unreleased). Ruby's traces test is marked irrelevant (no OTLP trace exporter), PHP incomplete_test_app, Rust missing_feature, consistent with the existing startup-log entries. NOTE: version pins are placeholders that MUST be confirmed before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…am can't hang _get_startup_logs read the entire container stderr in one tail-less container.logs() call. If a tracer misbehaves and emits an ever-growing stderr stream (e.g. a logging feedback loop), that read chases a moving target and never returns, hanging the whole parametric suite. Make the non-.NET read bounded and defensive: - Stream the current stderr snapshot (stream=True, follow=False), which self-terminates at end-of-snapshot in normal operation, so behavior is unchanged when the tracer is well-behaved. - Cap the read by line count, byte count, and wall-clock time; the first cap reached stops the read and returns what was accumulated. This bounds both the runaway case and the no-marker "disabled" case. - Accumulate raw byte chunks (docker log frames, not lines) and decode once at the end so a line split across chunks is handled, using errors="replace" for a multi-byte char truncated at a cap boundary. - Close the stream in a finally so a mid-snapshot stop releases the socket. Read the whole bounded snapshot rather than early-exiting at the first startup-config marker: some tracers emit content the tests assert on after that marker (Ruby logs "CONFIGURATION - CORE" at init, then "CONFIGURATION - TRACING" and its "Agent Error" line only after a flush), so stopping at the first marker would truncate content the OTLP-field and diagnostic tests depend on. The startup logs are emitted early, so the caps never truncate them in normal runs. The .NET on-disk managed-log path and the stderr-only stream selection are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bm1549
added
the
ai-generated
Contributor
|
|
This was referenced Jul 24, 2026
|
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
to DataDog/dd-trace-py
that referenced
this pull request
Jul 27, 2026
## Description When `DD_LOGS_OTEL_ENABLED=true`, the OTLP logs pipeline attaches an OpenTelemetry `LoggingHandler` to the Python root logger. ddtrace's own loggers (`ddtrace.*`) and the OpenTelemetry exporter's loggers (`opentelemetry.*`) propagate to root, so the handler captured the tracer's own log records: the per-export debug line and the exporter's export-failure warnings/errors. Exporting those records emits more log records, which were captured and exported again. The result is a self-amplifying export loop. This happens whenever those records reach the effective log level: with `DD_TRACE_DEBUG=true` (the per-export debug line), or on export failures (warnings/errors at the default level). The fix attaches a self-telemetry exclusion filter to the OTLP `LoggingHandler` that drops records whose logger name is in the `ddtrace` or `opentelemetry` namespaces. Matching is exact-or-dotted-prefix (`ddtrace` / `ddtrace.*`, `opentelemetry` / `opentelemetry.*`), so application loggers such as `ddtrace_myapp` are not affected. The filter is attached only to the handler ddtrace itself installs; application-configured handlers are not modified. Application logs are still captured and exported. This was discovered during cross-tracer startup-log work. It is an independent, pre-existing bug, so it ships on its own branch. ## Testing Adds `tests/opentelemetry/test_logs.py::test_otel_logs_exporter_excludes_self_telemetry`. With `DD_LOGS_OTEL_ENABLED=true`, it emits a `ddtrace.*` record, an `opentelemetry.*` record, and an application record, flushes the exporter to a mock gRPC log service, and asserts the application record is exported while the two self-telemetry records are not. The test fails without the fix and passes with it. It runs across the OpenTelemetry versions already covered by the `opentelemetry` test venv. ## Risks Low. The change only adds a logging filter to the handler ddtrace creates. Records from the `ddtrace` and `opentelemetry` logger namespaces are no longer exported through this pipeline; application logs are unchanged. The filter binds to the handler instance, so it persists across `logging.basicConfig` reconfiguration, which is the reconfiguration API the OpenTelemetry SDK patches to re-add the same handler instance (`dictConfig`/`fileConfig` are not patched by the SDK). ## Additional Notes Includes a reno release note. No configuration or public API changes. --- ### Context Independent, pre-existing bug surfaced while adding OTLP export-status fields to the tracer startup log across dd-trace-* (the Python system-test for the logs field hung on this loop). Related: - Startup-log feature (dd-trace-py): #19265 - system-tests coverage that surfaced it: DataDog/system-tests#7376 Co-authored-by: brian.marks <brian.marks@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Every tracer now reports three booleans in its startup diagnostic configuration that describe whether traces, metrics, and logs are exported over OTLP:
otlp_traces_export_enabled,otlp_metrics_export_enabled, andotlp_logs_export_enabled. Nothing in system-tests covered them, so a regression in any tracer's startup diagnostic would go unnoticed.Changes
Adds four parametric methods to
Test_Startup_Logsintests/parametric/test_startup_logs.py:test_startup_logs_otlp_export_fields: with no OTLP env vars set, asserts all three fields are present, are real booleans, and default tofalse.test_startup_logs_otlp_traces_export_enabled: assertsotlp_traces_export_enabledistruewithOTEL_TRACES_EXPORTER=otlp.test_startup_logs_otlp_metrics_export_enabled: assertsotlp_metrics_export_enabledistruewithDD_METRICS_OTEL_ENABLED=true(plusDD_RUNTIME_METRICS_ENABLED=true, which .NET requires to flip the metrics field).test_startup_logs_otlp_logs_export_enabled: assertsotlp_logs_export_enabledistruewithDD_LOGS_OTEL_ENABLED=true.The startup diagnostic is not serialized the same way across tracers: JSON with a
-marker for Node.js, .NET, and Ruby; JSON with no separator for Go and Java; and a Python dict repr that is not valid JSON. A shared helper,_extract_otlp_export_fields, locates the config line by field name and reads each boolean with a quote-style- and casing-tolerant regex. It requires an unquoted boolean, so a string-encoded value would not match, which keeps the assertion that each field is emitted as a real boolean.This PR also hardens
_get_startup_logs. The previous one-shotcontainer.logs()read the whole stderr in a single call. If a tracer emits an unbounded stderr stream (for example a logging feedback loop), that read chases a moving target and can hang the parametric suite. The read now streams the current snapshot (follow=False) and stops at the first of a line, byte, or wall-clock cap, each set far above any real tracer's startup output. Behavior is unchanged for a well-behaved tracer, and it warns rather than truncating silently when a cap is hit.Manifest gating is added for the eight tracers that run these startup-log tests. Ruby's traces test is marked
irrelevant(Ruby has no OTLP trace exporter), PHPincomplete_test_app, and Rustmissing_feature, matching the existing startup-log entries.Before merge
[lang@branch]override, so they build from their default branch. Their new-field tests may be red until the corresponding tracer changes merge and the manifest versions are set.[lang@branch]annotations in the title must be removed before merge. The "Fail if target branch is specified" guard job intentionally fails while an override is present.Workflow
🚀 Once your PR is reviewed and the CI green, you can merge it!
🛟 #apm-shared-testing 🛟
Reviewer checklist
tests/ormanifests/is modified ? I have the approval from R&P teambuild-XXX-imagelabel is presentRelated PRs — cross-tracer OTLP startup-log effort