[STAC-25236] Hardened RHEL 9.8 nodes do not report network data from process-agent

[kkaempf]

STAC-25236

Issue Description

Customer reported that SUSE Observability cannot gather network-related data from hardened RHEL 9.8 nodes even when FIPS mode is off. This appears separate from the server-side FIPS compatibility work in STAC-25127 Resolved .

Support case: 01845035

Current Assessment

The issue is likely in the node process-agent network tracer path. The closest historical match is STAC-22092 Closed , where process-agent failed on Rocky/RHEL 9-style 5.14 kernels with:

failed to create network tracer: error loading prebuilt tracer: error guessing offsets: overflow while guessing sk field on struct sk_buff, bailing out

That ticket was closed as Cannot Reproduce, not fixed. A newer related issue, STAC-22643 Closed , fixed a separate memlock / FnMapLookupElem startup failure, but the current imported DataDog fork already contains that rlimit.RemoveMemlock() fix, so it does not look like the likely cause for RHEL 9.8.

Findings

• No Jira issue was found for support case 01845035 or the reported customer wording.
• STAC-25127 Resolved is still in Merging and covers server FIPS compatibility. It should not be treated as proof that the hardened-node network-data issue is fixed.
• Kafka/ZooKeeper FIPS failures from STAC-25127 Resolved are fixed in docker-images PR STAC-20704: Always send increment to make sure deletes have low latency #155 and consumed by helm-charts-internal PR STAC-13308 Protocol inspection envvar #81 in the 2.10.3 pre-release chart stream.
• The original STAC-25127 Resolved Jira comment also mentioned Elasticsearch failing with missing libnssadapter.so; I did not find an Elasticsearch-specific STAC-25127/libnssadapter fix or Beest coverage for Elasticsearch in the FIPS Java probe.
• Current stackstate-process-agent explicitly disables CO-RE and runtime compilation:
  • EnableCORE: false
  • EnableRuntimeCompiler: false
  • AllowPrebuiltFallback: true
  • BTFPath: ""
• The imported DataDog network tracer then takes the prebuilt eBPF path and runs offset guessing. On hardened/custom RHEL 9.8 kernels this can plausibly fail in the same way as STAC-22092 Closed .
• Datadog/agent-side notes indicate prebuilt eBPF is deprecated in favor of CO-RE/runtime compilation on Linux 6+ and RHEL 5.14+ kernels, which is exactly the RHEL 9 kernel family.

Evidence Links

Related Jira tickets:

• STAC-25127 Resolved - Fix FIPS compatibility
• STAC-22092 Closed - Agent fails to start on Rocky linux (part 2); exact sk_buff offset-guessing failure; closed Cannot Reproduce
• STAC-21765 Resolved - Process agent does not start on Rockylinux; related earlier customer report
• STAC-22643 Closed - Process agent fails to start due to FnMapLookupElem; fixed memlock/helper-probe issue
• STAC-23356 To Bulk - EBPF conntrack expires ebpf side; related eBPF/network data technical debt, but not the startup failure

Related PRs:

• https://github.com/StackVista/docker-images/pull/155 - STAC-25127 Resolved Kafka/ZooKeeper FIPS image fix
• https://github.com/StackVista/helm-charts-internal/pull/81 - chart image bump consuming fixed Kafka/ZooKeeper images

Relevant code paths:

• repos/stackstate-process-agent/config/tracer_config.go - disables CO-RE/runtime compilation and forces prebuilt fallback
• repos/stackstate-process-agent/checks/net_linux.go - process-agent fails startup if local tracer init fails
• github.com/StackVista/datadog-agent-upstream-for-process-agent/pkg/network/tracer/connection/kprobe/tracer.go - prebuilt tracer path calls offset guessing and returns the same failure shape as STAC-22092 Closed

Suggested Next Steps

1. Request customer process-agent logs for case 01845035, including the first network tracer startup failure.
2. Capture node details: uname -a, /etc/os-release, kernel package build, whether /sys/kernel/btf/vmlinux exists, SELinux mode, kernel lockdown mode, and relevant BPF sysctls.
3. Reproduce on hardened RHEL 9.8 with FIPS off.
4. Investigate enabling CO-RE or runtime compilation for process-agent on RHEL 9.x, or provide a tested fallback that avoids fragile prebuilt offset guessing.
5. Keep STAC-25127 Resolved scoped to server FIPS unless this ticket proves a direct shared root cause.

Customer Impact

Customer loses network-related data in SUSE Observability on hardened RHEL 9.8 nodes. Disabling network tracing avoids process-agent startup failure but removes the customer-visible network data they need.

---

Created from Louis/Codex investigation on 2026-07-06.