Skip to content

feat(observability): collect process system metrics - #24

Draft
WenYuLuo wants to merge 1 commit into
inclusionAI:mainfrom
WenYuLuo:feat/otel-process-metrics
Draft

feat(observability): collect process system metrics#24
WenYuLuo wants to merge 1 commit into
inclusionAI:mainfrom
WenYuLuo:feat/otel-process-metrics

Conversation

@WenYuLuo

@WenYuLuo WenYuLuo commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

What changed

  • add an OpenTelemetry hostmetrics/process pipeline for long-lived AKernel and YuanRong processes
  • collect CPU time/utilization, RSS/virtual memory, memory utilization, open file descriptors, thread count, and uptime every 15 seconds
  • attach environment, role, pod, namespace, node, process, and normalized component resource attributes before Prometheus remote write
  • normalize the truncated DataSystem process name to component_name="ds_worker"
  • distinguish goruntime as frontend or scheduler from its stable runtime ID, then remove process.command_line before export
  • pass pod and node identity into the collector for master, frontend, node, and standalone deployments
  • provision an AKernel Process Resources Grafana dashboard with environment, node, Pod, role, and component filters

Dashboard

The dashboard provides six panels:

  • CPU utilization by component
  • resident memory by component
  • open file descriptors by component
  • thread count by component
  • process uptime
  • current resident-memory snapshot

Why

The collector previously accepted metrics pushed by applications but did not report operating-system resource usage for individual platform components. This left CPU, memory, file descriptor, thread, and uptime visibility unavailable at the component and node level.

The process allowlist intentionally excludes per-sandbox runsc processes to avoid unbounded time-series cardinality.

Validation

  • validated both collector configurations with otel/opentelemetry-collector-contrib:0.120.0
  • verified remote write end to end with shared-PID-namespace frontend, scheduler, and DataSystem test processes; Prometheus received frontend, scheduler, and ds_worker, with no command-line label exported
  • parsed every dashboard JSON and all six new PromQL expressions with Prometheus 3.5.0 promtool
  • provisioned the dashboard into Grafana 11.5.2 and verified its UID, title, and six panels through the Grafana API
  • make SHELL=/bin/bash deploy-script-check
  • Helm 3.18.4 lint and template rendering for the core and monitor charts
  • git diff --check

@WenYuLuo
WenYuLuo force-pushed the feat/otel-process-metrics branch 2 times, most recently from ee08a9c to 7ad2f77 Compare August 13, 2026 03:25
Add a hostmetrics process pipeline for long-lived AKernel and YuanRong
components. Export CPU, memory, file descriptor, thread, and uptime
metrics with pod, node, role, and normalized component labels.

Classify goruntime processes as frontend or scheduler by runtime ID.
Delete command-line attributes before remote write so arguments are not
exported.

Provision a Grafana dashboard for process CPU, memory, file descriptors,
threads, and uptime with node, pod, role, and component filters.

Signed-off-by: robbluo <luo1442@gmail.com>
@WenYuLuo
WenYuLuo force-pushed the feat/otel-process-metrics branch from 7ad2f77 to 8a8e323 Compare August 13, 2026 03:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant