Skip to content

Add robust NaN loss diagnostics#69

Draft
nightlessbaron wants to merge 2 commits into
prodfrom
nan-loss-diag
Draft

Add robust NaN loss diagnostics#69
nightlessbaron wants to merge 2 commits into
prodfrom
nan-loss-diag

Conversation

@nightlessbaron

Copy link
Copy Markdown

What changed

  • Add per-tensor finite/nonfinite diagnostics for policy loss inputs and outputs.
  • Make empty and all-nonfinite tensors safe, including rank-asymmetric distributed reductions.
  • Aggregate diagnostic minima, maxima, and counts with explicit distributed semantics.
  • Keep scalar diagnostics out of token/domain reducers.
  • Reuse the ratio already computed by policy loss and remove the full-vocabulary logits scan.
  • Rate-limit long-batch warnings.
  • Add 15 focused regression tests.

Why

NaN/Inf incidents in long-context policy training were difficult to localize. The original diagnostic branch also had correctness hazards around scalar TIS metrics, empty shards, context-parallel aggregation, and duplicate ratio computation.

Validation

  • 15 passed in test_loss_nan_diagnostics.py using the Miles environment with heavyweight unrelated fixtures excluded.
  • Ruff: passed.
  • Black: passed.
  • py_compile: passed.
  • git diff --check: passed.

Merge risk

Diagnostics are currently unconditional. They add several tensor-to-host synchronizations per policy microbatch and three small distributed collectives per train step. Benchmark on the target long-context workload, or add a sampling/config gate, before marking this ready.

@odp
odp force-pushed the nan-loss-diag branch from 044b1a0 to ffce7d9 Compare July 16, 2026 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants