Skip to content

[codex] report background job failures and health#103

Draft
BigDella wants to merge 1 commit into
StepFi-app:mainfrom
BigDella:codex/issue-99-job-monitoring
Draft

[codex] report background job failures and health#103
BigDella wants to merge 1 commit into
StepFi-app:mainfrom
BigDella:codex/issue-99-job-monitoring

Conversation

@BigDella

Copy link
Copy Markdown

Summary

Closes #99

Background jobs currently stop at local log output when they fail, so an indexer or transaction checker outage can silently leave API state stale. This PR adds one shared monitoring path for all five protocol jobs: successful runs update Prometheus, failed runs update a consecutive-failure gauge and report to Sentry at power-of-two intervals, and the health endpoint degrades when any job exceeds its configured staleness threshold.

This repo is for the NestJS backend API only

  • My changes are inside src/ or test/ (plus .env.example and the required progress tracker entry)
  • I have NOT added React, React Native, or frontend component files
  • I have NOT added Rust or Soroban contract code
  • This is NestJS/TypeScript backend work

Type of change

  • Bug fix
  • New endpoint
  • New service or module
  • Database migration
  • Background job
  • Test coverage

What changed

  • Added global JobMonitorService with recordSuccess and recordFailure.
  • Added Sentry job tags and consecutive-failure context, captured on failures 1, 2, 4, 8, and so on.
  • Added job_last_success_timestamp_seconds and job_consecutive_failures Prometheus gauges.
  • Wired the indexer, transaction checker, reminder, nonce cleanup, and Supabase keep-alive without removing their existing try/catch or overlap behavior.
  • Added validated, per-job staleness thresholds and surfaced job details in GET /health.
  • Updated the progress tracker and .env.example.

Testing

  • npm run build passes with zero TypeScript errors
  • npm test -- --runInBand passes — 29 suites, 322 tests
  • No new any types introduced anywhere
  • Swagger decorators are not applicable because no endpoint contract changed
  • No migration is required
  • New unit tests cover Sentry escalation, metric resets, threshold validation, and stale health state
  • Targeted ESLint passes for every changed TypeScript file

Context files reviewed

  • context/architecture-context.md
  • context/code-standards.md
  • context/progress-tracker.md updated

Problem this solves

Cron and indexer failures are now visible to both incident alerting and metric-based health monitoring. Power-of-two escalation prevents a 60-second failing job from flooding Sentry, while independent thresholds avoid applying indexer timing assumptions to daily or multi-day jobs.

How it was tested

  • npm run build
  • npm test -- --runInBand (29 suites, 322 tests)
  • Targeted ESLint across all changed TypeScript files
  • git diff --cached --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

core: report cron and indexer job failures to Sentry with health metrics

1 participant