core: report cron and indexer job failures to Sentry with health metrics

[EmeditWeb]

Problem

No background job reports its failures to anything except logs. grep -rn "captureException" src/jobs/ src/indexer/ returns zero matches, despite Sentry being installed and initialized (@sentry/nestjs, added 2026-06-19 per the tracker).

The affected jobs are the ones the protocol depends on to stay consistent with the chain:

• src/indexer/indexer.service.ts, @Cron('*/60 * * * * *')
• src/jobs/transaction-status-checker/transaction-status-checker.service.ts, @Cron('*/120 * * * * *')
• src/jobs/loan-payment-reminder/loan-payment-reminder.service.ts, @Cron('0 9 * * *')
• src/jobs/nonce-cleanup/nonce-cleanup.service.ts, @Cron(CronExpression.EVERY_HOUR)
• src/jobs/supabase-keepalive/supabase-keepalive.service.ts

context/code-standards.md requires every cron to catch its own errors and log them, and they do. Logging is where it stops. On Render, a job that throws every 60 seconds produces log lines nobody is watching and no alert anywhere.

The consequences are concrete. If the indexer stalls, on-chain events stop reaching the database and the API serves stale loan and pool state indefinitely. If the transaction status checker stalls, submitted transactions never move out of pending, so users see confirmed on-chain payments stuck as unconfirmed in the app. Both fail silently today.

This is about alerting on job health, not about reconciliation logic. On-chain to database reconciliation is #80 and default detection is #71. Do not duplicate either.

Ground Rules

This issue must be solved according to StepFi's established engineering standards. Before writing any code:

1. Read context/architecture-context.md in full, especially the "no BullMQ" decision. Do not reintroduce a queue library to solve this
2. Read context/code-standards.md in full, especially the Background jobs section
3. Read context/progress-tracker.md to understand what has already been built and why, so you do not duplicate or contradict recent work
4. Read all five job services listed above to see the existing isRunning guard and try/catch structure before changing them
5. Read src/modules/metrics/metrics.service.ts and metrics.updater.ts, since Prometheus is already wired and is the right place for job health gauges
6. Read src/indexer/indexer-status.service.ts, which already tracks indexer state and may already expose part of what is needed

Your PR will be rejected regardless of whether CI passes if it conflicts with anything in these files, introduces patterns inconsistent with what already exists, or duplicates functionality that was solved differently elsewhere in the codebase.

What To Build

1. Create a shared JobMonitorService in src/jobs/monitoring/. Do not copy Sentry calls into five separate catch blocks. Every job should report through one place.
2. Expose recordSuccess(jobName) and recordFailure(jobName, error). recordFailure calls Sentry.captureException with the job name as a tag and the consecutive failure count as context, and logs at error level.
3. Track consecutive failures per job in memory. Send to Sentry on the first failure and then on an escalating interval rather than on every tick. A job failing every 60 seconds must not generate 1,440 Sentry events per day, or the alert becomes noise and gets muted, which is the same as having no alert.
4. Add Prometheus gauges via the existing metrics module: last successful run timestamp per job, and consecutive failure count per job. These are what an external alert rule can actually watch.
5. Wire all five job services to call recordSuccess and recordFailure. Keep every existing isRunning guard and try/catch exactly as it is. This adds reporting, it does not restructure the jobs.
6. Add a staleness check to the health module: if any job's last success is older than a configured threshold, GET /health reports degraded. Make thresholds configurable per job through src/config/env.ts, since a 60-second indexer and a daily reminder job cannot share one threshold.
7. Do not swallow anything new. If a job currently rethrows, keep that behavior.

Files To Touch

• src/jobs/monitoring/job-monitor.service.ts (new)
• src/jobs/monitoring/job-monitor.module.ts (new)
• src/jobs/monitoring/job-monitor.service.spec.ts (new)
• src/indexer/indexer.service.ts
• src/jobs/transaction-status-checker/transaction-status-checker.service.ts
• src/jobs/loan-payment-reminder/loan-payment-reminder.service.ts
• src/jobs/nonce-cleanup/nonce-cleanup.service.ts
• src/jobs/supabase-keepalive/supabase-keepalive.service.ts
• src/modules/metrics/metrics.service.ts
• src/modules/health/health.service.ts
• src/config/env.ts

Acceptance Criteria

• A throwing job produces a Sentry event tagged with the job name
• Repeated failures do not produce one Sentry event per tick; the escalation interval is covered by a test
• Prometheus exposes last-success timestamp and consecutive-failure count for all five jobs
• GET /health reports degraded when a job's last success exceeds its configured threshold
• Thresholds are configurable per job and validated at startup
• Every existing isRunning guard and try/catch is preserved unchanged
• No queue library was introduced
• All state-changing operations validated and safe against invalid input

Mandatory Checks Before Opening PR

• All context/ files read and understood
• Code follows context/code-standards.md exactly
• context/progress-tracker.md updated with an accurate entry describing the change and why
• Build passes with zero errors (npm run build)
• All existing tests still pass, test count has not decreased
• New tests written covering the new functionality
• No new any types
• Full Swagger decorators on any new or changed endpoint
• PR template filled out completely, including the "problem this solves" and "how it was tested" sections
• PR references this issue number exactly

PRs that fail any check above will be closed without review, regardless of how much work went into them. No exceptions. This is a Grantfox-funded, quality-first project. Code that does not meet these standards will not be merged and will not be paid.

[coderolisa]

Hello Maintainer,

I'd like to work on this issue. I've reviewed the requirements and can complete the implementation as specified.

Please assign it to me. Thank you!

[Ebuka321]

Good day. Please can i work on these issue.

[BigDella]

i will love to work on this issue, i have gone through the requirement and i understood it

[grantfox-oss]

🦊 GrantFox — @BigDella has been assigned to this issue as part of the Official Campaign | FWC26 campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #99)
2. Your PR will be reviewed by the StepFi-app maintainers

Good luck! Track your progress on GrantFox.