Skip to content

Add V3 streaming event ingestion pipeline#2370

Draft
ejsmith wants to merge 11 commits into
mainfrom
feature/v3-event-ingestion
Draft

Add V3 streaming event ingestion pipeline#2370
ejsmith wants to merge 11 commits into
mainfrom
feature/v3-event-ingestion

Conversation

@ejsmith

@ejsmith ejsmith commented Jul 13, 2026

Copy link
Copy Markdown
Member

Summary

  • add controller-free V3 Minimal API routes that stream top-level JSON values directly from HttpRequest.BodyReader using source-generated System.Text.Json metadata
  • move raw stack parsing and fingerprint generation to the server, with .NET, Java, JavaScript, Python, caused-by, manual-stacking, and normalized fallback support
  • resolve discarded stacks before quota reservation/materialization so discarded events take the cheapest path and are not charged
  • add distributed atomic quota reservation, deterministic event IDs, inline bulk persistence, ambiguous-write reconciliation, and a durable asynchronous side-effect work item
  • make stack statistics, notifications/webhooks, and post-save plugin work replay-safe with stable identities and per-event completion markers
  • add bounded microbatches, concurrency/backpressure, cancellation, gzip/Brotli support, compressed/decompressed/event/request limits, ProblemDetails, observability, and disabled-by-default allowlist rollout controls
  • publish an isolated V3 OpenAPI document, HTTP samples, architecture/rollout documentation, BenchmarkDotNet suites, and a repeatable streaming load harness
  • pin Microsoft.OpenApi 2.7.5 to remediate CVE-2026-49451 discovered during dependency validation

API and compatibility

V3 is a new breaking-contract surface:

  • POST /api/v3/events
  • POST /api/v3/projects/{projectId}/events
  • Content-Type: application/x-ndjson
  • one complete top-level JSON event per stream item; top-level arrays are rejected

The V2 controller, payload formats, queued handoff, and response contract remain available and unchanged. V3 is disabled by default and can be rolled out by project or organization allowlist.

Verification

  • dotnet build Exceptionless.slnx --no-restore --disable-build-servers -m:1 -v:minimal — 0 warnings, 0 errors
  • full backend suite — 2,350 passed, 0 failed, 2 intentional performance fixtures skipped
  • focused V3 persistence/replay/outbox suite — 26 passed
  • focused ingestion/compatibility suite — 85 passed
  • BenchmarkDotNet dry run — all 10 ingestion benchmarks executed
  • NuGet transitive vulnerability audit — no vulnerable packages in all eight projects
  • git diff --check — clean

Rollout

Keep this PR in draft while reviewing the architecture and running the documented one-, two-, four-, and eight-instance load matrix against the intended Redis/Elasticsearch topology. Production activation remains a configuration-only, reversible step.

Closes #2368

@ejsmith
ejsmith force-pushed the feature/v3-event-ingestion branch from 1a60290 to b3ce174 Compare July 13, 2026 22:48
@github-actions

Copy link
Copy Markdown

Code Coverage

Package Line Rate Branch Rate Complexity Health
Exceptionless.AppHost 38% 39% 144
Exceptionless.Core 74% 66% 10512
Exceptionless.Insulation 45% 38% 308
Exceptionless.Web 84% 68% 7227
Summary 77% (24625 / 31978) 66% (11352 / 17227) 18191

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build a V3 streaming inline event ingestion pipeline

2 participants