fix: Optimize cleanup jobs, eliminate raw ES client usage, add test coverage

[niemyjski]

Summary

Comprehensive optimization of cleanup jobs: eliminate direct Elasticsearch client usage from jobs, fix critical bugs, preserve cleanup correctness under concurrent ingestion, and add full integration test coverage.

Critical Bug Fixes

@min_count:2 → @min:2 in GetDuplicateSignaturesAsync

The @min_count syntax is silently ignored by Foundatio Parsers, causing the aggregation to return ALL signature hashes (not just duplicates). This would have caused FixDuplicateStacksAsync to treat every stack as a duplicate on the next run.

FixDuplicateStacksAsync only ran one batch (regression from refactor)

The original code looped until GetDuplicateSignaturesAsync returned empty. The refactor broke this. Restored the loop with:

• Batch tracking and per-batch logging
• Infinite-loop guard: exits if an entire batch fails to process
• ImmediateConsistency() on CountAsync inside GetDuplicateSignaturesAsync (one refresh per batch, matching the original Indices.RefreshAsync call pattern — NOT per item)

ReassignStackAsync data-loss hazard on empty sequence

If sourceStackIds was empty, PatchAllAsync would have no stack filter and would reassign ALL events to the target stack. Added materialization + early return guard.

FixDuplicateStacksAsync event-first ordering

Event reassignment completes before a duplicate stack is finalized as a redirect tombstone. If event reassignment fails, duplicate stacks remain visible and no data is lost.

GetDistinctFieldValuesAsync cursor correctness

Distinct event identifiers use composite aggregation pagination with an explicit next-value cursor so callers reliably detect the end of pagination.

Architecture (eliminate raw ES client from jobs)

• Refactored CleanupOrphanedDataJob to use repository methods exclusively
• Added GetDistinctFieldValuesAsync using composite aggregation (encapsulated in the repository — composite aggregation is not in Foundatio's DSL, so raw client use is justified and documented)
• Added RemoveAllByProjectIdsAsync/RemoveAllByOrganizationIdsAsync and RemoveAllByStackIdsAsync to IEventRepository
• Added ReassignStackAsync using a parameterized Painless script and strict post-write verification
• Added GetDuplicateSignaturesAsync to IStackRepository

Concurrency and lossless reconciliation

• Added durable duplicate-stack redirects with canonical target resolution, cycle detection, and late-event reconciliation
• Added an idempotent contribution ledger so retrying duplicate merges cannot double-count occurrences, including transitive redirect chains
• Added optimistic concurrency to Stack persistence so stale full-document saves cannot overwrite cleanup merges
• Replaced full-document writes in hot update paths with retrying repository patches for tags, status, regression state, and event counters
• Kept internal redirect/version fields out of the public JSON contract while persisting them in Elasticsearch
• Treat missing event indexes as empty only when Elasticsearch explicitly reports index_not_found_exception; other shard/index verification failures fail closed

Other Fixes

• Added OperationCanceledException handling before the intentionally broad per-signature catch
• Added lock renewal at page boundaries and while awaiting long-running reassignment operations
• Removed redundant is_deleted:false filters where repository soft-delete filtering applies
• Kept page size at 5 for organization/project cleanup (2.5s sleep/item makes larger pages impractical)
• Aligned async method and test names with repository conventions
• Added :{Message} to error log format strings

Merged onto current main

Merged the latest origin/main with a normal merge commit and kept the final diff limited to the intended cleanup, repository, serialization, and test files.

Test Coverage

Coverage includes:

• Cleanup pagination, retention, usage accounting, and tenant isolation
• Orphaned stack/project/organization events, including large and multi-page datasets
• Duplicate signature lookup, soft-delete filtering, empty-source reassignment, and closed/missing event indexes
• Partial failure and retry idempotency, redirect chains/cycles, late events/counters, and stale saves
• Concurrent tag, status, regression, and event-counter updates
• Public serialization and API compatibility for internal persistence fields

[Copilot]

Pull request overview

This PR refactors cleanup jobs to rely on repository abstractions, improves duplicate stack cleanup behavior, and adds integration coverage for cleanup/repository operations.

Changes:

• Reworked orphaned-data and cleanup jobs with lock renewal, cancellation checks, and repository-based deletes/updates.
• Added event repository helpers for bulk deletion, stack reassignment, and distinct-id aggregation.
• Added integration tests for cleanup pagination, retention, duplicate signatures, and event repository operations.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/Exceptionless.Core/Jobs/CleanupDataJob.cs	Extends lock duration and renews locks during paged cleanup/retention loops.
src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs	Replaces direct Elasticsearch calls with repository methods for orphan cleanup and duplicate-stack fixing.
src/Exceptionless.Core/Repositories/EventRepository.cs	Adds bulk delete, stack reassignment, and composite aggregation helpers.
src/Exceptionless.Core/Repositories/Interfaces/IEventRepository.cs	Exposes new event repository cleanup/query APIs and composite cursor type.
src/Exceptionless.Core/Repositories/StackRepository.cs	Adds duplicate signature aggregation lookup.
src/Exceptionless.Core/Repositories/Interfaces/IStackRepository.cs	Exposes duplicate signature lookup API.
tests/Exceptionless.Tests/Jobs/CleanupDataJobTests.cs	Adds cleanup pagination and retention integration coverage.
tests/Exceptionless.Tests/Jobs/CleanupOrphanedDataJobTests.cs	Adds integration coverage for orphan cleanup and duplicate stack merging.
tests/Exceptionless.Tests/Repositories/EventRepositoryTests.cs	Adds coverage for distinct ids, stack reassignment, and bulk delete helpers.
tests/Exceptionless.Tests/Repositories/StackRepositoryTests.cs	Adds duplicate signature repository coverage.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[niemyjski]

Merged origin/main and addressed the outstanding review feedback in 92fa434.

Summary:

• Removed the accidental packages/ diff from the PR; the final diff against origin/main is back to the intended cleanup/repository/test files.
• Added/kept integration coverage for the new repository methods, duplicate signature lookup, soft-delete filtering, empty ReassignStackAsync safety, orphan cleanup, and cleanup usage accounting.
• Updated duplicate-stack cleanup to loop until no duplicate signatures remain and to renew locks during long-running cleanup paths.
• Documented the composite aggregation cursor path and aligned it with the current Elastic client API.
• Treated missing event indexes as zero matching events during event cleanup, which fixes empty soft-deleted project cleanup.

Verification:

• dotnet test -- --filter-class Exceptionless.Tests.Jobs.CleanupDataJobTests: 26 passed
• dotnet test -- --filter-class Exceptionless.Tests.Jobs.CleanupOrphanedDataJobTests: 24 passed
• dotnet test -- --filter-class Exceptionless.Tests.Repositories.EventRepositoryTests: 17 passed, 1 skipped performance test
• dotnet test -- --filter-class Exceptionless.Tests.Repositories.StackRepositoryTests: 13 passed

All review threads are resolved.

[niemyjski]

Production-readiness pass complete on f6fcc4b.

• Merged current origin/main with normal merge commit 65fe08e.
• Added lossless concurrent cleanup hardening, exact event-index verification/fail-closed behavior, durable/transitive duplicate-stack redirects, bounded late-event reconciliation, optimistic concurrency, and patch-based hot update paths.
• Added integration coverage for closed/missing indexes, partial failure/idempotency, stale saves, chain/cycle redirects, late events and counters, tenant deletion, serialization boundaries, and concurrent tag/status/regression updates.
• Local verification: dotnet build --no-restore -m:1 passed with 0 warnings/errors; dotnet test --no-build passed 2336 tests with 2 intentional performance-test skips and 0 failures.
• Current Build workflow is green: version, test-api with coverage, test-client, test-e2e with localhost AppHost, and docker-build all succeeded. Non-PR deploy/publish/preview jobs were skipped as designed.
• Final GraphQL audit: all 21 review threads are resolved; 0 unresolved actionable threads. PR merge state is CLEAN.

Production-readiness verdict: READY.

[github-actions]

Package	Line Rate	Branch Rate	Complexity	Health
Exceptionless.AppHost	38%	39%	144	❌
Exceptionless.Core	72%	64%	8934	➖
Exceptionless.Insulation	23%	23%	205	❌
Exceptionless.Web	84%	67%	6841	✔
Summary	76% (21607 / 28420)	65% (9857 / 15224)	16124	✔

[niemyjski]

Production-readiness refresh complete on bd560b0.

• Merged current origin/main (87af2a0) with normal merge commit cfc28b7; no rebase or force-push.
• Re-reviewed the full 18-file PR diff for correctness, compatibility, security, test coverage, and maintainability.
• Fixed the remaining documented convention issue by renaming FixDuplicateStacksAsync and aligning new test names with MethodUnderTest_Scenario_ExpectedBehavior.
• Updated the PR description to document the durable redirect/reconciliation, optimistic-concurrency, and patch-based hot-path hardening already present in the final diff.
• Local build: dotnet build --disable-build-servers --no-restore -m:1 passed with 0 warnings and 0 errors.
• Local service-free contract proof: PublicApiCompatibilityTests passed 4/4. The service-backed local suite was contaminated by concurrent shared Elasticsearch state (alias/index collisions, duplicate fixed test ids, missing shards); isolated CI is authoritative.
• GitHub Actions run 29467194402 is green: version, test-api with the full .NET suite and coverage, test-client, localhost Aspire/Playwright test-e2e, and docker-build all succeeded. Deploy/publish/preview-only jobs skipped as designed.
• Final GraphQL audit: 21/21 review threads resolved, 0 unresolved; merge state CLEAN; no incoming main/head/review changes.

Production-readiness verdict: READY. PR remains open and unmerged.