[#861] Fail fast when a total update request gets no answer - #864
Merged
vharseko merged 2 commits intoAug 12, 2026
Merged
Conversation
… no answer Four fixes for the dsreplication enable/initialize hang and its silent failure modes: - initializeFromRemote and its retry now fail the task immediately when the broker cannot publish the InitializeRequestMsg instead of silently dropping it and waiting forever; - InitializeTask arms a watchdog that aborts the initialization when neither InitializeTargetMsg nor ErrorMsg arrives within 2 minutes; - the ErrorMsg staleness gates accept answers created in the same millisecond as the import/export context (single-host round-trips); - setExceptionIfNoneSet checked the argument instead of the field for null (upstream bug since 2014), so every failed total update reported success over partially imported data.
vharseko
added
replication
bug
concurrency
…roughout initialize() CodeQL flagged four may-be-null dereferences of ieCtx in initialize() after the stalled-watchdog guard was added. Three were real on the remotely-initiated import path: when acquireIEContext() rejected a concurrent import/export, the catch/finally blocks dereferenced a null ieCtx - or, now that setExceptionIfNoneSet() actually records errors, poisoned and released the context of the unrelated on-going operation. Acquire (or validate) the context before entering the import try/finally so ieCtx is final and provably non-null; a rejected initialization now answers the exporter with an ErrorMsg and leaves the on-going operation's context untouched. Also collapse the duplicated 3A Systems Portions lines in the file header.
maximthomas
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #861 (root-cause analysis and the fourth-bug addendum are in the issue comments).
dsreplication enable/initializecan hang forever in the window between publishing theInitializeRequestMsgand receiving theInitializeTargetMsg: nothing bounds that wait, and both the request and its error answer can be lost silently. A CI occurrence held a runner for 6 hours (mitigated by #862). Four fixes:ReplicationDomain.initializeFromRemote()and its retry path now check the result ofbroker.publish(...): a request silently dropped by a broker caught between two sessions (connection error, recovery pending after a reconnect) fails the task immediately withERR_INITIALIZATION_FAILED_NOCONNinstead of leaving it waiting forever. The broker's resend-on-reconnect procedure only replaysUpdateMsgs, so nothing else would ever deliver that request.ReplicationDomain.abortStalledInitializeFromRemote(timeoutMs), polled every second byInitializeTaskwhile it waits: if neitherInitializeTargetMsgnorErrorMsgarrives within 2 minutes of the request, the task fails withERR_NO_REACHABLE_PEER_IN_THE_DOMAIN(which the dsreplication CLI already retries). A lateInitializeTargetMsgracing the abort is detected under the context's monitor and ignored by the listener, so the abort cannot race the start of an import.errorMsg.getCreationTime() > ieCtx.startTime) become>=: with all servers on one host the whole request/rejection round-trip can complete within the millisecond the import/export context was created in, and the strict comparison discarded the legitimate answer as stale — the exact single-host CI topology where the hang was observed.setExceptionIfNoneSetnever set anything — the null check was made on the argument instead of the field (upstream bug present since the 2014 "Recreate OpenDJ 3 development branch" import). Every error funneled through it was silently dropped: an acceptedErrorMsg, a failed LDIF import (partial data, no generation-id recomputation) and mid-import disconnections all ended with the initialize task reportingCOMPLETED_SUCCESSFULLY.New regression tests in
ReplicationDomainTest:errorMsgFromSameMillisecondTerminatesPendingInitialize— a craftedErrorMsgtimestamped one millisecond before the context stays ignored, one timestamped in the same millisecond terminates the pending initialization with the right error (covers fixes 3 and 4).stalledInitializeFromRemoteIsAborted— a peer that never answers the request: the watchdog refuses to abort before the delay, aborts after it, fails the task with the peer-unreachable error, releases the import/export context, and a second call is a no-op.Test results:
ReplicationDomainTest12/12,InitOnLineTest10/10 (the end-to-end online-initialization suite most sensitive to fix 4).