Skip to content

Hold the epoch lock while restoring from the WAL - #509

Open
samliok wants to merge 2 commits into
mainfrom
fix/epoch-start-wal-restore-race
Open

Hold the epoch lock while restoring from the WAL#509
samliok wants to merge 2 commits into
mainfrom
fix/epoch-start-wal-restore-race

Conversation

@samliok

@samliok samliok commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Fixes a data race caught by CI in TestReplicatedNotarizationRestart: https://github.com/ava-labs/Simplex/actions/runs/31749442819/job/94611614678

Epoch.Start() runs the WAL restore without holding the epoch lock. Restoring can schedule the block building task and arm monitor and timeout callbacks, all of which acquire the lock and mutate epoch state from other goroutines while the restore is still reading it. In the failing test, the restarted node recovers a notarized round from the WAL, becomes leader of the next round mid-restore, and the scheduled block building task writes the rounds map in storeProposal while the restore path reads it in handleFinalizeVoteMessage.

Every internal handler assumes the lock is held: HandleMessage takes it before dispatching, but restoreFromWal called the same handlers without it. Start() now holds the lock until the restore completes and the epoch is ready to receive messages, so scheduled tasks block until then. broadcastReplicationSync acquires the lock itself and stays outside the locked region.

@samliok
samliok marked this pull request as ready for review August 13, 2026 23:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants