Re-find the proclock after waiting instead of trusting a stale pointer - #83
Merged
Conversation
PANIC: proclock table corrupted, reported by Antithesis (ORI-247) on
insert into txn0 as t (id, sk, val) values ($1,$2,$3)
on conflict (id) do update set val = CONCAT(t.val, ',', $4) where t.id = $5
LockAcquireExtended() decides, after WaitOnLock() returns
PROC_WAIT_STATUS_OK, whether the lock was granted by reading holdMask out
of the proclock pointer it captured before going to sleep. That pointer
is not guaranteed to still be ours.
A waiter can be released by RemoveFromWaitQueue() rather than by the
normal grant path -- orioledb's oxid_notify_all() does exactly that, and
then reports PROC_WAIT_STATUS_OK, which is what the LOCKACQUIRE_NOT_AVAIL
branch here already exists to absorb. RemoveFromWaitQueue() calls
CleanUpLock(), and that deletes the proclock whenever it holds nothing
else -- always the case for a pure waiter, e.g. the ShareLock a
VirtualXactLock() sleeps on. The dynahash element then goes back on the
freelist, so holdMask afterwards reads whatever the slot's next owner
put there. Zero, usually, and the NOT_AVAIL branch does the right thing
by accident. A stray set bit and we believe in a grant that never
happened: GrantLockLocal() runs, and the LockRelease() that follows
ungrants counts that were never incremented and removes a proclock that
is no longer in the table.
Re-find the lock and the proclock from the locktag under the partition
lock, and decide from that. The lock itself can be gone too -- the same
CleanUpLock() drops it once nRequested reaches zero -- so the lookup
starts from the tag, not from the captured LOCK pointer.
Measured with a probe that compared the captured pointer against a
re-find: 313 reads of an already-freed proclock in two minutes of
contended upserts against orioledb. Forcing the recycled slot to carry
the lockmode bit turns the very first one into
TRAP: failed Assert("(lock->nRequested > 0) && ..."), lock.c:1767
UnGrantLock / LockRelease / VirtualXactLock / ...
table_tuple_insert_with_arbiter / ExecInsert
which is the assert-build face of the reported PANIC. With this change
the same workload takes the re-find path 163 times in 60 seconds and
runs clean.
PG16 and PG17 carry the same post-wait check and need the same fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes ORI-247 —
PANIC: proclock table corrupted, reported by Antithesison
and described there as "now happening consistently".
The defect
After
WaitOnLock()returnsPROC_WAIT_STATUS_OK,LockAcquireExtended()decides whether the lock was granted by reading
holdMaskout of theproclock pointer it captured before going to sleep. That pointer is not
guaranteed to still be ours.
A waiter can be released by
RemoveFromWaitQueue()rather than by the normalgrant path — orioledb's
oxid_notify_all()does exactly that and then reportsPROC_WAIT_STATUS_OK, which is what theLOCKACQUIRE_NOT_AVAILbranch herealready exists to absorb. But
RemoveFromWaitQueue()callsCleanUpLock(),and that deletes the proclock whenever it holds nothing else — always the
case for a pure waiter, e.g. the
ShareLockaVirtualXactLock()sleeps on.The dynahash element goes back on the freelist, so
holdMaskafterwards readswhatever the slot's next owner put there.
Zero, usually — and then the
NOT_AVAILbranch does the right thing byaccident. A stray set bit, and we believe in a grant that never happened:
GrantLockLocal()runs, and theLockRelease()that follows ungrants countsthat were never incremented and removes a proclock that is no longer in the
table.
Why this statement: orioledb calls
oxid_notify_all()from theINSERT ... ON CONFLICT DO UPDATEretry loop (o_tbl_insert_on_conflict()),which is also what makes the conflicting backends sleep in
wait_for_oxid()→VirtualXactLock()in the first place.Evidence
A probe comparing the captured pointer against a re-find, under contended
upserts against orioledb:
313 reads of an already-freed proclock in two minutes. Forcing the recycled
slot to carry the lockmode bit — which is what happens for real when another
backend takes the slot between the free and this read — turns the very first
one into:
i.e. the assert-build face of the reported PANIC, reached from the upsert
executor path named in the ticket. (Antithesis runs a non-cassert build, so it
goes on to
CleanUpLock()→hash_search(HASH_REMOVE)of an absent entry →PANIC: proclock table corrupted.)The fix
Re-find the lock and the proclock from the locktag under the partition lock,
and decide from that. The lock itself can be gone too — the same
CleanUpLock()drops it oncenRequestedreaches zero — so the lookup startsfrom the tag rather than from the captured
LOCKpointer.Validation
With this change the same workload takes the re-find path 163 times in 60
seconds (each one a freed-pointer read avoided) and runs clean: no PANIC, no
assert. orioledb's regress (48) and isolation (34) suites pass against the
patched backend.
Not done here
PG16 and PG17 carry the same post-wait check and need the same fix —
patches16andpatches17, same function, same!(proclock->holdMask & ...)test. I built and validated only 18; the other two deserve the same change
rather than a blind port.