Skip to content

Re-find the proclock after waiting instead of trusting a stale pointer - #83

Merged
akorotkov merged 1 commit into
patches18from
proclock-refind-after-wait
Aug 13, 2026
Merged

Re-find the proclock after waiting instead of trusting a stale pointer#83
akorotkov merged 1 commit into
patches18from
proclock-refind-after-wait

Conversation

@akorotkov

Copy link
Copy Markdown
Contributor

Fixes ORI-247PANIC: proclock table corrupted, reported by Antithesis
on

insert into txn0 as t (id, sk, val) values ($1,$2,$3)
  on conflict (id) do update set val = CONCAT(t.val, ',', $4) where t.id = $5

and described there as "now happening consistently".

The defect

After WaitOnLock() returns PROC_WAIT_STATUS_OK, LockAcquireExtended()
decides whether the lock was granted by reading holdMask out of the
proclock pointer it captured before going to sleep
. That pointer is not
guaranteed to still be ours.

A waiter can be released by RemoveFromWaitQueue() rather than by the normal
grant path — orioledb's oxid_notify_all() does exactly that and then reports
PROC_WAIT_STATUS_OK, which is what the LOCKACQUIRE_NOT_AVAIL branch here
already exists to absorb. But RemoveFromWaitQueue() calls CleanUpLock(),
and that deletes the proclock whenever it holds nothing else — always the
case for a pure waiter, e.g. the ShareLock a VirtualXactLock() sleeps on.
The dynahash element goes back on the freelist, so holdMask afterwards reads
whatever the slot's next owner put there.

Zero, usually — and then the NOT_AVAIL branch does the right thing by
accident. A stray set bit, and we believe in a grant that never happened:
GrantLockLocal() runs, and the LockRelease() that follows ungrants counts
that were never incremented and removes a proclock that is no longer in the
table.

Why this statement: orioledb calls oxid_notify_all() from the
INSERT ... ON CONFLICT DO UPDATE retry loop (o_tbl_insert_on_conflict()),
which is also what makes the conflicting backends sleep in
wait_for_oxid()VirtualXactLock() in the first place.

Evidence

A probe comparing the captured pointer against a re-find, under contended
upserts against orioledb:

WARNING: stale proclock after wait: captured 0x1150b0510, refound 0x0, holdMask 0, mode 5

313 reads of an already-freed proclock in two minutes. Forcing the recycled
slot to carry the lockmode bit — which is what happens for real when another
backend takes the slot between the free and this read — turns the very first
one into:

TRAP: failed Assert("(lock->nRequested > 0) && (lock->requested[lockmode] > 0)"), lock.c:1767
  ExceptionalCondition / UnGrantLock / LockRelease / VirtualXactLock
  ... / table_tuple_insert_with_arbiter / ExecInsert / ExecModifyTable

i.e. the assert-build face of the reported PANIC, reached from the upsert
executor path named in the ticket. (Antithesis runs a non-cassert build, so it
goes on to CleanUpLock()hash_search(HASH_REMOVE) of an absent entry →
PANIC: proclock table corrupted.)

The fix

Re-find the lock and the proclock from the locktag under the partition lock,
and decide from that. The lock itself can be gone too — the same
CleanUpLock() drops it once nRequested reaches zero — so the lookup starts
from the tag rather than from the captured LOCK pointer.

Validation

With this change the same workload takes the re-find path 163 times in 60
seconds
(each one a freed-pointer read avoided) and runs clean: no PANIC, no
assert. orioledb's regress (48) and isolation (34) suites pass against the
patched backend.

Not done here

PG16 and PG17 carry the same post-wait check and need the same fix
patches16 and patches17, same function, same !(proclock->holdMask & ...)
test. I built and validated only 18; the other two deserve the same change
rather than a blind port.

PANIC: proclock table corrupted, reported by Antithesis (ORI-247) on

  insert into txn0 as t (id, sk, val) values ($1,$2,$3)
    on conflict (id) do update set val = CONCAT(t.val, ',', $4) where t.id = $5

LockAcquireExtended() decides, after WaitOnLock() returns
PROC_WAIT_STATUS_OK, whether the lock was granted by reading holdMask out
of the proclock pointer it captured before going to sleep.  That pointer
is not guaranteed to still be ours.

A waiter can be released by RemoveFromWaitQueue() rather than by the
normal grant path -- orioledb's oxid_notify_all() does exactly that, and
then reports PROC_WAIT_STATUS_OK, which is what the LOCKACQUIRE_NOT_AVAIL
branch here already exists to absorb.  RemoveFromWaitQueue() calls
CleanUpLock(), and that deletes the proclock whenever it holds nothing
else -- always the case for a pure waiter, e.g. the ShareLock a
VirtualXactLock() sleeps on.  The dynahash element then goes back on the
freelist, so holdMask afterwards reads whatever the slot's next owner
put there.  Zero, usually, and the NOT_AVAIL branch does the right thing
by accident.  A stray set bit and we believe in a grant that never
happened: GrantLockLocal() runs, and the LockRelease() that follows
ungrants counts that were never incremented and removes a proclock that
is no longer in the table.

Re-find the lock and the proclock from the locktag under the partition
lock, and decide from that.  The lock itself can be gone too -- the same
CleanUpLock() drops it once nRequested reaches zero -- so the lookup
starts from the tag, not from the captured LOCK pointer.

Measured with a probe that compared the captured pointer against a
re-find: 313 reads of an already-freed proclock in two minutes of
contended upserts against orioledb.  Forcing the recycled slot to carry
the lockmode bit turns the very first one into

  TRAP: failed Assert("(lock->nRequested > 0) && ..."), lock.c:1767
    UnGrantLock / LockRelease / VirtualXactLock / ...
    table_tuple_insert_with_arbiter / ExecInsert

which is the assert-build face of the reported PANIC.  With this change
the same workload takes the re-find path 163 times in 60 seconds and
runs clean.

PG16 and PG17 carry the same post-wait check and need the same fix.
@akorotkov
akorotkov merged commit 2859252 into patches18 Aug 13, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant