Skip to content

Fix: Channel failure on shm resize due to CacheMiss#254

Open
griffinmilsap wants to merge 4 commits into
devfrom
fix/channel-failure-on-shm-resize
Open

Fix: Channel failure on shm resize due to CacheMiss#254
griffinmilsap wants to merge 4 commits into
devfrom
fix/channel-failure-on-shm-resize

Conversation

@griffinmilsap

Copy link
Copy Markdown
Collaborator

If a Publisher resizes SHM repeatedly, it can happen that the receiving MessageChannel either fails to lease the block before it becomes deallocated, or that the associated block is no-longer leased and is deallocated when backpressure is cleared, resulting in an uncaught CacheMiss exception which kills the Task servicing the connection silently.

The test case and hardening of message servicing in MessageChannel in this PR likely closes #253.

cboulay added a commit that referenced this pull request Jul 20, 2026
Combines the two independent fixes for the silent channel-death on
mid-stream SHM resize (issue #253):

- On reattach failure, recover instead of raising. A fast resize storm
  can dealloc the segment before we attach; broaden the catch to
  (ValueError, OSError) and drop to the shm-is-None ack path rather than
  killing the channel task. (from #254)
- Swallow CacheMiss in _release_backpressure: the cached entry can be
  evicted by the reattach path before backpressure clears; a miss just
  means nothing to free, so keep acking. This is the specific crash #253
  reports. (from #254)

Retains the UninitializedMemory guards at the msg_id() call sites, which
#254 lacks -- msg_id() raises UninitializedMemory on an uninitialized
slot (the exact race), which would otherwise escape #254's buffer-id
check and kill the task it aims to protect.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cboulay

cboulay commented Jul 20, 2026

Copy link
Copy Markdown
Member

The attach-recovery + CacheMiss guard here are the right root-cause fixes for #253 — nice.

One gap worth closing before merge: the two MessageMarshal.msg_id(shm_buf) call sites in the resize path (the repopulate loop and the if MessageMarshal.msg_id(shm_buf) != msg_id: buffer-id check) aren't guarded against UninitializedMemory. msg_id() calls _assert_initialized() and raises UninitializedMemory on an uninitialized slot — which is exactly the state a slot can be in mid-resize. That exception would escape the receive loop and kill the channel task, i.e. the same silent-death this PR is closing.

I pushed a branch on top of this one that wraps both sites (treat uninitialized as skip/mismatch → drop + release) and adds a parametrized cross-process grow test (single grow / two successive grows / grow-on-first-message) alongside your shm_resize_race_runner.py:

cboulay/shm-uninitialized-guards (branched from fix/channel-failure-on-shm-resize, commit 5ae385c)

Both your repro and the new grow tests pass against it (16 passed). Happy to open it as a PR into your branch or you can cherry-pick — whichever you prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cross-process SHM grow (buf_size exceeded) can silently and permanently kill a subscriber channel via uncaught assert in MessageChannel.monitor

2 participants