A namespace the routing tier still serves is not empty - #299
Open
bjmeetsfo wants to merge 2 commits into
Open
Conversation
set_namespace_state checked "namespace still holds a live table" under a read lock, released it, then recorded and applied the change. A table created in that window was stranded: the namespace reached Dropped with a live table still inside it, which is what the check exists to prevent. The metaserver serves each connection on its own thread, so a drop and a table creation arriving together is ordinary traffic. It is the same end state as the raft-path defect, reached differently -- there the guard was absent, here it was released too early. The check, the record and the apply now happen under one write lock. The apply body moves into apply_namespace_state_locked, taking an already-held &mut MetaState, so the guarded path can check and apply without letting go while replay keeps entering through apply_set_namespace_state and reapplying unconditionally. record_mutation stays before the state moves, so a crash between them replays the change rather than losing it, and it does not touch self.inner so holding the lock cannot deadlock. Holding the lock across record_mutation means one fsync with readers blocked. That is affordable here and nowhere else: the only callers are freeze, unfreeze and drop of a namespace, which are operator actions rather than a background loop or a per-request path. This was also the only method in the metaserver that released its lock between checking and applying.
Dropping a namespace refuses while a table in it is still live, so that dropping one cannot strand it. A proxy group is the other thing that can still depend on a namespace, and nothing counted it. Dropping a namespace with a group routing to it succeeded and left the contradiction standing: the group stayed Normal, its proxies stayed attached, and the very next proxy heartbeat still handed out the dropped namespace to serve. Calibration does not clean this up, because it keys off the group's own state and nothing changed it. No report flags it either, so the tier goes on routing to a namespace that is gone. Count a live proxy group as a live dependent, refused with namespace_still_routed. This keeps the existing order of operations -- drop the group, then drop the namespace -- and the check sits in the same write-locked block as the table check, so it cannot race a group created alongside the drop. Tests: a routed namespace is refused and nothing moves; dropping the group first still lets the namespace go, so the guard lets go; and freeze/unfreeze, which share this path, are untouched.
This was referenced Aug 25, 2026
bjmeetsfo
changed the base branch from
oss/close-the-namespace-drop-window
to
main
August 25, 2026 07:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dropping a namespace already refuses while a table in it is still live, so
that dropping one cannot strand a table. A proxy group is the other thing
that can still depend on a namespace, and nothing counted it.
What happens today
Declare a namespace, put a proxy group on it, calibrate so a proxy attaches,
then drop the namespace. Measured on
main:The drop succeeds and the contradiction stays standing. The group is still
Normal, its proxy is still attached, and the very next heartbeat still handsthat proxy the dropped namespace to serve. Calibration does not clean it up,
because it keys off the group's own state and nothing changed that. No report
flags it either, so the routing tier goes on routing to a namespace that is
gone.
The change
Count a live proxy group as a live dependent of the namespace, refused with
namespace_still_routed. That keeps the order of operations the code alreadyimplies -- drop the group, then drop the namespace -- and
drop_proxy_groupalready releases the proxies through the ordinary calibration path.
The check sits in the same write-locked block as the table check, so it cannot
race a group created alongside the drop.
Tests
Normaland the proxy keeps its assignment.
rather than making a routed namespace permanently undroppable.
The first test fails on the unmodified code with
a namespace with a proxy group routing to it was dropped.Base and a merge hazard worth knowing about
This is stacked on #285, which restructured the same function to hold one
write lock across the check and the apply. Merge #285 first.
It targets
mainrather than #285's branch on purpose:rust-ciandoss-readinessboth declarepull_request: branches: [main, rust-main], andfor
pull_requestthat filters the base. A pull request aimed at any otherbranch therefore gets only auto-approve, SPDX and gitleaks -- it is never
compiled and its tests are never run, while still reporting
CLEAN. Targetingmainmeans the diff here also carries #285's commit, and CI builds the twotogether, which is the state that will actually land. Once #285 merges, this
diff reduces to its own commit on its own.
Separately, and worth flagging because it affects #285 and #231 whether or not
this change exists: #285 and #231 conflict with each other, even though
GitHub reports both as mergeable against
main-- they are only ever comparedagainst
main, never against each other. Merging #285 and then #231 gives:The resolution needs care. #231 moves the emptiness check into
admission_refusal, which takesself.inner.read(). #285 holdsself.inner.write()across that same region. Resolving by simply taking#231's side puts a read acquisition inside the write lock, on the same thread
and the same lock. Only the
Droppedarm ofadmission_refusaltakes thatlock, so freeze and unfreeze would look fine and dropping a namespace would
hang.
That is measured, not predicted. Resolving it that way and running the
namespace drop test gives:
The resolution that works is to factor the guard so it takes
&MetaState:the public path calls it with the write guard it already holds, and the
propose path acquires its own read lock. If that factoring lands in #231,
the guard added here should move into it too, so the raft propose path
refuses a routed namespace as well -- today it bypasses this check exactly
as it bypasses the table check.