fix(lock): make a leaked registry lock visible, and budget the wait in seconds - #794
Open
fujibee wants to merge 7 commits into
Open
fix(lock): make a leaked registry lock visible, and budget the wait in seconds#794fujibee wants to merge 7 commits into
fujibee wants to merge 7 commits into
Conversation
… stuck Two halves of the same silence, reported from a Windows machine. A lock directory was created with nothing in it, so a lock that leaked said something was holding it and nothing about what. The operator's options were to guess or to remove it blind, and removing a live lock is worse than the leak. `agmsg_lock_acquire` now writes pid, command and host into the directory the moment it is held — best-effort, because the lock IS held as of the mkdir and a failure to annotate must not undo that. And `rmdir "$l" 2>/dev/null || true` treated two different events as one. A lock already gone is a released lock, nothing to report. A lock that will not go is the permanent leak this file's contract promises not to leave — every later command for that team waits for a holder that is never coming back — and it was indistinguishable from success. Release now reports the second and stays quiet about the first. Writing inside the directory makes the removal two steps, so both release paths go through one helper rather than each growing its own copy. THE REMEDY IT PRINTS WAS WRONG FIRST TIME. It said `rmdir <lock>`, which is the command that had just failed — a route ending where the operator already is. Measured by running the printed line: it fails. The only way release reaches that branch with the directory present is something inside it, which is exactly what rmdir refuses. It now prints `ls -la` then `rm -r`, and the test lifts the line out of the message and runs it, so the two cannot drift apart. Mutations, both run: release back to `rmdir … || true` -> 2 tests red drop the holder write -> 1 test red 7/7 in the lock suite. The three suites that use this library are green too: actas 22, plugin registry 9, type registry 20, no failures. Not addressed here, deliberately: nothing cleans up a lock left by a dead process. That is a different change with a real hazard — deciding a live lock is dead — and #778 does not ask for it. It asks for the leak to be visible, which is what this does.
…779) The wait was 1000 attempts with "= ~10s" written beside it. That arithmetic holds only where an mkdir and a sleep are free. Measured on macOS before touching anything: 100 attempts take 3 seconds, not 1 — already three times the stated figure, on the platform this was developed on. The report that raised this saw minutes on Windows, where a filesystem operation costs more. A wait announced in seconds has to be counted in seconds, or the number in the message is not about the wait. `AGMSG_LOCK_SECONDS` (default 10) is the budget now. `AGMSG_LOCK_TRIES` stays as a ceiling, still defaulting to 1000. Derived rather than assumed: it is set in four places, all of them tests making themselves fail fast, and nowhere in production. Whichever bound is reached first ends the wait, and the message says WHICH — "1000 tries" and "10 seconds" send an operator deciding whether to retry to different places. Two things worth recording about getting here. The first version of the new test set TRIES to 1000000 so that only the time bound could stop it. Under a mutation removing that bound, the test does not go red — it hangs the suite, twice, for ten minutes each. A check that hangs instead of reddening is worse than a slow one. It now uses 300, which is about nine seconds if the wait were counted in iterations and cannot be what ends a two-second budget. Both arms finish in seconds. And while chasing that, an interrupted restore put the MUTATED file back and the suite stayed green — the time bound was out of the condition and every test passed. Caught by reading the line rather than trusting the restore. That is the same defect as the one being fixed, one level up: a cleanup step whose failure is invisible. Mutations, both run and both red rather than hanging: remove the time bound from the condition -> tests 1 and 8 red (from #778, still holding) swallow the release failure -> 2 red 9/9 in the lock suite; actas 22, plugin registry 9, type registry 20, no failures.
…remedy Both findings are hazards this branch created. That is worth saying plainly: the holder file and the `rm -r` remedy were added to make a leak visible, and each of them opened a way to do something worse than the leak. **A release could delete a SUCCESSOR's lock.** The sequence is the one the new remedy invites: a lock gets stuck, the operator removes the directory as the message tells them to, another process takes the same path, and the original release — still believing "I locked this path" — removes the new holder and rmdir's the new lock. Mutual exclusion gone, taken from a process that was using it. Acquire now records a token, and release compares it against what is in the holder file. Not a pid: pids recur. A mismatch means this process has no standing to remove anything, and it says so and stops. Ordering matters and got this wrong once. Removing the holder and then failing to rmdir left a lock nobody could prove was theirs — the next release read no token, concluded the lock belonged to someone else, and printed that instead of the stuck-lock report. Two contradictory lines about one directory, on the trap's second call. The holder is restored when the removal fails. **The printed remedy was not shell-safe.** `rm -r $l` unquoted, on a store root or team name containing a space, is several arguments — and team names are validated against empty / `.` / `..` / `/` / `\` / a leading `-` / control characters, and nothing else. The existing test ran the printed line under a TMPDIR with no spaces, so it passed the defect through. Quoted with the same scheme as lib/shquote.sh, inline rather than sourced to keep this library single-file. Controls, both specified in review and both run: disable the ownership check -> test 10 red (successor's lock deleted) unquote the printed remedy -> test 11 red (neighbour directory removed) The new tests use a path with a space and assert a sibling directory survives, because the old one could not have caught this. 11/11 in the lock suite; actas 22, plugin registry 9, type registry 20.
Three findings, all of them defects in the previous round's fix rather than in the original code. Worth stating that way: each attempt to close the hazard opened a smaller one, and the reviewer found each. **The token was per process, and this library holds several locks at once.** Its own opening comment says so — rename-team takes two. The second acquire overwrote the single token, so releasing the first read a mismatch, called it someone else's, and leaked it. Measured before the fix: BOTH locks leaked, with two false "held by another process" reports. Tokens are now stored per lock path, in a newline-separated variable because bash 3.2 has no associative arrays, matched whole for the reason AGMSG_HELD_LOCKS already documents. **A failed removal restored only the token line.** pid, command and host are what "a leaked lock says who left it" means, and a stuck removal is precisely when an operator needs them — so the one failure this PR exists for would have been the one failure with no diagnosis. The holder is read before removal and restored byte for byte. And the comment above it claimed "rmdir FIRST, holder still in place" while the code removed the holder first. A directory with a file in it cannot be rmdir'd, so that ordering was never possible; the comment described a design that could not exist. It now describes what runs: read, remove, try, restore on failure. **The token had too little entropy to decide ownership.** `$$` and a second collide across hosts on a shared store, and `$RANDOM` is 15 bits where it exists at all. 16 bytes from /dev/urandom, with a degraded fallback — and a token that cannot be established leaves the lock unprovable, which sends release down the "do not touch someone else's" branch rather than the deleting one. Controls, all specified in review and all run: token back to one per process -> 3 red (12 among them) restore only the token line -> 13 red constant nonce (either branch) -> 14 red 14/14 in the lock suite; actas 22, plugin registry 9, type registry 20. One thing found while measuring rather than by reading: after wiring the per-lock store, the holder file was still written from the old global variable, so the token landed in the store and never in the file. Every release then read an empty token and refused. The probe that caught it printed both values side by side; the code looked correct.
CI found what local runs had not: three shards red, all of them mine. `test_remote.bats` asserts on the string "timed out acquiring registry lock" with a five-attempt budget. The previous commit invented a second sentence — "gave up acquiring registry lock" — for the attempt ceiling, which is a contract change dressed as a message improvement. The caller broke, and the operator gained nothing a clause could not carry. One phrase now, with the bound in a clause: "after 3s" or "after 5 attempts (0s)". Both bounds are still distinguishable, which is the point of separating them, and nothing that matched before stops matching. The other two shards were the multi-lock defect fixed in the previous commit, confirmed rather than assumed: rename-team: an inert empty target dir does not block the rename now ok rename-team: escapes quoted team names and migrates only the … now ok releasing the engine's own lock leaves other held locks alone now ok Each was measured on this head and on the base, so "my change did that" is a comparison rather than a guess: the base passes, the previous head failed, this head passes. The lock suite's own assertion for the attempt bound was following my invented phrase, so it moved back to the shared one and now separates the two by the clause instead. 14/14 lock; test_sync_cipher 7/7; the three previously-failing tests pass.
… goes CI found two failures a full-PATH local run never could, and both come from where the holder file was written. **Inside the lock directory, the lock could not be removed without `rm`.** The core join path runs under an allow-listed PATH — `test_local_team_ids.bats` builds one with exactly the tools that path may use — and `rm` is not on it. A holder written inside made the directory non-empty, so `rmdir` failed and the one path that promises to work without python3 leaked a lock on every call. The holder is a sibling now: `<lock>.holder`. `rmdir` succeeds with no `rm` anywhere. **Which then broke `rename-team`.** It ends with `rmdir "$OLD_DIR"`, and a leftover sibling holder keeps the old directory standing — the rename appears to half-happen, with no lock message anywhere in the output to point at the cause. That is what the quoted-team-name test was reporting. The holder is removed when the lock is successfully removed, best-effort for the same PATH reason: a holder beside no lock is inert, and the removal that had to succeed did. Also dropped a hard dependency on `uname`, which is not on that PATH either; `$HOSTNAME` first, `uname` only as a fallback. Every test that has been red at any point in this branch, measured on this head: core join does not require python3 ok disconnect: no replacement can land while the engine … ok rename-team: an inert empty target dir does not block … ok rename-team: escapes quoted team names and migrates only … ok releasing the engine's own lock leaves other held locks alone ok 14/14 lock; actas 22, plugin registry 9, type registry 20, sync cipher 7. The pattern worth naming: three rounds of this PR were fixed by measuring what CI reported rather than by reading the diff, and each fix moved the defect somewhere a full PATH cannot see. A local suite with every tool installed is not the environment the code runs in.
The comment beside the holder relocation said the local run "never saw it", and the commit message went further: "CI found two failures a full-PATH local run never could". Both are wider than what happened. Asked whether it was really unreproducible, I built a directory of symlinks to the tools `test_local_team_ids.bats` allow-lists, pointed PATH at it, and got the negative control in one command: the pre-fix library leaks, this one releases. The defect was never invisible locally — the local suite runs with a full PATH by default, which is a habit, not a limit. The comment now says that, including how to reproduce it. A comment is enforcement to the next reader: "you cannot see this without CI" would send someone to push a branch where a shell loop would have answered them. The commit message cannot be rewritten, so this stands as its correction. 14/14, unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Declared reviewers: 1
References #778 and #779. No closing keyword —
integration/remoteis not thedefault branch, so one would not fire, and #779's Node-side half is untouched.
Measured once, at head
f924c833caaa297398d9f9c0b9f2f01143215b9f.#778 — a leaked lock could not say whose it was, or that it was stuck
The lock directory was created empty, so a lock that leaked said something held
it and nothing about what.
agmsg_lock_acquirewrites pid, command, host and atoken beside it —
<lock>.holder— as it takes the lock.And
rmdir "$l" 2>/dev/null || trueat both release sites treated "alreadygone" (a released lock) and "will not go" (the permanent leak this file promises
not to leave) as the same event. Release reports the second and stays quiet
about the first, with a remedy that works on the case that produced it.
#779 — the wait was counted in attempts and announced in seconds
1000 attempts with "= ~10s" beside it. Measured before changing anything: 100
attempts take 3 seconds, three times the stated figure, on the platform this
was written on.
AGMSG_LOCK_SECONDS(default 10) is the budget;AGMSG_LOCK_TRIESstays as a ceiling — derived, not assumed: set in fourplaces, all tests, none in production.
Every hazard in this branch was one the branch created
Five review rounds, and each finding was a defect in the previous round's fix
rather than in the original code. Recorded because the pattern is the point:
rmdir— the command that had just failed. Running theprinted line is what caught it.
invites: operator removes it, another process takes the path, the original
release deletes theirs. Ownership is checked against the token now.
its own opening comment says so. Both locks leaked. Tokens are per lock path.
diagnosis disappearing exactly when it is needed.
rmdirneededrmto succeed. Thecore join path runs under an allow-listed PATH with no
rm, and leaked a lockon every call. The holder is a sibling now, tidied when the lock goes.
Controls
Checks
Every test that was red at any point on this branch passes here, each compared
against the base rather than assumed:
core join does not require python3,disconnect: no replacement can land …, bothrename-teamcases, andreleasing the engine's own lock leaves other held locks alone.bats (ubuntu-latest 4/4)fails atnot ok 265 watch: relaunch with the SAME instance id replaces the previous watcher, on_wait_pidfile "$pf" "$w2"—the test and the assertion #595 already names. Tracked there. This branch cannot reach it:
watch.shreachesregistry-lock.shonly throughreset.sh, guarded byif [ -n "$DESPAWN_TARGET" ],and the failing test never sets it. That is one path closed, not a proof of
innocence.
Not in this change
Cleaning up a lock left by a dead process — a real hazard (deciding a live lock
is dead) and not what #778 asks for. The Node-side iteration budgets #779 also
mentions are in
remote-sync.mjs, which other branches are editing.