Skip to content

fix(lock): make a leaked registry lock visible, and budget the wait in seconds - #794

Open
fujibee wants to merge 7 commits into
integration/remotefrom
fix/778-lock-says-who-holds-it
Open

fix(lock): make a leaked registry lock visible, and budget the wait in seconds#794
fujibee wants to merge 7 commits into
integration/remotefrom
fix/778-lock-says-who-holds-it

Conversation

@fujibee

@fujibee fujibee commented Aug 14, 2026

Copy link
Copy Markdown
Owner

Declared reviewers: 1

References #778 and #779. No closing keyword — integration/remote is not the
default branch, so one would not fire, and #779's Node-side half is untouched.

Measured once, at head f924c833caaa297398d9f9c0b9f2f01143215b9f.

#778 — a leaked lock could not say whose it was, or that it was stuck

The lock directory was created empty, so a lock that leaked said something held
it and nothing about what. agmsg_lock_acquire writes pid, command, host and a
token beside it — <lock>.holder — as it takes the lock.

And rmdir "$l" 2>/dev/null || true at both release sites treated "already
gone" (a released lock) and "will not go" (the permanent leak this file promises
not to leave) as the same event. Release reports the second and stays quiet
about the first, with a remedy that works on the case that produced it.

#779 — the wait was counted in attempts and announced in seconds

1000 attempts with "= ~10s" beside it. Measured before changing anything: 100
attempts take 3 seconds
, three times the stated figure, on the platform this
was written on. AGMSG_LOCK_SECONDS (default 10) is the budget;
AGMSG_LOCK_TRIES stays as a ceiling — derived, not assumed: set in four
places, all tests, none in production.

Every hazard in this branch was one the branch created

Five review rounds, and each finding was a defect in the previous round's fix
rather than in the original code. Recorded because the pattern is the point:

  • the remedy printed rmdir — the command that had just failed. Running the
    printed line is what caught it.
  • release could delete a SUCCESSOR's lock — the sequence the new remedy
    invites: operator removes it, another process takes the path, the original
    release deletes theirs. Ownership is checked against the token now.
  • the token was per process, and this library holds several locks at once —
    its own opening comment says so. Both locks leaked. Tokens are per lock path.
  • a failed removal restored only the token, losing pid/command/host — the
    diagnosis disappearing exactly when it is needed.
  • the holder lived inside the lock, so rmdir needed rm to succeed. The
    core join path runs under an allow-listed PATH with no rm, and leaked a lock
    on every call. The holder is a sibling now, tidied when the lock goes.

Controls

swallow the release failure again        2 red
drop the holder write                    1 red
remove the time bound                    2 red (red, not hanging — an earlier
                                         version of that test hung instead)
disable the ownership check              3 red
restore only the token line              1 red
constant nonce, either branch            1 red
unquote the printed remedy               1 red

Checks

test_registry_lock    14        test_type_registry     20
test_actas_lock       22        test_local_team_ids     5
test_plugin_registry   9        test_team              76
                                          no failures
drift at push time    structurally empty (merge-base == the base)

Every test that was red at any point on this branch passes here, each compared
against the base rather than assumed: core join does not require python3,
disconnect: no replacement can land …, both rename-team cases, and
releasing the engine's own lock leaves other held locks alone.

bats (ubuntu-latest 4/4) fails at not ok 265 watch: relaunch with the SAME instance id replaces the previous watcher, on _wait_pidfile "$pf" "$w2"
the test and the assertion #595 already names. Tracked there. This branch cannot reach it: watch.sh reaches
registry-lock.sh only through reset.sh, guarded by if [ -n "$DESPAWN_TARGET" ],
and the failing test never sets it. That is one path closed, not a proof of
innocence.

Not in this change

Cleaning up a lock left by a dead process — a real hazard (deciding a live lock
is dead) and not what #778 asks for. The Node-side iteration budgets #779 also
mentions are in remote-sync.mjs, which other branches are editing.

… stuck

Two halves of the same silence, reported from a Windows machine.

A lock directory was created with nothing in it, so a lock that leaked said
something was holding it and nothing about what. The operator's options were to
guess or to remove it blind, and removing a live lock is worse than the leak.
`agmsg_lock_acquire` now writes pid, command and host into the directory the
moment it is held — best-effort, because the lock IS held as of the mkdir and a
failure to annotate must not undo that.

And `rmdir "$l" 2>/dev/null || true` treated two different events as one. A lock
already gone is a released lock, nothing to report. A lock that will not go is
the permanent leak this file's contract promises not to leave — every later
command for that team waits for a holder that is never coming back — and it was
indistinguishable from success. Release now reports the second and stays quiet
about the first.

Writing inside the directory makes the removal two steps, so both release paths
go through one helper rather than each growing its own copy.

THE REMEDY IT PRINTS WAS WRONG FIRST TIME. It said `rmdir <lock>`, which is the
command that had just failed — a route ending where the operator already is.
Measured by running the printed line: it fails. The only way release reaches
that branch with the directory present is something inside it, which is exactly
what rmdir refuses. It now prints `ls -la` then `rm -r`, and the test lifts the
line out of the message and runs it, so the two cannot drift apart.

Mutations, both run:

  release back to `rmdir … || true`   -> 2 tests red
  drop the holder write               -> 1 test red

7/7 in the lock suite. The three suites that use this library are green too:
actas 22, plugin registry 9, type registry 20, no failures.

Not addressed here, deliberately: nothing cleans up a lock left by a dead
process. That is a different change with a real hazard — deciding a live lock is
dead — and #778 does not ask for it. It asks for the leak to be visible, which
is what this does.
…779)

The wait was 1000 attempts with "= ~10s" written beside it. That arithmetic
holds only where an mkdir and a sleep are free. Measured on macOS before
touching anything: 100 attempts take 3 seconds, not 1 — already three times the
stated figure, on the platform this was developed on. The report that raised
this saw minutes on Windows, where a filesystem operation costs more.

A wait announced in seconds has to be counted in seconds, or the number in the
message is not about the wait. `AGMSG_LOCK_SECONDS` (default 10) is the budget
now.

`AGMSG_LOCK_TRIES` stays as a ceiling, still defaulting to 1000. Derived rather
than assumed: it is set in four places, all of them tests making themselves fail
fast, and nowhere in production. Whichever bound is reached first ends the wait,
and the message says WHICH — "1000 tries" and "10 seconds" send an operator
deciding whether to retry to different places.

Two things worth recording about getting here.

The first version of the new test set TRIES to 1000000 so that only the time
bound could stop it. Under a mutation removing that bound, the test does not go
red — it hangs the suite, twice, for ten minutes each. A check that hangs
instead of reddening is worse than a slow one. It now uses 300, which is about
nine seconds if the wait were counted in iterations and cannot be what ends a
two-second budget. Both arms finish in seconds.

And while chasing that, an interrupted restore put the MUTATED file back and the
suite stayed green — the time bound was out of the condition and every test
passed. Caught by reading the line rather than trusting the restore. That is the
same defect as the one being fixed, one level up: a cleanup step whose failure
is invisible.

Mutations, both run and both red rather than hanging:

  remove the time bound from the condition  -> tests 1 and 8 red
  (from #778, still holding) swallow the release failure -> 2 red

9/9 in the lock suite; actas 22, plugin registry 9, type registry 20, no
failures.
…remedy

Both findings are hazards this branch created. That is worth saying plainly: the
holder file and the `rm -r` remedy were added to make a leak visible, and each
of them opened a way to do something worse than the leak.

**A release could delete a SUCCESSOR's lock.** The sequence is the one the new
remedy invites: a lock gets stuck, the operator removes the directory as the
message tells them to, another process takes the same path, and the original
release — still believing "I locked this path" — removes the new holder and
rmdir's the new lock. Mutual exclusion gone, taken from a process that was
using it.

Acquire now records a token, and release compares it against what is in the
holder file. Not a pid: pids recur. A mismatch means this process has no
standing to remove anything, and it says so and stops.

Ordering matters and got this wrong once. Removing the holder and then failing
to rmdir left a lock nobody could prove was theirs — the next release read no
token, concluded the lock belonged to someone else, and printed that instead of
the stuck-lock report. Two contradictory lines about one directory, on the
trap's second call. The holder is restored when the removal fails.

**The printed remedy was not shell-safe.** `rm -r $l` unquoted, on a store root
or team name containing a space, is several arguments — and team names are
validated against empty / `.` / `..` / `/` / `\` / a leading `-` / control
characters, and nothing else. The existing test ran the printed line under a
TMPDIR with no spaces, so it passed the defect through. Quoted with the same
scheme as lib/shquote.sh, inline rather than sourced to keep this library
single-file.

Controls, both specified in review and both run:

  disable the ownership check   -> test 10 red (successor's lock deleted)
  unquote the printed remedy    -> test 11 red (neighbour directory removed)

The new tests use a path with a space and assert a sibling directory survives,
because the old one could not have caught this.

11/11 in the lock suite; actas 22, plugin registry 9, type registry 20.
Three findings, all of them defects in the previous round's fix rather than in
the original code. Worth stating that way: each attempt to close the hazard
opened a smaller one, and the reviewer found each.

**The token was per process, and this library holds several locks at once.**
Its own opening comment says so — rename-team takes two. The second acquire
overwrote the single token, so releasing the first read a mismatch, called it
someone else's, and leaked it. Measured before the fix: BOTH locks leaked, with
two false "held by another process" reports. Tokens are now stored per lock
path, in a newline-separated variable because bash 3.2 has no associative
arrays, matched whole for the reason AGMSG_HELD_LOCKS already documents.

**A failed removal restored only the token line.** pid, command and host are
what "a leaked lock says who left it" means, and a stuck removal is precisely
when an operator needs them — so the one failure this PR exists for would have
been the one failure with no diagnosis. The holder is read before removal and
restored byte for byte.

And the comment above it claimed "rmdir FIRST, holder still in place" while the
code removed the holder first. A directory with a file in it cannot be rmdir'd,
so that ordering was never possible; the comment described a design that could
not exist. It now describes what runs: read, remove, try, restore on failure.

**The token had too little entropy to decide ownership.** `$$` and a second
collide across hosts on a shared store, and `$RANDOM` is 15 bits where it exists
at all. 16 bytes from /dev/urandom, with a degraded fallback — and a token that
cannot be established leaves the lock unprovable, which sends release down the
"do not touch someone else's" branch rather than the deleting one.

Controls, all specified in review and all run:

  token back to one per process     -> 3 red (12 among them)
  restore only the token line       -> 13 red
  constant nonce (either branch)    -> 14 red

14/14 in the lock suite; actas 22, plugin registry 9, type registry 20.

One thing found while measuring rather than by reading: after wiring the
per-lock store, the holder file was still written from the old global variable,
so the token landed in the store and never in the file. Every release then read
an empty token and refused. The probe that caught it printed both values side by
side; the code looked correct.
CI found what local runs had not: three shards red, all of them mine.

`test_remote.bats` asserts on the string "timed out acquiring registry lock"
with a five-attempt budget. The previous commit invented a second sentence —
"gave up acquiring registry lock" — for the attempt ceiling, which is a contract
change dressed as a message improvement. The caller broke, and the operator
gained nothing a clause could not carry.

One phrase now, with the bound in a clause: "after 3s" or "after 5 attempts
(0s)". Both bounds are still distinguishable, which is the point of separating
them, and nothing that matched before stops matching.

The other two shards were the multi-lock defect fixed in the previous commit,
confirmed rather than assumed:

  rename-team: an inert empty target dir does not block the rename    now ok
  rename-team: escapes quoted team names and migrates only the …      now ok
  releasing the engine's own lock leaves other held locks alone       now ok

Each was measured on this head and on the base, so "my change did that" is a
comparison rather than a guess: the base passes, the previous head failed, this
head passes.

The lock suite's own assertion for the attempt bound was following my invented
phrase, so it moved back to the shared one and now separates the two by the
clause instead.

14/14 lock; test_sync_cipher 7/7; the three previously-failing tests pass.
… goes

CI found two failures a full-PATH local run never could, and both come from
where the holder file was written.

**Inside the lock directory, the lock could not be removed without `rm`.** The
core join path runs under an allow-listed PATH — `test_local_team_ids.bats`
builds one with exactly the tools that path may use — and `rm` is not on it.
A holder written inside made the directory non-empty, so `rmdir` failed and the
one path that promises to work without python3 leaked a lock on every call.

The holder is a sibling now: `<lock>.holder`. `rmdir` succeeds with no `rm`
anywhere.

**Which then broke `rename-team`.** It ends with `rmdir "$OLD_DIR"`, and a
leftover sibling holder keeps the old directory standing — the rename appears to
half-happen, with no lock message anywhere in the output to point at the cause.
That is what the quoted-team-name test was reporting. The holder is removed when
the lock is successfully removed, best-effort for the same PATH reason: a holder
beside no lock is inert, and the removal that had to succeed did.

Also dropped a hard dependency on `uname`, which is not on that PATH either;
`$HOSTNAME` first, `uname` only as a fallback.

Every test that has been red at any point in this branch, measured on this head:

  core join does not require python3                              ok
  disconnect: no replacement can land while the engine …          ok
  rename-team: an inert empty target dir does not block …         ok
  rename-team: escapes quoted team names and migrates only …      ok
  releasing the engine's own lock leaves other held locks alone   ok

14/14 lock; actas 22, plugin registry 9, type registry 20, sync cipher 7.

The pattern worth naming: three rounds of this PR were fixed by measuring what
CI reported rather than by reading the diff, and each fix moved the defect
somewhere a full PATH cannot see. A local suite with every tool installed is
not the environment the code runs in.
The comment beside the holder relocation said the local run "never saw it",
and the commit message went further: "CI found two failures a full-PATH local
run never could".

Both are wider than what happened. Asked whether it was really unreproducible,
I built a directory of symlinks to the tools `test_local_team_ids.bats`
allow-lists, pointed PATH at it, and got the negative control in one command:
the pre-fix library leaks, this one releases. The defect was never invisible
locally — the local suite runs with a full PATH by default, which is a habit,
not a limit.

The comment now says that, including how to reproduce it. A comment is
enforcement to the next reader: "you cannot see this without CI" would send
someone to push a branch where a shell loop would have answered them.

The commit message cannot be rewritten, so this stands as its correction.

14/14, unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant