feat(sync): keep the engine up when the server refuses, and let the reason be read

[fujibee]

Declared reviewers: 1

Closes #773.

Landing on integration/remote. Head 288545154f84e334460eec9d1e6e934170afceeb.

Closes does not fire off the default branch, so this issue is closed by hand after landing.

What was wrong

A remote may answer a write with a status meaning "the caller must do something" rather than "try again later". The engine treated that as a transport failure: not retryable, so it left the loop, main() rejected, and the process exited.

status then said:

<team>  connected (engine stopped — run: remote.sh sync start <team>) since <date>

which invites the one action that cannot work — starting it again produces the same refusal and the same exit, for as long as the server's answer stands.

The reason was never missing. event() writes it to run/remote-sync.<team>.log with a timestamp, at a known path. What was missing is a place to read it from: status opens a pidfile and the cycle stamp, and nothing an agent consults mentions the log. So this adds one fact beside the cycle stamp, not a new mechanism — the same move #760 made.

By class, not by number

A refusal is a 4xx the retry policy does not cover. The one such status in use today is deliberately not named anywhere in the engine — not in the code and not in a comment, and a test enforces that. Naming it invites the next reader to special-case it, and the one after that to add a sentence about what it means.

5xx stays a transport failure; the retryable 4xx stay retryable, because isRetryable is asked first.

The engine stays up

A refusal is not retryable — asking again does not change a decision — and it is not a transport failure either. Exiting turns a recoverable condition into a dead process whose status recommends a no-op.

And the answer can change, out of band: someone pays, a quota resets, an operator fixes a setting. Staying up means the next cycle recovers with nobody typing anything. It backs off to the longest interval and does not advance the failure count — a refusal is not evidence that the transport is degrading, so it must not shorten anything else's backoff.

What is stored, and what is deliberately not

Stored: the status, the code, the time, and the host from the endpoint the config already holds — verbatim, as the server said them.

Not stored: any sentence about what the refusal means or what to do about it. This engine talks to a remote — self-hosted, someone else's, or a service — and it cannot know why a particular one refused. A sentence it invents is wrong for some server. Interpretation belongs to whoever operates that server, and the host is there so a reader knows who that is.

A record that outlives its truth is worse than none, and the engine's delete is not what makes that hold.

The engine clears the record after a successful cycle, best-effort, and swallows the failure — an unwritable run directory, a permission change, a crash between the two writes — for the same reason recordCycleSuccess does: bookkeeping must never cost a cycle that worked. The first version of this PR stopped there, and the reader believed whatever file it found. A failed delete therefore reported a reversed decision for ever (raised in review; my own body claimed the two facts were written in one place and so could never disagree — they are, and they still could, because one of the two writes may fail).

So the guarantee is on the reading side: remote.sh compares the record to the last successful cycle and reports nothing older. Deleting can fail. Comparing cannot. Both readers go through one helper, so the human line and the JSON cannot disagree; with no cycle stamp there is nothing to compare against and the refusal is reported, because an engine that has refused and never succeeded is the case the record exists for.

The surface an agent reads

/agmsg remote status runs remote.sh status --json, and it is the only thing an agent consults about the engine — so the refusal is there, not only in a line a human reads. The whole record is passed through, so a field the engine adds later needs no change on the reading side. An unreadable record reports as absent rather than guessed at.

Tests

Engine (tests/remote_sync_engine.test.mjs, 4 new): classification by class — 400/401/402/403/409/422/451 refuse, 408/429 stay retryable, 5xx stays transport, a statusless error is neither; the loop does not exit on a refusal (it comes back for a third cycle, which is how the test can end at all) and records verbatim with the host; a successful cycle clears the record; and a non-retryable error that is not a refusal still ends the loop — the negative control, because staying up for everything would turn a malformed config into an engine that spins forever saying nothing useful.

Reading (tests/test_remote_refusal.bats, 9 new): the server's words repeated and nothing invented (subscri, upgrade, billing, plan all refused in the output); a status this protocol never enumerated surviving the trip; the JSON carrying it verbatim; null rather than a missing key when nothing was refused; an unreadable record read as absent; and that the engine names no particular remote — with a positive control on that search, since three refutes prove nothing if the search cannot find anything in that file.

Plus three on staleness: a refusal older than the last cycle is not reported though the file remains — the failed-delete case, made deliberate; a newer one still is, without which the check would be satisfied by never reporting anything; and with no stamp at all it is reported.

That check on the engine's vocabulary caught its own author: the only occurrence of the number was in my comment explaining that the number is not written anywhere. The comment now says the rule applies to itself, which is enforced.

Measurements, at this head

node --test tests/remote_sync_engine.test.mjs   82 tests, 82 pass, 0 fail
bats tests/test_remote_refusal.bats              9 tests,  0 failures
.github/scripts/check-enforced-assertions.sh   638, at the baseline (638)

Mutation: remove the currency comparison — that one case red, the other eight green, so the filter is not simply hiding everything.

Rebased onto origin/integration/remote before opening this, while no verdict existed to invalidate — cheaper than carrying a stale base into review. Everything above was re-run on the new base; the green from the old one was not reused.

merge-base = origin/integration/remote = d0b762d47747682e5a466cc1ad0df903c2712ac1
RESULT: STRUCTURAL, NOT MEASURED — structurally empty, not "empty".

Re-measured immediately before landing.

Three seats are editing this file

scripts/internal/remote-sync.mjs is also touched by #779 and by #781/#782. Ranges were exchanged and measured, not assumed: this branch adds a node:fs/promises import, a block after recordCycleSuccess, and three hunks inside runLoop. A grep of this diff for runDriver, child.on, readBoundedAuthorityFile, readRetainedCheckpointFile and TRUST_DIR returns 0.

No text conflict is expected. Drift is measured per file, so whichever of the three lands last will see non-empty drift here and needs a rebase and a fresh verdict; a verdict from an older head is not carried forward.

The same shape landed twice tonight

#778 in scripts/lib/registry-lock.sh is this defect in another file: cleanup done best-effort, its failure swallowed, and whatever remains read as the current truth. A lock that outlives its holder and a refusal that outlives its reversal are the same mistake.

The answer here — make the reader refuse to lie, rather than make the cleanup certain — was passed to that seat. If it turns up a third time it is a checker, and a checker only counts once it is wired into something that runs.

Not in this PR

The auto-start reading this record before starting a team. That belongs in scripts/lib/sync-autostart.sh, which does not exist on this branch — it arrives with #774 (PR #775), so those lines land and are reviewed there.

The ordering constraint runs the other way: #775 must not land before this. An engine that exits on a refusal, plus an auto-start that does not read refusals, is a restart loop; a refusal recorded with no reader yet is inert.