server: elect the claim skip, and have clients advertise it - #122
Open
erikhortsch wants to merge 5 commits into
Open
server: elect the claim skip, and have clients advertise it#122erikhortsch wants to merge 5 commits into
erikhortsch wants to merge 5 commits into
Conversation
WithClientSkipClaim only reaches clients built through ClientParams.Options(). Constructors that hand-pick options off the params struct drop it silently, and some of those live in other repos, so the caller-side switch cannot cover the fleet. The skipped round trip is server->client->server before the handler runs, so the server already holds the decision; the request bit is only authorization. Let the server elect it too, for every queue rpc it handles. Queue is still re-checked, and the outcome is still observed as ClaimSkipped. Takes func() bool rather than bool to keep the same runtime revocability as the client option. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two switches for one decision left the caller able to force an announcement at a server that never opted in, and put the compatibility risk on the wrong end: a server electing to skip would announce to callers predating the field. Make the request bit an advertisement -- always set on a queue rpc, since a caller built from this version can always accept an announcement -- and let the server alone decide whether to make one. Both ends must now agree, so an older caller is never sent an announcement it cannot read, and WithClientSkipClaim has nothing left to configure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CS-1992: a handler that errors on receipt publishes its response right behind the announcement, on a different channel, and nothing orders their delivery. When the response won the race, selection stashed it as a fallback and the announcement then sent the caller off to wait on a channel already drained, turning an instant error into a request timeout. The stash exists for broadcast, where an early error is one server rejecting a request it could not read and another may yet bid. On a queue rpc the responder is the only server that received the request, so nothing else can ever arrive: return the response the moment it appears. This also stops a queue caller from waiting out the selection timeout to surface a malformed-request rejection, the case with no announcement at all. Reported in #123, which surfaced the stashed error once the announcement arrived; resolving it at the response site keeps selection from consuming the answer in the first place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Field 9 carried the caller's election of the skip, which pre-election servers honor unconditionally. A caller that advertises on the same number would switch those servers on for every queue rpc, with no opt-in and no kill switch, since they have no server-side election to revoke. At 10, the advertisement is invisible to every server that predates it: they see field 9 absent and negotiate. The reverse skew already degraded safely -- an old caller electing on 9 is simply negotiated with. Field 9 is reserved so it cannot come back meaning something else. Also catches pkg/client tests up to selectServer's queue parameter, which the previous commit missed, and covers the queue/broadcast split directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
reserved 9 stays: protoc rejects any later field reusing the number, so it is enforcement, not documentation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WithClientSkipClaim(#120) only reaches clients constructed throughrpc.ClientParams.Options()/Args(). Constructors that hand-pick options offthe params struct drop it silently — and some of them live in other repos, so
the caller-side switch cannot cover the fleet. In staging,
DirectorRoomStoreand
RoomManagerare still granting claims for exactly this reason;RoomManager's client is built inlivekit-server, which cloud does notoverride.
The round trip being skipped is server→client→server, and it delays the handler
starting, so the server is where the decision belongs. This moves it there:
WithServerSkipClaim— a server elects to announce rather than negotiate,for every queue rpc it handles. Takes
func() boolto keep the runtimerevocability the client option had.
WithClientSkipClaimis removed. The request bit becomes anadvertisement: always set on a queue rpc, because a caller built from this
version can always accept an announcement.
handlingbecomesir.SkipClaim && serverSkip && h.i.Queue— both ends mustagree. The queue re-check is unchanged, so a broadcast rpc still negotiates, and
the outcome is still reported as
ClaimSkipped.Why one switch instead of two
Two switches for one decision let a caller force an announcement at a server that
never opted in, and put the version-skew risk on the wrong end. With the server
electing and the client merely advertising, an announcement can only ever reach a
caller that asked for one — a caller predating the field leaves it false and is
always negotiated with.
That closes the concern this PR originally shipped with. The earlier revision had
a new server announcing
Handling: trueto clients on v0.7.2/v0.7.3 that havenever heard of the field; I argued that degraded benignly, but "benign" was code
reading rather than a mixed-version run. It is now structurally impossible
instead, and covered by a test.
The advertisement also moves to a new field number (10, with 9 reserved).
Field 9 carried v0.7.4's election, which v0.7.4 servers honor unconditionally
— an always-on advertisement on the same number would have switched skip-claim
on at every v0.7.4 server, with no opt-in and no kill switch, since v0.7.4 has
no server-side election to revoke. At 10, the advertisement is invisible to
every server that predates it. Both skew directions now degrade to negotiating:
Verification
go build ./...— cleango test -race ./...— passes, except a pre-existing failure inTestRPC/Local/RPC:counter++in the test's ownaddOnehandler(
internal/test/psrpc_test.go:81) is unsynchronized. Confirmed identical onorigin/mainat c7c1207 with these changes stashed. Not addressed here.TestSkipClaimexercises Local, NATS and Redis, and asserts the queue rpcskips, the handler runs exactly once, and a broadcast rpc still claims.
TestSkipClaimCallerDoesNotAdvertisestrips the field on the wire to stand infor an older caller, and asserts it is still granted against a server that has
elected to skip. I mutation-checked this one — dropping
ir.SkipClaim &&fromthe gate fails it.
TestSkipClaimDisabledByDefaultandTestSkipClaimRevokedAtRuntimenow coverthe server end: unset means claim, and revoking takes effect on the next
request.
Race fix (CS-1992, supersedes #123)
A handler that errors on receipt publishes its response right behind the
announcement, on a different channel, and nothing orders their delivery. When
the response won the race,
selectServerstashed it as a broadcast-stylefallback, and the announcement then sent the caller off to wait on a channel
already drained — turning an instant error into a request timeout.
#123 surfaced the stashed error once the announcement arrived. Fixed here at the
response site instead: on a queue rpc the responder is the only server that
received the request, so any response — including an error — is the request's
answer, returned the moment it appears. The broadcast stash behavior is
unchanged, since there an early error is one server rejecting a request it could
not read while another may yet bid. This also stops a queue caller from waiting
out the full selection timeout to surface a malformed-request rejection (the
variant with no announcement at all, which #123 did not reach).
TestSkipClaimFastFailingHandlerreproduces the lost race deterministically —the test bus delays every announcement so the response always wins — rather than
looping and hoping for unlucky scheduling; on this machine a 20-iteration loop
never lost the race at all. Mutation-checked: restoring the stash on the queue
path fails it with
deadline_exceeded.Note
The proto field keeps the name
skip_claimthough it now means "can accept askip" rather than "please skip". Renaming it would churn
internal.pb.gofor anaming nicety; the comment carries the meaning. Say the word if you would rather
have the rename.
Related
Follow-ups that depend on a tag cut from this: livekit/protocol#1729 (wire into
WithDefaultServerOptions) and livekit/cloud#4586 (config + client-constructorcleanup).
🤖 Generated with Claude Code