Skip to content

server: elect the claim skip, and have clients advertise it - #122

Open
erikhortsch wants to merge 5 commits into
mainfrom
erik/server-skip-claim
Open

server: elect the claim skip, and have clients advertise it#122
erikhortsch wants to merge 5 commits into
mainfrom
erik/server-skip-claim

Conversation

@erikhortsch

@erikhortsch erikhortsch commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

WithClientSkipClaim (#120) only reaches clients constructed through
rpc.ClientParams.Options()/Args(). Constructors that hand-pick options off
the params struct drop it silently — and some of them live in other repos, so
the caller-side switch cannot cover the fleet. In staging, DirectorRoomStore
and RoomManager are still granting claims for exactly this reason;
RoomManager's client is built in livekit-server, which cloud does not
override.

The round trip being skipped is server→client→server, and it delays the handler
starting
, so the server is where the decision belongs. This moves it there:

  • WithServerSkipClaim — a server elects to announce rather than negotiate,
    for every queue rpc it handles. Takes func() bool to keep the runtime
    revocability the client option had.
  • WithClientSkipClaim is removed. The request bit becomes an
    advertisement: always set on a queue rpc, because a caller built from this
    version can always accept an announcement.

handling becomes ir.SkipClaim && serverSkip && h.i.Queue — both ends must
agree. The queue re-check is unchanged, so a broadcast rpc still negotiates, and
the outcome is still reported as ClaimSkipped.

Why one switch instead of two

Two switches for one decision let a caller force an announcement at a server that
never opted in, and put the version-skew risk on the wrong end. With the server
electing and the client merely advertising, an announcement can only ever reach a
caller that asked for one — a caller predating the field leaves it false and is
always negotiated with.

That closes the concern this PR originally shipped with. The earlier revision had
a new server announcing Handling: true to clients on v0.7.2/v0.7.3 that have
never heard of the field; I argued that degraded benignly, but "benign" was code
reading rather than a mixed-version run. It is now structurally impossible
instead, and covered by a test.

The advertisement also moves to a new field number (10, with 9 reserved).
Field 9 carried v0.7.4's election, which v0.7.4 servers honor unconditionally
— an always-on advertisement on the same number would have switched skip-claim
on at every v0.7.4 server, with no opt-in and no kill switch, since v0.7.4 has
no server-side election to revoke. At 10, the advertisement is invisible to
every server that predates it. Both skew directions now degrade to negotiating:

old server (≤ v0.7.4) new server
old client unchanged elects on 9, never read → negotiated
new client advertises on 10, never read → negotiated skips iff server elected

Verification

  • go build ./... — clean
  • go test -race ./... — passes, except a pre-existing failure in
    TestRPC/Local/RPC: counter++ in the test's own addOne handler
    (internal/test/psrpc_test.go:81) is unsynchronized. Confirmed identical on
    origin/main at c7c1207 with these changes stashed. Not addressed here.
  • TestSkipClaim exercises Local, NATS and Redis, and asserts the queue rpc
    skips, the handler runs exactly once, and a broadcast rpc still claims.
  • TestSkipClaimCallerDoesNotAdvertise strips the field on the wire to stand in
    for an older caller, and asserts it is still granted against a server that has
    elected to skip. I mutation-checked this one — dropping ir.SkipClaim && from
    the gate fails it.
  • TestSkipClaimDisabledByDefault and TestSkipClaimRevokedAtRuntime now cover
    the server end: unset means claim, and revoking takes effect on the next
    request.

Race fix (CS-1992, supersedes #123)

A handler that errors on receipt publishes its response right behind the
announcement, on a different channel, and nothing orders their delivery. When
the response won the race, selectServer stashed it as a broadcast-style
fallback, and the announcement then sent the caller off to wait on a channel
already drained — turning an instant error into a request timeout.

#123 surfaced the stashed error once the announcement arrived. Fixed here at the
response site instead: on a queue rpc the responder is the only server that
received the request, so any response — including an error — is the request's
answer, returned the moment it appears. The broadcast stash behavior is
unchanged, since there an early error is one server rejecting a request it could
not read while another may yet bid. This also stops a queue caller from waiting
out the full selection timeout to surface a malformed-request rejection (the
variant with no announcement at all, which #123 did not reach).

TestSkipClaimFastFailingHandler reproduces the lost race deterministically —
the test bus delays every announcement so the response always wins — rather than
looping and hoping for unlucky scheduling; on this machine a 20-iteration loop
never lost the race at all. Mutation-checked: restoring the stash on the queue
path fails it with deadline_exceeded.

Note

The proto field keeps the name skip_claim though it now means "can accept a
skip" rather than "please skip". Renaming it would churn internal.pb.go for a
naming nicety; the comment carries the meaning. Say the word if you would rather
have the rename.

Related

Follow-ups that depend on a tag cut from this: livekit/protocol#1729 (wire into
WithDefaultServerOptions) and livekit/cloud#4586 (config + client-constructor
cleanup).

🤖 Generated with Claude Code

WithClientSkipClaim only reaches clients built through ClientParams.Options().
Constructors that hand-pick options off the params struct drop it silently, and
some of those live in other repos, so the caller-side switch cannot cover the
fleet.

The skipped round trip is server->client->server before the handler runs, so the
server already holds the decision; the request bit is only authorization. Let the
server elect it too, for every queue rpc it handles. Queue is still re-checked,
and the outcome is still observed as ClaimSkipped.

Takes func() bool rather than bool to keep the same runtime revocability as the
client option.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two switches for one decision left the caller able to force an announcement at a
server that never opted in, and put the compatibility risk on the wrong end: a
server electing to skip would announce to callers predating the field.

Make the request bit an advertisement -- always set on a queue rpc, since a
caller built from this version can always accept an announcement -- and let the
server alone decide whether to make one. Both ends must now agree, so an older
caller is never sent an announcement it cannot read, and WithClientSkipClaim has
nothing left to configure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@erikhortsch erikhortsch changed the title server: let a server skip the claim for every queue rpc it handles server: elect the claim skip, and have clients advertise it Aug 20, 2026
erikhortsch and others added 3 commits August 21, 2026 11:36
CS-1992: a handler that errors on receipt publishes its response right behind
the announcement, on a different channel, and nothing orders their delivery.
When the response won the race, selection stashed it as a fallback and the
announcement then sent the caller off to wait on a channel already drained,
turning an instant error into a request timeout.

The stash exists for broadcast, where an early error is one server rejecting a
request it could not read and another may yet bid. On a queue rpc the responder
is the only server that received the request, so nothing else can ever arrive:
return the response the moment it appears. This also stops a queue caller from
waiting out the selection timeout to surface a malformed-request rejection, the
case with no announcement at all.

Reported in #123, which surfaced the stashed error once the announcement
arrived; resolving it at the response site keeps selection from consuming the
answer in the first place.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Field 9 carried the caller's election of the skip, which pre-election servers
honor unconditionally. A caller that advertises on the same number would switch
those servers on for every queue rpc, with no opt-in and no kill switch, since
they have no server-side election to revoke. At 10, the advertisement is
invisible to every server that predates it: they see field 9 absent and
negotiate. The reverse skew already degraded safely -- an old caller electing on
9 is simply negotiated with.

Field 9 is reserved so it cannot come back meaning something else.

Also catches pkg/client tests up to selectServer's queue parameter, which the
previous commit missed, and covers the queue/broadcast split directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
reserved 9 stays: protoc rejects any later field reusing the number, so it is
enforcement, not documentation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@erikhortsch
erikhortsch marked this pull request as ready for review August 21, 2026 21:12
@erikhortsch
erikhortsch requested a review from a team August 21, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant