Skip to content

Stop reporting an ordinary disconnect as a send error - #977

Open
M-r-A wants to merge 1 commit into
rwmt:devfrom
romangr:pr/servertest-disconnect-flake
Open

Stop reporting an ordinary disconnect as a send error#977
M-r-A wants to merge 1 commit into
rwmt:devfrom
romangr:pr/servertest-disconnect-flake

Conversation

@M-r-A

@M-r-A M-r-A commented Aug 7, 2026

Copy link
Copy Markdown

ServerTest.Test fails intermittently — about 1 run in 140 idle, and 1 in 14
when the machine is under CPU load — always with the same message:

SendRaw() called with invalid connection state (127.0.0.1:PORT): Disconnected

Why it fails

Two independent state machines guard the same send:

Layer Guard Changed by
ConnectionBase.Send protocol state is not Disconnected the server thread, via PlayerManager.SetDisconnected
LiteNetConnection.SendRaw transport state is Connected LiteNetLib's own thread, asynchronously

The server only learns a peer has gone when OnPeerDisconnected is delivered,
and that happens at exactly one point per tick, inside
netManagers.ForEach(m => m.Tick()). The transport state changes whenever
LiteNetLib decides. A change landing just after that point leaves the rest of
the tick believing the peer is live — and TickNet broadcasts
ServerTimeControlPacket to every playing player unconditionally, every tick.

So Send approves the send, SendRaw finds the transport already gone, and
logs ServerLog.Error. ServerTest routes that into Assert.Fail, which is
what turns an ordinary client disconnect into a red build.

No teardown ordering can close this, because the state changes on a thread the
server does not control.

Why the guard is removed rather than corrected

NetPeer.Send on a peer that has gone away returns normally, throws nothing,
and logs nothing — LiteNetLib already discards it. I verified this directly
against the referenced version rather than assuming it.

So the guard suppressed nothing the library would not have suppressed anyway,
and existed only to report an unavoidable race as though it were a programming
error. SendRaw becomes the unconditional send; a seven-line method becomes
two.

What else is in this change

The flake sits in a test class that had two further problems, and fixing them
turned out to be a prerequisite rather than a nicety.

Two of the four tests asserted nothing. One waited for
Players.Count == 0 before its client had connected; the other for
Players.Count == 1 when it had already seeded that player itself. Both
conditions were true on the first poll, so the tests passed whether or not the
join they are named for ever happened — a broken loading state would still have
reported green. They now watch the client arrive and then leave, and the
keep-alive test asserts the join point it was blocked on has completed.

Making those waits real exposed a registry problem.
MpConnectionState.SetImplementation is additive and rejects a duplicate
handler for the same packet, which suits production registering each state once
at start-up. TestNetListener registers per connection with a per-test type, so
once two tests genuinely connect with different state classes it threw
Packet ClientJoining:Server_KeepAlive already has a handler and took the whole
test host down. This had stayed hidden precisely because those two tests
returned before their client finished connecting.

The polling waits are now a shared helper that reports what it observed
rather than bare Assert.Fail("Timeout"), and emits one measurement line per
wait on success as well as failure. That is what made the diagnosis possible:
the failures turned out to be confined to a single poll count, which is what
identified this as an ordering race rather than a slow machine.

Its budget also goes from 2000 ms to 5000 ms. The 2000 was never derived; the
longest wait measured over 1000 runs was 757 ms, so the margin is thinner than
it looks on a fast machine. A budget costs nothing when the condition holds,
because the wait returns on the poll that satisfies it.

Verification

Campaign Before After
100 iterations, 8 background load threads 7 failures 0 / 100
1000 iterations, idle 7 failures 0 / 1000

1100 runs, zero failures, zero timeouts.

Worth more than the failure count: every failure had occurred at one specific
poll count, and that bucket is still reached — 38 of 1000 runs — and now never
fails. A green run would also be produced by a test that stopped reaching the
window, so this distinguishes a fixed race from a test that stopped looking.

Notes for review

  • The production change is a deletion. The only production addition is
    MpConnectionState.ClearImplementation, which exists so a caller can replace
    a registration rather than only add one. Production still registers each state
    once at start-up and never calls it. I would rather add the seam than have the
    test suite depend on registration order, but it is the one line here that is
    arguably test-driven, and I am happy to take a different approach if you'd
    prefer.
  • ServerLog.error = Assert.Fail is deliberately kept. Silencing it during
    teardown was considered and rejected: it would suppress every genuine
    server-side error the suite can catch, in order to hide one misclassified
    line.
  • A send to a peer that is still connecting is now also unreported. That state
    was not exercised by the runs above, so it is the one behavioural gap I would
    flag.
  • The flake reproduces reliably under CPU load, so dotnet test on a loaded
    runner is the quickest way to confirm it independently.

LiteNetConnection.SendRaw refused to send unless the LiteNetLib peer was
Connected, and logged an error otherwise. The peer's transport state changes on
LiteNetLib's own thread, and ConnectionBase.Send has already approved the send
against the protocol state by the time SendRaw runs, so the two can disagree and
no check here can be race-free. LiteNetLib discards sends to a peer that has gone
away, silently and without throwing, so the guard suppressed nothing the library
would not have suppressed anyway.

ServerTest routes ServerLog.Error into Assert.Fail, so that log line failed the
suite whenever a client disconnected while the server was still ticking: 7 runs
in 1000 idle and 7 in 100 under CPU load, against zero in 1100 runs afterwards.

Two tests in the same class also asserted nothing. One waited for
Players.Count == 0 before its client had connected, the other for
Players.Count == 1 when it had already seeded that player itself; both returned
on their first poll and passed whether or not the join happened. They now watch
the client arrive and then leave.

Making those waits real exposed a further problem: SetImplementation is additive
and rejects duplicate handlers, which suits registering each state once at
start-up, but the test listener registers per connection with a per-test type, so
two tests using different state classes threw and took the test host down.
ClearImplementation lets a caller replace a registration; production still
registers once and never calls it.

The shared wait helper's budget goes from 2000 ms, which was never derived and is
thin on slower CI runners, to 5000 ms. The longest wait measured was 757 ms.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant