SSH KeepAlive blocking stdin?

[DenuxPlays]

Hey,

i have a weird issue.
We are currently working on a cross-platform ssh & sftp app and we are using dartssh2 for our SSH & SFTP connection.
We implement the terminal parsing ourself etc. and I have noticed a very weird issue.
When running a high throughput command like yes you then press ctrl + c the delay to the server is around 5 seconds.

I tried to debug the issue and the code hangs for around 5 seconds (almost exactly the input delay) here: https://github.com/TerminalStudio/dartssh2/blob/master/lib/src/ssh_keepalive.dart#L19

I am not sure if this is a bug in dartssh2 or at our end.

Does it even make sense that the code hangs there?
is keepalive not getting through because of yes blocking the streams?

I am just here to check and basically ensure that this is a bug at our end not in this library.

[vicajilau]

This behavior is a classic symptom of bufferbloat and backpressure bypass occurring under high-throughput conditions (such as running yes). It is not exactly a bug in the library, but rather how network socket buffers, Dart's Event Loop, and SSH flow control interact when the client falls behind.

Technical Breakdown

1. How keepalive works:
   The keepalive helper (SSHKeepAlive) schedules a periodic ping(). The ping() call sends a global request over the transport and then awaits the next response in _globalRequestReplyQueue.next.

2. The KeepAlive "Hang":
   When running a high-throughput command like yes, the server floods the socket with stdout channel data, filling the client's TCP receive buffers.
   The server receives the keepalive request and immediately replies, but this reply packet is queued in the TCP stream after the massive amount of stdout data already in flight.
   Since Dart is single-threaded, it must read, decrypt, parse, and process all the preceding stdout channel data from the socket before it can reach the keepalive reply packet in the stream. If the client's terminal parser/rendering is slower than the incoming stream, it will take several seconds to catch up, during which ping() remains suspended (hanging).

3. Why Ctrl + C is delayed:
   The exact same queueing happens in the socket. When you press Ctrl + C, the client sends the input to the server instantly. The server receives it and stops yes immediately. However, there are already megabytes of stdout data in flight in the TCP buffers. The client must finish reading and rendering this entire backlog before the terminal visually stops outputting text.

4. Bypassed SSH Flow Control:
   In SSHChannel._handleDataMessage, SSH flow control is bypassed at the application level. The incoming data is pushed to _remoteStream and WINDOW_ADJUST is immediately sent back to the server as soon as packets are read from the socket, rather than when the application layer actually consumes and renders those bytes. This allows the server to keep flooding the connection even if the client's rendering pipeline is falling behind.

---

Mitigations & Solutions

1. Implement Application-Level Backpressure:
   Since dartssh2 respects stream pause status (_remoteStream.isPaused), the correct way to handle high-throughput streams in your application is to pause your subscription to the channel stream (subscription.pause()) when your terminal emulator's render queue is overwhelmed, and resume it (subscription.resume()) once it clears. This stops dartssh2 from sending WINDOW_ADJUST packets, freezing the server's transmission and avoiding buffer bloat.

2. KeepAlive Robustness (PR fix(keepalive): prevent overlapping pings and catch errors #184):
   I've opened PR fix(keepalive): prevent overlapping pings and catch errors #184 to guard SSHKeepAlive against overlapping pings if the connection is busy, and to catch potential exceptions so that failed pings don't throw unhandled asynchronous errors:

   bool _isPinging = false;
   
   void start() {
     _timer ??= Timer.periodic(interval, (timer) async {
       if (_isPinging) return;
       _isPinging = true;
       try {
         await ping();
       } catch (_) {
         // Ignore errors, the client transport will handle disconnection.
       } finally {
         _isPinging = false;
       }
     });
   }

[DenuxPlays]

Thanks for the answer

We already have a byte queue and backpressure system in place.
In fact I noticed this when implementing these.

We only pause stdErr & stdOut is that correct?

I have not noticed any unhandled exceptions during debugging but I will test this pr and see if it improved things.

[vicajilau]

Yes, that is correct. Pausing session.stdout and session.stderr is the correct way to trigger backpressure on the SSH session. Under the hood, here is how the backpressure propagates in dartssh2:

1. Pausing either stdout or stderr triggers the session's internal _pauseChannelData(), which pauses the subscription (_channelDataSubscription) on the underlying SSHChannel.stream.
2. When the subscription is paused, the channel controller's _remoteStream.isPaused becomes true.
3. While paused, any incoming data packet will still decrement _localWindow, but _sendWindowAdjustIfNeeded() will check _remoteStream.isPaused and return early without sending a MSG_CHANNEL_WINDOW_ADJUST message back to the server.
4. Once the server exhausts the remaining channel window size, it will stop sending further data packets, effectively halting the flow.

Note: Because stdout and stderr share the same underlying SSHChannel, pausing either stream will pause the data subscription for the entire channel, as SSH flow control operates at the channel level rather than per-stream.

[DenuxPlays]

I did some more testing.
KeepAlive is actually not the problem and your PR while it does harden error handling it does actually not resolve this issue.

Our EventLoop is correctly setup and it instantly adds the input to the input.
After we give it to dartssh2 (and dartssh2 seems to just pass it directly to the stream) it waits for a still unknown reason.

It maybe dart directly idk

[DenuxPlays]

@vicajilau Is there a way to force flush for the sink?

[vicajilau]

@DenuxPlays I have updated the PR to add the requested socket flush support.

In dart:io, Socket implements IOSink, which buffers written data (sink.add()). It relies on an auto-flush microtask scheduled at the end of the current Event Loop iteration. However, when the Event Loop is heavily saturated (processing the massive flood of incoming socket read events from yes along with the application's terminal rendering microtasks), the auto-flush microtask gets starved. This leaves the input (like Ctrl+C) sitting in Dart's buffer instead of being sent to the OS TCP stack.

What is added in PR #185

I've added a .flush() method across the socket, transport, and channel API:

1. Future<void> flush() on SSHSocket (with a default empty body for backward compatibility).
2. Override on _SSHNativeSocket that calls await _socket.flush().
3. Delegation of flush() on SSHTransport, SSHClient, SSHChannel, and SSHForwardChannel.

This allows the client application to force-write the buffer to the network immediately:

channel.sink.add(utf8.encode('\x03')); // Send Ctrl+C
await channel.flush(); // Instantly write to the OS TCP stack, bypassing Event Loop delay