Skip to content

flowcat-core: opt-in full-duplex (barge-in) support for the cascaded pipeline - #61

Open
rolandknight wants to merge 1 commit into
AreevAI:mainfrom
rolandknight:cascaded-full-duplex
Open

flowcat-core: opt-in full-duplex (barge-in) support for the cascaded pipeline#61
rolandknight wants to merge 1 commit into
AreevAI:mainfrom
rolandknight:cascaded-full-duplex

Conversation

@rolandknight

Copy link
Copy Markdown

Closes #60.

Adds an opt-in full-duplex path for the cascaded pipeline with working barge-in, validated live end-to-end (str0m WebRTC + aiortc client, whisper.cpp STT, OpenRouter LLM with tool calls, Kokoro TTS). The stock half-duplex builder and its TurnMute behavior are untouched; nothing changes unless you call the new builder.

The problem (details in #60)

  1. The runtime intercepts Frame::Interruption (drain + forward) and never delivers it to process_frame — the existing Interruption arms in the transport sinks are unreachable, so frame-level barge-in cannot work on the cascaded path.
  2. Nothing on the cascaded path emits BotStartedSpeaking/BotStoppedSpeaking, so VadProcessor's barge-in gate never arms.
  3. Even delivered, the frame path stalls behind any mid-await hop — we measured detection→sink delivery of 14 ms to 2.1 s depending on TTS/LLM activity — and an in-flight LLM stream cannot be cancelled, so the interrupted reply is spoken afterwards anyway.

The changes

Piece What it does
FrameProcessor::on_interruption() (new, default no-op) Runtime calls it in the Interruption arm after draining — the real delivery path for barge-in reactions
VadProcessor::{with_interrupt_flag, with_interrupt_notify} Generation counter + Notify bumped synchronously at detection (before the broadcast)
LlmProcessor::with_interrupt_flag Cooperative cancel between streamed chunks; closes response framing on cancel
AssistantContextAggregator::on_interruption Keeps the partial reply in context, drops the open span (late LlmResponseEnd can no longer speak the reply)
SpeechGate (new) VAD-edged segmentation: 300 ms pre-roll at the rising edge, all-zero flush marker (SPEECH_GATE_FLUSH_SAMPLES) at the falling edge — without the turn lock, fixed-window batch STT hallucinates turns on silence and splits utterances
BotSpeakingNotifier (new) + sink wiring Emits bot-speaking edges into the pipeline head from playout tracking; arms the VAD gate
Out-of-band interrupt reactor + stale-audio latch Notify-woken task flushes the carrier immediately (~110 µs from detection in our runs); the latch drops TTS audio that outran its interruption
build_cascaded_call_duplex (new, exported) Assembles all of the above; generic over VadAnalyzer so it stays feature-agnostic (caller passes e.g. SileroVad)

Validation

  • Live barge-in test (long spoken reply interrupted mid-playback over WebRTC): bot audio stops within the harness' 1 s budget consistently; interrupted reply does not resume; the follow-up turn (a tool call) executes normally. Repeated 3× + a 7-test regression suite, all green.
  • cargo test -p flowcat-core --lib: 302 passed. cargo clippy -p flowcat-core --lib -- -D warnings: clean. cargo fmt applied.

Notes for review

  • The reactor + latch exist because of measured need (the 2.1 s frame-path stall); if you'd rather solve preemption differently (e.g. cancellable process_frame), the hook + flag still stand alone.
  • One default worth a look while you're here: VAD_MIN_VOLUME = 0.6 gated out moderate-volume speech entirely in our runs (only the loudest tail of utterances passed); we had to run with 0.2. Left untouched here since it's pipecat parity.
  • Happy to split this into smaller PRs (hook/runtime fix first) if preferred.

🤖 Generated with Claude Code

The cascaded builder is half-duplex by design (TurnMute); this adds an
opt-in duplex path with working barge-in, validated end-to-end over the
str0m WebRTC transport (aiortc client, whisper.cpp STT, OpenRouter LLM
with tool calls, Kokoro TTS):

- FrameProcessor::on_interruption() hook (default no-op): the runtime
  intercepts Frame::Interruption (drain + forward) and never delivers it
  to process_frame, so the existing Interruption arms in sinks are
  unreachable. The hook gives processors a real delivery path.
- VadProcessor: optional barge-in generation counter + Notify, bumped
  synchronously at detection. A busy process_frame (LLM mid-stream, TTS
  mid-synthesis) cannot be preempted by the frame path; these enable
  cooperative cancellation and an out-of-band reactor.
- LlmProcessor: cooperative stream cancel between chunks on barge-in;
  closes response framing so aggregators cannot wedge open.
- AssistantContextAggregator::on_interruption: keeps the partial reply
  in context, drops the open span (a late LlmResponseEnd from a
  cancelled stream can no longer speak the interrupted reply).
- SpeechGate: VAD-edged speech segmentation (300 ms pre-roll, all-zero
  flush marker at the falling edge) so fixed-window batch STT gets one
  utterance per VAD turn instead of hallucinating turns on silence.
- CascadedTransportOutput: emits BotStarted/StoppedSpeaking via a
  playout-tracking notifier (nothing armed the VAD barge-in gate on the
  cascaded path), flushes the carrier in on_interruption, and drops
  stale audio behind a reactor-armed latch.
- build_cascaded_call_duplex: assembles the above; the stock builder is
  unchanged. Measured detection-to-flush: ~110us via the reactor vs
  14ms-2.1s via the frame path (stalls behind mid-await hops).

All existing unit tests pass (302); clippy -D warnings clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cascaded path: barge-in is unreachable — Interruption never delivered to process_frame, VAD gate never armed

1 participant