flowcat-core: opt-in full-duplex (barge-in) support for the cascaded pipeline - #61
Open
rolandknight wants to merge 1 commit into
Open
flowcat-core: opt-in full-duplex (barge-in) support for the cascaded pipeline#61rolandknight wants to merge 1 commit into
rolandknight wants to merge 1 commit into
Conversation
The cascaded builder is half-duplex by design (TurnMute); this adds an opt-in duplex path with working barge-in, validated end-to-end over the str0m WebRTC transport (aiortc client, whisper.cpp STT, OpenRouter LLM with tool calls, Kokoro TTS): - FrameProcessor::on_interruption() hook (default no-op): the runtime intercepts Frame::Interruption (drain + forward) and never delivers it to process_frame, so the existing Interruption arms in sinks are unreachable. The hook gives processors a real delivery path. - VadProcessor: optional barge-in generation counter + Notify, bumped synchronously at detection. A busy process_frame (LLM mid-stream, TTS mid-synthesis) cannot be preempted by the frame path; these enable cooperative cancellation and an out-of-band reactor. - LlmProcessor: cooperative stream cancel between chunks on barge-in; closes response framing so aggregators cannot wedge open. - AssistantContextAggregator::on_interruption: keeps the partial reply in context, drops the open span (a late LlmResponseEnd from a cancelled stream can no longer speak the interrupted reply). - SpeechGate: VAD-edged speech segmentation (300 ms pre-roll, all-zero flush marker at the falling edge) so fixed-window batch STT gets one utterance per VAD turn instead of hallucinating turns on silence. - CascadedTransportOutput: emits BotStarted/StoppedSpeaking via a playout-tracking notifier (nothing armed the VAD barge-in gate on the cascaded path), flushes the carrier in on_interruption, and drops stale audio behind a reactor-armed latch. - build_cascaded_call_duplex: assembles the above; the stock builder is unchanged. Measured detection-to-flush: ~110us via the reactor vs 14ms-2.1s via the frame path (stalls behind mid-await hops). All existing unit tests pass (302); clippy -D warnings clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #60.
Adds an opt-in full-duplex path for the cascaded pipeline with working barge-in, validated live end-to-end (str0m WebRTC + aiortc client, whisper.cpp STT, OpenRouter LLM with tool calls, Kokoro TTS). The stock half-duplex builder and its
TurnMutebehavior are untouched; nothing changes unless you call the new builder.The problem (details in #60)
Frame::Interruption(drain + forward) and never delivers it toprocess_frame— the existingInterruptionarms in the transport sinks are unreachable, so frame-level barge-in cannot work on the cascaded path.BotStartedSpeaking/BotStoppedSpeaking, soVadProcessor's barge-in gate never arms.awaithop — we measured detection→sink delivery of 14 ms to 2.1 s depending on TTS/LLM activity — and an in-flight LLM stream cannot be cancelled, so the interrupted reply is spoken afterwards anyway.The changes
FrameProcessor::on_interruption()(new, default no-op)VadProcessor::{with_interrupt_flag, with_interrupt_notify}Notifybumped synchronously at detection (before the broadcast)LlmProcessor::with_interrupt_flagAssistantContextAggregator::on_interruptionLlmResponseEndcan no longer speak the reply)SpeechGate(new)SPEECH_GATE_FLUSH_SAMPLES) at the falling edge — without the turn lock, fixed-window batch STT hallucinates turns on silence and splits utterancesBotSpeakingNotifier(new) + sink wiringNotify-woken task flushes the carrier immediately (~110 µs from detection in our runs); the latch drops TTS audio that outran its interruptionbuild_cascaded_call_duplex(new, exported)VadAnalyzerso it stays feature-agnostic (caller passes e.g.SileroVad)Validation
cargo test -p flowcat-core --lib: 302 passed.cargo clippy -p flowcat-core --lib -- -D warnings: clean.cargo fmtapplied.Notes for review
process_frame), the hook + flag still stand alone.VAD_MIN_VOLUME = 0.6gated out moderate-volume speech entirely in our runs (only the loudest tail of utterances passed); we had to run with 0.2. Left untouched here since it's pipecat parity.🤖 Generated with Claude Code