TurnSense.cpp

A production C/C++ runtime for TurnSense, a semantic end-of-utterance detector for voice agents. It converts the upstream FP32 ONNX model to deterministic GGUF and runs a model-specific native graph using official upstream ggml. ONNX is not present in the production library.

Features

• Native SmolLM2 classifier graph over canonical F32 or selective Q8_0 GGUF weights
• Exact embedded GPT-2 byte-level BPE tokenizer and required <|user|> prompt
• Stable C ABI and command-line tool
• CPU, Vulkan, and Metal backends with CPU fallback scheduling
• Strict GGUF metadata, tensor, tokenizer, type, and shape validation
• Deterministic LoRA merge, ONNX-to-F32-GGUF conversion, and native Q8_0 quantization
• Independent FP32 ONNX Runtime reference fixtures
• Installable static/shared CMake package

The production graph uses only official ggml operations and backend scheduling; there are no ggml_map_custom* callbacks or CPU-only custom graph operations. The same graph runs through ggml's CPU, Vulkan, and Metal backends. Vulkan has been validated on an AMD Radeon RX 7900 XTX; Metal is the default GPU backend on macOS. For Q8_0 models, the runtime also checks that every quantized get_rows and mul_mat node remains on the selected upstream backend rather than unexpectedly falling back.

Build

Initialize dependencies, then build:

git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
ctest --test-dir build --output-on-failure

Vulkan is enabled by default on non-Apple platforms. Metal is enabled by default on Apple platforms, with its shader library embedded in the binary. Disable both for a CPU-only build:

cmake -S . -B build-cpu -G Ninja \
  -DTURNSENSE_VULKAN=OFF -DTURNSENSE_METAL=OFF

Useful options:

Option	Default	Purpose
TURNSENSE_VULKAN	ON except Apple	Build ggml's Vulkan backend
TURNSENSE_METAL	ON on Apple	Build ggml's Metal backend
TURNSENSE_BUILD_CLI	ON	Build turnsense_cli
TURNSENSE_BUILD_QUANTIZER	ON	Build turnsense_quantize
TURNSENSE_BUILD_TESTS	ON	Build tests and parity helper
TURNSENSE_INSTALL	ON	Generate install/package targets
TURNSENSE_WERROR	OFF	Treat warnings in project sources as errors
TURNSENSE_MODEL_PATH	empty	Enable model-dependent CTest cases
TURNSENSE_QUANTIZED_MODEL_PATH	empty	Enable accelerated Q8_0 parity CTests

Obtain and convert the model

Download the immutable artifacts listed in docs/model-provenance.md. Conversion requires Python 3, NumPy, and ONNX but production inference does not.

python -m venv .venv
.venv/bin/pip install -r tools/converter/requirements.txt

# F32 ONNX -> canonical merged F32 GGUF
.venv/bin/python -m tools.converter.convert \
  --model-dir models \
  --output models/turnsense-f32.gguf

# Canonical F32 GGUF -> selective Q8_0 GGUF using upstream ggml
./build/turnsense_quantize \
  models/turnsense-f32.gguf \
  models/turnsense-q8_0.gguf

The Python converter emits F32 only, verifies pinned input hashes by default, and rejects incompatible graph structures. The native quantizer accepts only a strictly valid canonical F32 GGUF and refuses to overwrite an existing output. It stores the embedding and 210 transformer projection matrices as Q8_0 while keeping 61 normalization vectors and the classifier F32. It calls upstream ggml_quantize_chunk; there is no project-specific quantization implementation.

Pinned deterministic outputs from the current converter are:

Model	Size	SHA-256
turnsense-f32.gguf	540,047,168 bytes	0eb7d0ea7bccbcd2f9aa4b722ee202efbf53b4e430ba90c527548f8578d2460e
turnsense-q8_0.gguf	145,012,704 bytes	e978f2462e1887c2959173066e2f534f3c02fea686323bfc454aa37842970966

Repeated native quantization is byte-identical. Q8_0 tensor payload is 136.40 MiB versus 513.14 MiB for F32, a 3.76x reduction. Model files remain ignored by Git.

CLI

./build/turnsense_cli --model models/turnsense-f32.gguf \
  --backend cpu --json "I think we should go now."
# {"non_eou":...,"eou":...,"label":1}

echo "I think we should" | ./build/turnsense_cli \
  --model models/turnsense-f32.gguf --backend vulkan

# On macOS:
./build/turnsense_cli --model models/turnsense-f32.gguf \
  --backend metal --json "I think we should go now."

label=0 means NON_EOU; label=1 means EOU. Applications should generally use prob_eou with a product-specific threshold rather than treating the model's argmax as an unchangeable policy.

C API

#include <turnsense.h>
#include <stdio.h>

turnsense_params params = turnsense_default_params();
params.model_path = "turnsense-f32.gguf";
params.backend = TURNSENSE_BACKEND_AUTO;
params.n_threads = 4;

turnsense_ctx *ctx = turnsense_load(&params);
if (!ctx) {
    fprintf(stderr, "%s\n", turnsense_last_error());
    return 1;
}

turnsense_result result = turnsense_predict_text(ctx,
    "Could you send that report tomorrow?");
printf("EOU probability: %.6f\n", result.prob_eou);
turnsense_free(ctx);

turnsense_predict_text automatically prepends the exact prompt used upstream. turnsense_tokenize and turnsense_predict_tokens are available for systems that cache or manage tokens themselves. A custom tokenizer callback can override the embedded tokenizer, but the caller then owns exact prompt compatibility.

Each context is reusable and internally serializes concurrent calls. Do not free a context while a call using it is active. For parallel inference rather than serialized access, use one context per worker; each context owns a model copy, so size the pool deliberately for the selected model and backend.

Voice-agent integration

TurnSense operates on text, not audio. Feed it the latest punctuated STT utterance when VAD indicates a candidate pause:

1. Keep collecting partial STT text while speech is active.
2. At a candidate pause, pass the best current punctuated transcript to turnsense_predict_text.
3. Combine prob_eou with VAD duration, latency budget, and application policy.
4. If below threshold, continue listening and evaluate the updated transcript.

The upstream model is punctuation-dependent and English-focused. Do not append <|im_end|>; the runtime applies only the documented literal <|user|> prefix.

Validation

Generate immutable ONNX Runtime fixtures:

.venv/bin/python tools/generate_onnx_reference.py

Compare the native F32 runtime with its source model:

python validate_parity.py
python validate_parity.py --helper build-vulkan/test_parity_helper \
  --vulkan --fp32-tolerance 1e-3

# Also compare Q8_0 directly against the native F32 baseline
python validate_parity.py --helper build/test_parity_helper \
  --quantized models/turnsense-q8_0.gguf --quantized-tolerance 0.05

Measured maximum absolute probability differences on the bundled fixtures:

Backend / GGUF	Reference	Max delta	Labels
CPU F32	FP32 ONNX	9.84e-7	identical
Vulkan F32	FP32 ONNX	5.69e-4	identical
CPU Q8_0	native CPU F32	0.04161	identical
Vulkan Q8_0	native Vulkan F32	0.01413	identical

The Q8_0 gate is 0.05 absolute probability drift plus classification parity. The dynamically quantized ONNX graph is retained only as an independent reference artifact; it is not a source for production GGUF quantization.

Installation

cmake --install build --prefix /usr/local

Consumers can use:

find_package(turnsense CONFIG REQUIRED)
target_link_libraries(my_agent PRIVATE turnsense::turnsense)

For static installs, the package installs and resolves the pinned ggml package as a transitive dependency. Shared installs expose only the documented TurnSense C symbols.

Licenses

Project code is MIT licensed. Upstream TurnSense model artifacts are Apache-2.0. See THIRD_PARTY_NOTICES.md for pinned dependency and model notices.