Smart Turn v3 silently breaks when audio_in_sample_rate=8000 (Twilio/telephony)

[drew-royster]

Summary

Setting audio_in_sample_rate=8000 in PipelineParams (as recommended by the Twilio WebSocket integration guide) silently breaks Smart Turn v3 end-of-turn detection. The model receives 8kHz audio but its internal WhisperFeatureExtractor hardcodes sampling_rate=16000, causing it to interpret speech at 2x speed with shifted pitch. This leads to aggressive/incorrect turn predictions with no errors or warnings.

The Documentation Conflict

The Twilio guide recommends:

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
    ),
)

The Smart Turn v3 README states:

Smart Turn takes 16kHz PCM audio as input.

Since Smart Turn is auto-injected as the default stop strategy via UserTurnStrategies.__post_init__, anyone following the Twilio docs will unknowingly break turn detection.

Root Cause

1. TwilioFrameSerializer upsamples 8kHz μ-law to the pipeline's configured audio_in_sample_rate
2. When audio_in_sample_rate=16000 (the default), the serializer upsamples to 16kHz — Smart Turn works correctly
3. When audio_in_sample_rate=8000, the serializer outputs 8kHz PCM — Smart Turn receives 8kHz audio
4. LocalSmartTurnAnalyzerV3._predict_endpoint uses WhisperFeatureExtractor(sampling_rate=16000) with no internal resampling
5. The model processes 8kHz audio as if it were 16kHz — no error, no warning, just wrong classifications

Evidence

We ran A/B tests comparing Smart Turn predictions on the same utterances at 8kHz vs 16kHz:

• 6 out of 20 utterances had classification flips (Complete ↔ Incomplete)
• Max probability delta: 0.9391 (a confident "complete" at 8kHz flipped to confident "incomplete" at 16kHz)
• Mean turn duration dropped 51% in production (2.33s → 1.14s) when using 8kHz
• Phone numbers were fragmented across multiple turns — the model couldn't recognize digit sequences as incomplete at 8kHz

Community Impact

Multiple Discord threads and issues show users hitting this without identifying the root cause:

• "Chipmunk audio" reports with Twilio + OpenAI Realtime (audio at 2x speed)
• "Issues with Local Smart Turn Analyzer" — Smart Turn marking turns COMPLETE incorrectly
• "Premature turn finalization" — bot replying twice per utterance
• Telnyx users finding mysterious "2x mismatch" workarounds
• Support staff giving the correct workaround ("just don't set it") without explaining the Smart Turn connection

Suggested Fixes

1. Update Twilio/telephony docs to warn against setting audio_in_sample_rate=8000 when using Smart Turn, or recommend only setting audio_out_sample_rate=8000
2. Add a runtime warning in LocalSmartTurnAnalyzerV3 when set_sample_rate is called with a value != 16000
3. Add internal resampling in the Smart Turn analyzer to handle non-16kHz input gracefully

Workaround

Leave audio_in_sample_rate at the default (16000). Only set audio_out_sample_rate=8000 for telephony. The TwilioFrameSerializer handles the 8kHz → 16kHz upsampling internally.

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_out_sample_rate=8000,  # Twilio needs 8kHz output
        # Do NOT set audio_in_sample_rate=8000 — breaks Smart Turn v3
    ),
)

Environment

• pipecat-ai 0.0.103
• LocalSmartTurnAnalyzerV3 (bundled ONNX weights)
• TwilioFrameSerializer
• Silero VAD with stop_secs=0.2

[markbackman]

Setting audio_in_sample_rate=8000 in PipelineParams (as recommended by the Twilio WebSocket integration guide) silently breaks Smart Turn v3 end-of-turn detection. The model receives 8kHz audio but its internal WhisperFeatureExtractor hardcodes sampling_rate=16000, causing it to interpret speech at 2x speed with shifted pitch. This leads to aggressive/incorrect turn predictions with no errors or warnings.

I can confirm that Twilio running at 8khz when using Smart Turn works fine. I've tested the 07-interruptible.py example using this small change:

    task = PipelineTask(
        pipeline,
        params=PipelineParams(
            enable_metrics=True,
            enable_usage_metrics=True,
            audio_in_sample_rate=8000,
            audio_out_sample_rate=8000,
        ),
        idle_timeout_secs=runner_args.pipeline_idle_timeout_secs,
    )

and it works as expected.

You mention:

6 out of 20 utterances had classification flips (Complete ↔ Incomplete)

If there were an audio incompatibility, you simply wouldn't get a response at all.

You can try this for yourself by running the same example:
https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07-interruptible.py

Guidance on how to run the foundational examples:
https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/README.md#running-examples-with-other-transports

So the question is: what is different about your Pipeline that you're hitting this issue? Perhaps you can compare to the 07 example I shared.

[drew-royster]

Hey Mark! Me and my buddy Claude pulled down a subset of the turn model training data and were able to replicate the issue with this script:

pip install datasets pipecat-ai[local-smart-turn-v3]
python prove_smart_turn_sample_rate_bug.py

For each sample in pipecat-ai/smart-turn-data-v3.2-test, we run _predict_endpoint twice — once with the audio at 16kHz, once downsampled to 8kHz (simulating audio_in_sample_rate=8000):

Results (first 100 English samples)

Metric	16kHz	8kHz
Accuracy	98.0%	60.0%
Classification flips	—	38/100
Max probability delta	—	0.9820
Samples where 8kHz is more confident COMPLETE	—	79/100

Of the 38 flips, 16kHz was correct all 38 times. Every single flip was an INCOMPLETE utterance that 8kHz misclassified as COMPLETE.

A few examples

Sample	Ground Truth	16kHz	8kHz
00082f3f	INCOMPLETE	0.0177 (correct)	0.9875 COMPLETE (wrong)
00a9e2a5	INCOMPLETE	0.0069 (correct)	0.9889 COMPLETE (wrong)
026d8bb8	COMPLETE	0.6525 COMPLETE (correct)	0.4902 INCOMPLETE (wrong)

prove_smart_turn_sample_rate_bug.py

#!/usr/bin/env python3
"""
Evaluates Smart Turn v3 accuracy at 16kHz vs 8kHz using the published
pipecat-ai/smart-turn-data-v3.2-test dataset.

Usage:
    pip install datasets pipecat-ai[local-smart-turn-v3]
    python prove_smart_turn_sample_rate_bug.py
"""

import sys
import numpy as np
from loguru import logger

logger.remove()
logger.add(sys.stderr, level="WARNING")

from datasets import load_dataset
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3


def resample_linear(audio, from_rate, to_rate):
    if from_rate == to_rate:
        return audio
    new_len = int(len(audio) / from_rate * to_rate)
    indices = np.linspace(0, len(audio) - 1, new_len)
    return np.interp(indices, np.arange(len(audio)), audio).astype(np.float32)


def main():
    N = 100
    print(f"Loading model and streaming first {N} English samples from pipecat-ai/smart-turn-data-v3.2-test...\n")

    analyzer = LocalSmartTurnAnalyzerV3()
    ds = load_dataset("pipecat-ai/smart-turn-data-v3.2-test", split="train", streaming=True)

    results = []
    for sample in ds:
        if sample.get("language") != "eng":
            continue

        audio = np.array(sample["audio"]["array"], dtype=np.float32)
        sr = sample["audio"]["sampling_rate"]
        truth = sample["endpoint_bool"]

        if sr != 16000:
            audio = resample_linear(audio, sr, 16000)

        # Path A: 16kHz audio -> model (correct)
        r16 = analyzer._predict_endpoint(audio)

        # Path B: downsample to 8kHz -> model (simulates audio_in_sample_rate=8000)
        audio_8k = resample_linear(audio, 16000, 8000)
        r8 = analyzer._predict_endpoint(audio_8k)

        flip = r16["prediction"] != r8["prediction"]
        results.append((truth, r16, r8, flip, sample.get("id", "?")))

        gt = "COMPLETE" if truth else "INCOMPL."
        p16 = f"{r16['probability']:.4f} {'COMP' if r16['prediction']==1 else 'INC '}"
        p8 = f"{r8['probability']:.4f} {'COMP' if r8['prediction']==1 else 'INC '}"
        print(f"  {len(results):>3d}  {gt:>8s}  16k={p16}  8k={p8}  {'FLIP' if flip else ''}")

        if len(results) >= N:
            break

    n = len(results)
    acc_16k = sum(1 for t, r16, r8, f, _ in results if (r16["prediction"]==1)==t) / n * 100
    acc_8k = sum(1 for t, r16, r8, f, _ in results if (r8["prediction"]==1)==t) / n * 100
    flips = sum(1 for _, _, _, f, _ in results if f)

    print(f"\n{'='*50}")
    print(f"  Accuracy at 16kHz: {acc_16k:.1f}%")
    print(f"  Accuracy at 8kHz:  {acc_8k:.1f}%")
    print(f"  Flips: {flips}/{n}")

if __name__ == "__main__":
    main()

[markbackman]

Interesting. So, you're saying that the lower sample rate (8khz instead of 16khz) affects that accuracy of the smart-turn model, not "breaking" it, per say.

cc @marcus-daily for input since he's the master of the smart-turn domain.

[drew-royster]

yes exactly, just makes results worse. Noticeable if you try stress testing with pauses and spelling out numbers, names etc. it'll interrupt with default settings fairly reliably

[kollaikal-rupesh]

Fix submitted in #3857.

Root cause: _predict_endpoint() in LocalSmartTurnAnalyzerV3 has five hardcoded 16000 references — when running at 8kHz (e.g. Twilio), the Whisper feature extractor receives audio labeled as 16kHz but actually sampled at 8kHz, causing the model to perceive speech at 2x speed with shifted formants.

Fix: Added automatic resampling to 16kHz via numpy.interp before feature extraction, plus a one-time warning log when resampling is active. No new dependencies needed.

@markbackman would appreciate a review when you get a chance!

[kollaikal-rupesh]

Great analysis by @drew-royster, the 98% → 60% accuracy drop across the training dataset is very compelling. Worth noting that the resample_linear function in the test script uses the exact same numpy.interp approach that this fix applies in production. So the accuracy difference drew-royster measured is essentially what users would see without vs. with the resampling fix in #3857.

@markbackman to your point about it not fully "breaking" agreed, the model still returns predictions, but 38/100 flips (all INCOMPLETE → COMPLETE) means it aggressively cuts off users mid-sentence. That's the "interrupted while spelling names/numbers" behavior drew-royster described. The fix is lightweight (no new deps, ~15 lines of actual logic) and the resampling only kicks in when sample rate ≠ 16kHz.

Happy to wait for @marcus-daily's input — would love to know if there's a preferred resampling approach for the smart turn pipeline.

[marcus-daily]

Thanks @rupesh-svg, I've left a minor comment on that PR.

[drew-royster]

Thanks @marcus-daily and @rupesh-svg !

[markbackman]

@drew-royster how easy is it to run that benchmark on this branch to see if it improves things?

[drew-royster]

Significantly helps!

Testing with a sample size of 100 from the smart turn set where we downsample the 16khz set

59% accuracy on main
94% accuracy on 3857 (6% loss probably just attributable to downsampling/upsampling)

Thanks again!