RePlay: Retrieval-Based Voice Playback for Multi-Turn spoken dialogue

What it claims

A full-duplex speech model already "knows" what it will say at the start of its turn. RePlay uses this to select a pre-recorded line in one step, with no ASR, no LLM decoding and no TTS. Probing PersonaPlex (a 32-layer Moshi-derived model) shows the upcoming response becomes linearly recoverable only at the frame where the turn-start <EPAD> token is fed back as input, from about layer 13–14 onward. RePlay keeps the first 16 layers, which halves LLM compute, and reads layer 15. In simulated job interviews over a 300-line set, it reaches 383 ms median latency with a 3.6% user-cutoff rate. Its intent agreement (0.83) is close to a GPT-4.1 cascade (0.85), but its exact-line recall is lower (R@1 0.35 vs 0.52).

How it works

flowchart LR
  U[User audio + played-line audio/text] --> L[PersonaPlex layers 0–15]
  L --> T{Head Turn: EPAD most probable?}
  T -- no: PAD/WRD --> U
  T -- yes --> E[Next frame: Head Embed projects layer-15 state]
  E --> R[Cosine vs frozen line embeddings]
  R --> P[Play recorded line]
  P -- audio on system channel, words at aligned timings --> U

• Probe: a ridge regression at each layer, at two candidate frames (<EPAD> predicted vs <EPAD> input), maps the hidden state to the nomic-embed-text embedding of the response. At "predict" the probe stays near chance at every layer. Training used 200 single-turn QA prompts, testing used 100.
• Heads on layer 15:
• Head Turn classifies each frame as PAD, EPAD or WRD.
• Head Embed runs one frame after EPAD and projects the state into the frozen nomic text-embedding space.
• No parameters are tied to individual lines, so adding a line only means embedding its text.
• Loss: a one-directional SigLIP loss plus a cosine alignment term. Negatives are all D1 lines plus 600 shared negatives: half nearest-neighbour, half uniform. Lines with cosine > 0.9 to the target are excluded.
• Fine-tuning:
• LoRA rank 256 on layers 0–9 and full fine-tuning on layers 10–15, to preserve robustness to real speech.
• EPAD class weight 9; τ initialised at 10 and b at −10.
• Sequences up to 90 s, trained on 8×A100 40 GB.

Evidence

Data:

• D1: 100 LLM-generated characters, 29,225 lines, 52,768 dialogues (1071 h), voiced with Chatterbox TTS plus WHAM! noise.
• D2: 2 h of simulated interviews with "Nobu", 300 lines across 30 intents.

Simulated evaluation: 100 dialogues per system, with an LLM interviewer and a GPT-4o-mini judge.

RePlay's p10–p90 latency spread is 106 ms, because retrieval happens at turn start.

Ablation (R@1 / IntentAgr):

User study: 14 participants.

• vs C (n=8): RePlay preferred 63% vs 12%, p=0.008.
• vs B (n=6): RePlay preferred 46% vs 21%, p=0.50, not significant. Users liked RePlay's pace but found B's replies slightly more appropriate.

How to use it

The page names no code, weights or licence. To reproduce, you need PersonaPlex, a line set embedded with nomic-embed-text, synthetic dialogues that select lines by ID, and roughly 2 h of in-domain fine-tuning data.

• New lines in existing intents can be added by embedding them, without retraining (UL result).
• New intents need their own training data.

Limits

• Synthetic evaluation: training and evaluation data are entirely synthetic. Mixing in real dyadic corpora gave no benefit.
• Weak judging: quality is judged by an LLM (GPT-4o-mini), and the user study is small, with Cascade A excluded.
• Exact-line accuracy: this is where RePlay clearly trails a large LLM.
• Narrow testing: only one user-driven interview domain was tested.
• Fixed responses: there is no generation fallback, so responses are limited to the line set.