9B end-to-end model that sees, listens and speaks at the same time. Core idea: Omni-Flow — chop time into 1 s chunks and serialize every chunk as [vision ; audio ; output] into one causal sequence, so a plain LLM decides each second whether and what to say. TAIL keeps the spoken audio from lagging behind what the model currently sees.
1 · Architecture — click any block
Encoders → one causal LLM → a small speech-token decoder → a streaming vocoder. Everything except the flow-matching vocoder is connected at the hidden-state level and trained end-to-end (~9.34B params). Press Run one chunk to watch one 1-second Omni-Flow step travel through the network.
Colour by training stage:
Click a block for its role, numbers and why it matters. Colours: visionaudiotext / LLMspeech outstreaming controlprompt
2 · Omni-Flow, live
A 16-second illustrative session (cooking assistant). Each column is one 1 s chunk k. Inside a chunk the model first ingests the new frame and audio, then emits [listen] or [speak] + text. Watch three behaviours a turn-based model can't do: responding without a VAD, proactive speech (chunk 9, nobody asked), and barge-in (chunk 11).
Speed:
Scenario is illustrative, written for this page — not a demo from the paper.
↑ The single causal sequence the LLM actually sees (latest tokens on the right). ⟨b⟩ = explicit chunk boundary token.
Back-of-envelope (mine): one chunk ≈ 1 boundary + 64 visual + 10 audio + 1 control + ~3 text ≈ 79 tokens. 40,960 ÷ 79 ≈ 520 chunks ≈ 8–9 min of 1-fps streaming before the context window fills. The paper doesn't say how long sessions are handled (eviction, frame dropping, summarization).
2b · Listen or speak? How the decision is made
There is no VAD and no separate turn-taking network. After reading a chunk's new tokens, the LLM just predicts its next token, and special tokens in its vocabulary carry the decision. The paper compares two ways to set that up (§3.3, Table 1). Pick a formulation and a situation, then step through decoding.
Formulation:
What's in this chunk's audio:
Chunk k as the LLM sees it: inputs it has read, then tokens it has emitted
Probabilities are illustrative, not from the paper. LS shows only the two control tokens: the paper doesn't say whether other tokens are masked at that position or simply learned to be ~0.
Why LS wins (Table 1, 1 s chunks): IFEval 0.29 vs 0.24, SDQA 0.36 vs 0.35, MMLU 0.65 vs 0.56. In LT, the probability of "say something" is split over thousands of possible first words, while "stay silent" is one token, so silence can win the argmax even when speaking is more likely overall. LS makes whether-to-speak a clean binary decision and keeps what-to-say a normal LM step.
Same output layer, one extra word
Both ways use the same output layer: the one the model always uses to pick its next word. Nothing new is added to the network. The only change is what the model writes:
LT: … a×10 | [listen] or … a×10 | Add the onions LS: … a×10 | [listen] or … a×10 | [speak]Add the onions
Why one extra word helps:
There is a fixed moment to decide. Right after reading the new second of video and sound, the model must write one of two words: [listen] or [speak]. In training, only these two words ever appear in this spot. So the model learns: "here I just answer yes or no".
Every "yes" looks the same. In LT, "yes, I'll talk" is written as some first word: "Add", "Sure", "Hi", and so on. So "talk" is spread over many different words, while "stay quiet" is always the same one word. In LS, every "talk" is the same word: [speak].
First decide, then pick the words. After [speak], the model only thinks about what to say. It never has to choose between "stay quiet" and one particular word.
The paper doesn't say whether all other words are blocked at the decision spot, or whether the model simply learns to give them ~0 chance.
Where the training targets come from
Every duplex training clip has time-stamped video, user audio, and the assistant's reference text + speech. Cutting it into 1 s chunks gives the targets directly: a chunk where an assistant word starts gets [speak] + those words; every other chunk gets [listen]. The same start-time rule is what TAIL uses (section 3).
The rule "no output → oᵏ = [listen]" and the start-time assignment are from §3.2 and §3.4; drawing them on one clip is my reconstruction.
3 · TAIL — keeping speech on the clock
What TAIL controls: how many words the model writes in each 1-second chunk. Every word it writes is then spoken out loud, and speaking takes time. Write too many → the speaker falls behind, and you hear old things. Write too few → the speaker runs out and goes quiet. TAIL aims for: seconds of speech written so far ≈ seconds that have passed. So a chunk gets three short words, or just one long one.
Is TAIL a neural network? How does it pick the words?
No separate network, and no rule that runs while talking. TAIL is a way of preparing the training data. The words are picked by the normal LLM (Qwen3-8B).
Making the training data: take real recordings where we know when each word starts being spoken. Cut time into 1-second boxes. Put each word in the box where it starts. Example: "Add" starts at 0.0 s, "the" at 0.3 s, "chopped" at 0.45 s, "onions" at 0.85 s → all four go in box 0. "and" starts at 1.35 s → box 1.
Training: the model learns to copy this. For chunk 0 it writes "Add the chopped onions", then the "end of chunk" marker, and then it waits for the next second of input.
While talking: nothing measures time. The model has simply seen many examples of how long words take to say, and how far ahead the speech already is. So it knows when to stop writing for this chunk.
In the demo below I apply the box rule directly, as a stand-in for what the model has learned. One word = one token here; the real model may split a long word into pieces (e.g. "chop" + "ped").
The paper says chunks are separated by marker tokens. That the model ends its chunk by writing that marker is my reading; the paper doesn't spell out this step.
Step by step: one sentence, chunk by chunk
Strategy:
How to read this chart (example: chunk 0).
Writing is fast, speaking is slow. At the start of chunk 0 the model writes all 4 words ("Add the chopped onions") in a split second. Saying them takes 1.35 s.
So speech spills into the next second. The voice starts at 0.15 s (a small delay I assume for writing + making sound). 0.15 + 1.35 = 1.50 s. That is why "onions" is heard in the chunk-1 column, even though it was written in chunk 0. This is normal: the box rule only cares where a word starts in the planned speech ("onions" starts at 0.85 s).
"+0.50 s" = how much is still left to say when the chunk ends. At the 1.0 s mark, 0.5 s of already-written speech is still waiting (most of "onions"). It is how far the voice is behind the clock. If the user interrupts right then, at most 0.5 s of old speech is still coming.
What matters is that this number does not grow. With TAIL it stays around 0.2–0.5 s. With "Fixed 4" it climbs every second (the voice keeps falling further behind). With "Fixed 2" it goes negative and the voice goes quiet ("runs dry").
Speech-decoder input inside chunk (click a chunk column above). Each text token is followed by its S3 speech tokens (25 per second of audio): text and speech are interleaved.
Look-ahead, in simple words.
Why: how you say a word can depend on the next word ("thuh car" vs "thee apple"). At the end of a chunk, the next word isn't written yet.
What happens: the text of the last word(s) of chunk k is written in chunk k as normal. Only their sound is held back. The other words of chunk k are spoken right away.
Then: chunk k+1 starts, the model writes the next word(s), and now the speech decoder makes the sound for the held word, knowing what follows. After that it continues with the new words. It is not one big blob spoken at the end.
Cost: the held word waits only until the next chunk begins and its first new word is written. It does not wait for the whole next chunk's speech. If older speech is still playing (e.g. "+0.50 s left to say"), the wait is hidden and you hear no pause. If not, there is a short silence (the striped bits in the chart when look-ahead is on).
The paper says the speech of "the last few" text tokens is moved to chunk k+1. The exact order inside chunk k+1 is my reading. The demo holds back just one word.
At inference time: what TAIL buys you (my example, not from the paper)
Two moments. Don't mix them.
Training: TAIL lives here. The data is cut into 1-second boxes. The model learns a habit: write about 1 second of speech per second.
Inference (using the trained model): the demo below lives here. We watch the habit pay off.
The story. The model is describing a street, live. At 6 s, a dog runs into the road. The model sees it and writes "oh — a dog…" right away. But you can only hear it after all older words are spoken. Think of words waiting in a line at the speaker.
(a) Text lead: writes far ahead, not chunk by chunk. Long line. You hear about the dog ~4.5 s late, while the voice still talks about traffic lights.
(b) Fixed ratio: chunk by chunk, but the same number of words each time. The line slowly grows. ~1.6 s late.
(c) TAIL-trained model: chunk by chunk, about 1 s of speech each time. The line stays short. Under 1 s late.
How to read it: each block is a word, placed where it is heard. The thin line goes back to when it was written. Slanted line = waiting in line. Red-dashed = old news (written before the dog, heard after). Press Pause and toggle look-ahead to play with it.
Strategy
How much text per chunk
Dog first heard
Stale words after event
Now heard
How TAIL is supervised: from word-level timestamps in duplex training data, every text token whose start time falls in [(k−1)t, kt) — plus its speech tokens — is assigned to chunk k. So the number of text tokens per chunk isn't fixed; the model learns it from accumulated playback progress. Look-ahead exists because pronunciation needs the next word ("the apple" vs "the car").
Catch: TAIL costs quality. SeedTTS EN WER: fixed interleave 2.38 → TAIL 3.93 (+65% rel.); ZH CER 0.86 → 1.04. The headline TTS numbers in Table 5 (0.86 / 2.38) match the fixed-text mode, not the duplex mode.
Hover a word block for its generation/playback times. Blue underline = speech deferred by look-ahead (toggle it off to see TAIL's pure alignment; look-ahead buys pronunciation context at the cost of a little lag). Simulation uses a toy word-duration model (0.2 s + 0.05 s/char), 1 s chunks, 3 words/chunk for fixed ratio, 4 s lead for text-lead. Shapes are illustrative, not measured.
4 · Training pipeline & the length reward
Starts from the MiniCPM-V 4.5 pretraining checkpoint (vision + Qwen3-8B already aligned). Speech is bolted on gently first, then everything is unfrozen. Click show on diagram to colour section 1 by trainable/frozen.
Smooth length reward (RL stage) — play with it
si = (0.5 − (ℓi−ℓmin)/(ℓmax−ℓmin)) · min(1, (ℓmax−ℓmin)/τ) rlen = si if correct · min(0, si) if wrong
Scale factor min(1, spread/τ) = . Groups whose responses are all similar length get almost no length signal → avoids noisy pressure. Wrong answers can only be penalized for length, never rewarded for being short. Disabled for the first 480 steps.
Result (Table 9): Kimi-K1.5-style reward cuts thinking length 50.7% but drops avg 73.5 → 73.0; this smooth version cuts 35.3% and raises avg to 74.3. Less aggressive = less conflict with the accuracy reward.
5 · Omni-Flow design ablations (Table 1)
Three knobs: chunk size, explicit vs implicit chunk boundaries, and control formulation — LS (predict [listen]/[speak] first, then text) vs LT ([listen] competes with ordinary text tokens in one softmax).
Compare:
Why tiny chunks break (my read)
Numbers worth remembering
54.4
LiveSports-3K-CC win rate (vision-only duplex) vs StreamingVLM 45.6, LiveCC 41.5
OmniDocBench EN edit dist ↓ — best by far (next 0.214)
0.21 RTF
llama.cpp-omni INT4 on RTX 4090, 11 GB (PyTorch INT4: 1.26)
82.1 vs 81.6
Text avg vs its own Qwen3-8B backbone — omni training didn't erode language
6 · Critical read
The full-duplex claim is under-evaluated. The only duplex benchmark is LiveSports-3K-CC — vision-only commentary, scored by win rate. No audio barge-in, turn-taking latency, or false-interrupt metrics. The headline capability is shown mostly via demos.
"Full-duplex" = time-division multiplexing at 1 s. Decisions happen once per chunk, so reaction latency has a ~1 s floor plus perception→playback delay. Smaller chunks were tried and collapsed (Table 1).
Table 1 is noisy. 0.2 s scores AlpacaEval 1.22 but 0.1 s scores 2.40 — non-monotonic. Treat the chunk-size conclusion as directional; training budget per config isn't stated.
LS > LT is the cleanest finding. Decoupling whether to speak from what to say (separate control token before content) beats a shared vocabulary. Explicit boundaries also help. Both cheap to adopt.
Speech quality headline ≠ duplex mode. Table 5 numbers equal the fixed-interleave row; TAIL (the duplex mode) is notably worse in EN WER.
Efficiency comes from two compressions, not new ideas: 16× visual token compression (64 tok/slice vs typical 256) and keeping the 8B backbone in text space (3–4 decode steps/s instead of 25 speech tok/s). Thinker/talker-style split, but with hidden-state + text-embedding summation and time alignment.
From a video-representation angle: the streaming visual path is per-frame image encoding (SigLIP + resampler) — no temporal modeling in the encoder; all temporal reasoning lives in the LLM over a growing KV cache. Obvious room for a streaming video encoder / temporal token compression, and for context management beyond ~9 minutes.
Proactivity is supervised, not emergent. It comes from hand-built full-duplex scenarios (reminders, scene description) in SFT; the paper itself calls proactive behaviour "relatively simple".
Reference
Everything on this page explains, and where marked, critiques the following paper. Numbers, tables and section references (§, Table N) are from it; scenarios, simulations and notes marked "mine" are my own illustrations.
@article{cui2026minicpmo45,
title = {MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction},
author = {Cui, Junbo and Xu, Bokai and Wang, Chongyi and Yu, Tianyu and Sun, Weiyue and Xu, Yingjing and Wang, Tianran and He, Zhihui and Ma, Wenshuo and Cai, Tianchi and Gui, Jiancheng and Zhang, Luoyuan and Sun, Xian and Huang, Fuwei and Chen, Moye and Lin, Zhuo and Liu, Hanyu and Gui, Qingxin and Han, Qingzhe and Wen, Yuyang and Liu, Huiping and Wang, Rongkang and Zhang, Yaqi and Wei, Hongliang and Chen, Chi and Li, You and Fang, Kechen and Zhou, Jie and Li, Yuxuan and Zeng, Guoyang and Xiao, Chaojun and Lin, Yankai and Han, Xu and Sun, Maosong and Liu, Zhiyuan and Yao, Yuan},
journal = {arXiv preprint arXiv:2604.27393},
year = {2026},
url = {https://arxiv.org/abs/2604.27393}
}