Audio Interaction Model
Zhifei Xie, Zihang Liu, Ze An, Xiaobin Hu, Yue Liao, Ziyang Ma, Dongchao Yang, Mingbao Lin, Deheng Ye, Shuicheng Yan, Chunyan Miao
arXiv:2606.05121 · 121▲ · cs.SD, cs.AI, cs.CL, cs.MM, eess.AS
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
The paper builds Audio-Interaction, a streaming audio language model (fine-tuned from Qwen2.5-Omni-3B) that consumes audio in 400ms chunks and predicts a per-chunk control token (<silent> or <response>) to decide whether to speak or keep listening. It is trained via a framework called SoundFlow using a 2.6M-item synthetic streaming corpus (StreamAudio-2M) and served with a FIFO-scheduled asynchronous inference scheme that decouples encoding from decoding.
Why it matters
Today's audio models are either offline (wait for the full clip, then answer once) or single-task streaming systems (ASR only, or voice chat only). This work folds transcription, translation, dialogue, and proactive interruption into one always-on model that judges in real time when to respond, which is closer to how a live assistant would need to behave. The practical payoff is lower latency and one model instead of several specialized pipelines.
Practical takeaway
Watch for unified streaming audio assistants that can proactively interject based on non-speech cues (a cough, an alarm) rather than only reacting after an utterance ends. The released StreamAudio-2M dataset and the chunk-level <silent>/<response> token formulation give a concrete recipe if you want to convert an offline audio model into a streaming one.
Key result
On MMAU under audio instructions the model scores 58.15 average, slightly above its Qwen2.5-Omni-3B base (57.81) and competitive with 7B systems, while ASR regresses modestly (LibriSpeech clean WER 3.17 vs base 2.87). The proactive-response claim rests on their own new Proactive-Sound-Bench (644 hand-designed events, 61.2 Single / 62.8 Multi tier), where they beat larger baselines but on a benchmark they authored themselves.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.