CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models

CoSTA decouples an SDM's generation and synthesis token spaces by extending the generation vocabulary with BPE-compressed speech units, adapting the model, and deterministically expanding generated units back before the unchanged decoder, yielding about 1.7× speech-token generation speed.

Existing SDMs use the same speech-token space for autoregressive generation and synthesis, so long token sequences increase generation cost, complicate text-speech alignment, and slow responses. Lower-rate tokenizers can shorten sequences, but they may introduce a compression-quality trade-off and can require retraining or adapting the tokenizer, paired decoder, and SDM together while still coupling generation and synthesis. The paper asks whether generation and synthesis token spaces can be decoupled while preserving the original speech tokenizer and decoder.

CoSTA applies BPE to the original speech-token sequences and extends the SDM generation vocabulary with the added BPE units while retaining the original speech tokenizer and decoder. For each added BPE unit, the embedding and LM-head row are initialized by averaging the rows of its deterministic component original tokens. It then performs pronunciation adaptation on compressed TTS data with interleaved text and speech targets, compressing only speech tokens; the first stage trains embedding and LM head on 50% of data at 1.6×10−4, and the second unfreezes intermediate layers on the remaining 50% at 2.0×10−5. Dialogue adaptation follows on compressed speech-to-speech dialogue data and updates only LLM parameters while the speech encoder, adaptor, and decoder stay frozen. At inference, deterministic BPE expansion is applied only to the generated speech-token stream, and the recovered original-token sequence is fed to the unchanged speech decoder.

• CoSTA-ZH and CoSTA-EN achieve average generated-token compression of 66.81% and 70.17%, with relative expanded speech-token generation speed of 173.63% and 165.94% on Chinese and English benchmarks. §3.3
• At concurrency 8, CoSTA-ZH has 9.49 s full-response latency and 0.820 req/s throughput, compared with 15.56 s and 0.515 for Step-Audio 2 mini and 13.71 s and 0.601 for Vanilla-FT-ZH. §3.3
• CoSTA reduces the LLM decode component from 164.42 to 122.98 ms in Chinese and from 165.08 to 115.96 ms in English, corresponding to reductions of 25.2% and 29.8%; first-packet latency reductions are 9.4% and 9.6%. §C.5
• Removing dialogue adaptation raises generated-token compression from 66.73% to 81.05% and lowers G-Eval from 72.27 to 24.35; CoSTA-ZH has ASR-CER 1.51% and UTMOS 3.75. §4.3
• Increasing added vocabulary from 10K to 80K reduces generated-token compression from 76.66% to 67.50% and raises expanded speech-token speed from 78.67 to 103.03 tokens/s; at 160K, compression further decreases to 63.62% but speed reaches only 104.18 tokens/s, a 1.1% increase over 80K, while G-Eval decreases from 81.36 to 78.87. §C.9
• Human evaluation uses 60 queries, six evaluators, and 1,080 response-level evaluations; CoSTA has more wins than losses in every aggregate comparison, with ties forming the majority for fluency. §C.6

• Dialogue-adaptation data is constructed from synthesized speech-to-speech examples; the paper says future work may explore how synthetic and real conversational speech data complement each other.
• The method adopts BPE as a deterministic, lossless, and easily integrated compression method; other lossless speech-token segmentation strategies are left for future work.
• Evaluation focuses on general speech-to-speech dialogue and efficiency; finer-grained scenarios such as emotional dialogue, personalized speaking styles, and diverse acoustic environments are left open.
• The reported inference savings exclude one-time adaptation and data-preparation costs, which the paper notes should be considered alongside inference savings.

• Speech-token BPE compression shortens sequences with increasing vocabulary and follows an approximate log-linear trend over 10K–320K added units across eight tokenizer configurations, data sources, and languages. §1
• Held-out compression across data sources and languages retains the approximate log-linear trend with R² = 0.9942 – 0.9997 and RMSE 0.10 – 0.47 pp. §C.4
• The original speech-token vocabulary contains 6,656 tokens; CoSTA extends it with 80K additional BPE speech tokens and maps token IDs to Unicode for BPE training. §A
• Vocabulary extension initializes each added unit's embedding and LM-head row as the average of its component original rows, leaving original vocabulary parameters unchanged. §3.1
• Pronunciation adaptation is progressive: the first stage trains only embedding and LM head on 50% of data at 1.6×10−4; the second stage unfreezes intermediate layers on the remaining 50% at 2.0×10−5. §3.2
• Dialogue adaptation updates only LLM parameters while keeping the speech encoder, adaptor, and speech decoder frozen. §3.3
• At inference, deterministic expansion is applied only to the generated speech-token stream; original tokens remain unchanged and the expanded sequence feeds the unchanged speech decoder. §3.4
• On OpenS2S, CoSTA retains the native GLM-4-Voice tokenizer and decoder and achieves average generated-token compression of 78.83% in Chinese and 81.36% in English, improving average content scores and alignment while slightly lowering average UTMOS. §C.7