Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Falcon is a single-pass causal audio-visual target speaker extractor paired with the RealSSA benchmark, achieving strong extraction quality with much lower parameter, computation, and latency costs than causal AV-TFGridNet.

Existing causal AVTSE benchmarks fix the number of speakers at two and ignore near-/far-field interference, reverberation, and sudden noise, and they lack real-recorded evaluation. High-performance causal architectures are too heavy for edge deployment, while lightweight alternatives repeatedly invoke a small separator and offset the savings. SI-SNR training does not explicitly push estimates away from near-field interferers, leaving residual leakage.

The paper releases RealSSA, with RealSSA-Sim providing controllable simulated scenes with 1–5 near-field interfering speakers and 10–15 far-field interferers when applicable, and RealSSA-Real providing a held-out real-recorded evaluation set. Falcon uses a frozen FRCNN visual backbone and an STFT audio encoder, refines audio and visual features before fusion, and predicts a complex time-frequency mask. Its Separator is an asymmetric U-Net that performs multi-scale time-frequency modeling in a single encoder–decoder pass with level-specific TFRBs. Training adds an encoder-space speaker contrastive loss that uses the estimated target as query and clean target plus near-field interferers as references, with stop-gradient on references. The loss is discarded at inference and can attach to other audio-encoder AVTSE backbones.

• Falcon cuts parameters by 91.9%, computation by 20.2x, and GPU inference latency by 11.1x relative to causal AV-TFGridNet. §Abstract
• On RealSSA-Sim, Falcon achieves 7.7 dB SI-SNRi versus 7.6 dB for AV-TF-GridNet. §6.1
• Falcon's SI-SNRi gain over Swift-Net-12 is largest on VoxCeleb2-2Mix, reaching 1.1 dB. §6.1
• On RealSSA-Real, Falcon obtains DNSMOS 3.276, tying Swift-Net-12 and exceeding AV-TF-GridNet at 3.030 and Dolphin at 2.960. §6.1
• On RealSSA-Sim, Falcon obtains the lowest extraction WER at 25.47%, improving on AV-TF-GridNet by 1.84 percentage points. §6.1
• Compared with Swift-Net-12, Falcon uses 10.6 G MACs versus 45.3 G, a 76.6% reduction, and GPU latency falls from 172.1 ms to 51.3 ms, a 70.2% reduction. §6.2

• The separator and encoder-space speaker contrastive loss are single-channel; the paper leaves a four-element array simulated in Appendix A.2 to future multi-channel extensions.
• RealSSA-Real covers approximately 84 minutes over 10 participants and eight environments; the paper says it can be scaled up in follow-up releases.
• The contrastive loss relies on clean per-source waveforms of near-field interferers available in simulation; adapting it to unlabeled real recordings is left future work.
• The missing-visual experiments leave sensitivity to mouth-crop displacement and tracking a different speaker to separate evaluation.

• RealSSA-Sim has four scene types and jointly samples RT60, near-/far-field geometry, environmental noise, and sudden noise; near-field interfering speakers vary from 1 to 5, with 10 to 15 far-field interferers when applicable. §3
• RealSSA-Real is a held-out real-recorded evaluation set used exclusively for evaluation and never for training or validation. §3
• The Separator is an asymmetric U-Net with time-frequency modeling at the encoder bottleneck and level-specific TF blocks at every decoder resolution; one pass yields four TF scales. §4.2
• The encoder-space speaker contrastive loss uses the estimated target as query and the clean target plus each near-field interferer as positive and negative references with stop-gradient; far-field interferers are excluded. §4.3
• The contrastive loss improved Falcon by 0.4 dB SI-SNRi, SwiftNet-12 by 0.2 dB, and Dolphin by 0.3 dB. §6.4
• Increasing downsampling depth from one to three stages improved SI-SNRi by 0.6 dB; three stages is the maximum depth permitted by streaming causality. §6.4
• RealSSA-Sim training improved DNSMOS for all five models and NISQA for four; Falcon NISQA rose from 2.431 to 3.615. §6.3
• Falcon has 0.5 M parameters, 10.6 G MACs, 51.3 ms GPU latency, 125.3 MB CPU RAM, and 364.7 MB GPU memory. §6.2