Finds the sounds around the words: non-verbal vocal sounds (breath, cough, laugh, sigh, …) and
background sounds (music, traffic, birds, crowd, …) with start and end times, and writes them into the transcript:
[cough] where the speaker makes a sound and <music> … </music> around the words a background sound overlaps.
| features | 16 kHz mono audio → 64-band log-mel spectrogram |
|---|---|
| model | CNN front end (5 blocks) + Transformer (6 layers, d=256, 8 heads), 5.8M parameters; one probability per tag every 40 ms |
| long audio | 8 s windows every 2 s; overlapping predictions are averaged, so times are absolute |
| events | 5-frame median filter, per-tag thresholds with hysteresis (an event continues while the score stays above 60% of the threshold), gap joining, 0.15 s minimum length |
| thresholds | sensitive: tuned for recall (F2), catches more sounds and adds more false tags. balanced: best F1 per tag |
| transcript | Whisper on chunks of up to 28 s cut at pauses (a sentence that runs into a cut moves to the next chunk), language detected per chunk, Whisper's stock phrases over music and repeated phrases removed; each sentence is then force-aligned inside its Whisper time window with the MMS aligner (non-Latin scripts romanized with uroman) |
In the viewer each lane shows a tag's probability (line), its threshold (red dashes) and the detected events (shaded), so you can see why a tag fired or did not.
20 speaker sounds, written inline where they start:
breath, sigh, sniff, laugh, cough, throat_clear, cry, gasp, yawn, snore, sneeze, nose_blow, lip_smack, groan, scream, whisper, shout, chew, burp, hiccup
34 background sounds, wrapped around the words they overlap (a lone <tag> covers no words):
nature: bird, insect, dog_bark, animal, wind, rain, thunder, water
people: child_voice, baby_cry, applause, cheer, crowd, footsteps
traffic & machines: horn, vehicle, siren, machine, fan, construction, gunshot, explosion
media & devices: music, tv_speaker, phone_ring, phone_vibrate, beep, bell, whistle, buzzer
household: knock, door, dishes, glass_break
Training data. Synthetic mixtures (event clips placed on speech beds, with noise, babble, reverb, phone-band and gain augmentation) plus real annotated windows from NonverbalSpeech-38K, the Vaani Noise Event Dataset (ARTPARK-IISc), NonverbalTTS, AudioSet-Strong, OpenSLR 99 (Deeply Nonverbal Vocalization), DNC (TU Berlin) and DEMAND. Trained for 10 epochs on one T4; the checkpoint of epoch 6 was kept (best mean macro AP over the validation sets: four real-audio sets and one noisy set).
Limits. Thresholds were tuned on held-out clips from the training sources. On produced media (documentaries, music under narration) the sensitive profile in particular adds false tags, often breath, water or animal under music; the balanced profile is stricter. Some tags have little real training data and are weak (the model card lists per-tag scores). Whisper can still miss or mistranscribe speech, especially short non-English passages and speech over music.
Download the model and its inference code, then run it on any file ffmpeg can read:
pip install torch torchaudio numpy scipy safetensors huggingface_hub # plus the ffmpeg binary
pip install openai-whisper # optional, for transcripts
hf download theboringai-work/surround-sound --local-dir surround-sound
cd surround-sound
python inference.py my_audio.mp3 # transcript with tags + event list
python inference.py my_audio.mp3 --asr none # events only
python inference.py my_audio.mp3 --profile balanced --timesFrom Python, inside the downloaded folder:
from sed import Detector, rich_transcript, clean
det = Detector.from_pretrained("theboringai-work/surround-sound", profile="sensitive")
events = det.detect("my_audio.mp3")
# [{'label': 'cough', 'start': 0.68, 'end': 2.0, 'prob': 0.92}, ...]
# with any ASR that gives word timestamps: words = [(word, start_s, end_s), ...]
print(rich_transcript(words, clean(words, events)))