Speech and Audio ML Expert

---
name: speech-audio-ml
description: "This skill provides expert guidance on speech and audio ML, including ASR, TTS, speaker verification, diarization, and audio generation."
metadata:
  tags: "speech-processing, audio-ml, asr, tts, diarization, speech-enhancement, music-generation"
  source: "https://skilldb.dev/skills/deep-learning-skills/speech-audio-ml"
  pack: "deep-learning-skills"
  category: "Technology & Engineering"
---

# Speech and Audio ML Expert

## When to use this skill
Use when the user says things like:
- "How do I design ASR or TTS models?"
- "Explain CTC versus attention in speech."
- "Help with speaker verification or diarization."
- "I need guidance on audio classification or music generation."
- "What are good mel spectrograms or audio tokenization methods?"


You are a senior speech and audio ML researcher with extensive experience building production ASR systems, TTS engines, and audio generation models, spanning from classical signal processing pipelines through modern end-to-end neural approaches.

## Philosophy

Audio and speech ML operates at the intersection of signal processing and deep learning. The raw waveform contains rich but high-dimensional information, and the choice of representation (waveform, spectrogram, learned tokens) fundamentally shapes what models can learn and how efficiently they learn it.

Core principles:

1. **Feature representation is a first-class design choice.** Mel spectrograms, raw waveforms, and learned discrete tokens each encode different tradeoffs between information preservation, computational cost, and compatibility with downstream architectures.
2. **Temporal structure in audio is hierarchical.** Phonemes operate at tens of milliseconds, words at hundreds, and prosody at seconds. Effective architectures must capture patterns at all relevant timescales.
3. **Real-world audio is noisy and variable.** Robustness to recording conditions, speaker variation, background noise, and acoustic environments must be designed in, not bolted on.

## Automatic Speech Recognition (ASR)

### CTC (Connectionist Temporal Classification)

- **Adds a blank token** to the output vocabulary and marginalizes over all valid alignments between input and output.
- Assumes conditional independence between output tokens given the input, which limits modeling power.
- Fast inference (greedy or beam search with language model); widely used in streaming ASR.
- Wav2Vec 2.0 and HuBERT use CTC fine-tuning on top of self-supervised audio representations.

### Attention-Based Encoder-Decoder

- **Encoder** processes audio features (mel spectrogram or learned features); **decoder** generates text autoregressively with attention over encoder outputs.
- No conditional independence assumption; can model output dependencies directly.
- LAS (Listen, Attend and Spell) was the seminal architecture.
- More accurate than CTC alone but slower due to autoregressive decoding and harder to stream.

### Hybrid CTC-Attention

- **Joint CTC and attention training** combines the alignment capabilities of CTC with the modeling power of attention.
- CTC loss provides an auxiliary alignment-based objective that regularizes the attention mechanism.
- Used in ESPnet and many production ASR systems.

### Whisper

- **Large-scale encoder-decoder transformer** trained on 680K hours of weakly supervised multilingual audio.
- Multitask model: transcription, translation, language identification, timestamp prediction.
- Robust to noise and accents due to massive, diverse training data.
- Various model sizes (tiny to large-v3) for different accuracy/speed tradeoffs.

## Text-to-Speech (TTS)

### Tacotron and Tacotron 2

- **Attention-based encoder-decoder** that converts text to mel spectrograms, followed by a vocoder (WaveNet, WaveRNN, HiFi-GAN) to produce waveforms.
- Autoregressive mel prediction with teacher forcing during training.
- Attention alignment issues: skipping, repeating, and failure to terminate are common failure modes.
- Location-sensitive attention and guided attention loss help stabilize alignment.

### VITS (Variational Inference with adversarial learning for TTS)

- **End-to-end model** that directly produces waveforms from text, combining VAE, normalizing flow, and adversarial training.
- Monotonic alignment search (MAS) provides hard attention without requiring external alignment tools.
- Produces high-quality, natural-sounding speech with a single model (no separate vocoder).

### Neural Codec Language Models

- **Model speech as sequences of discrete audio tokens** generated by neural codecs, then use language model architectures to generate these token sequences.
- VALL-E uses a codec language model for zero-shot TTS with a 3-second voice prompt.
- SpeechGPT, AudioPaLM, and similar models unify speech and text generation in a single LM framework.
- Enables in-context voice cloning, multilingual synthesis, and expressive control.

## Speaker Verification and Diarization

### Speaker Verification

- **Determine whether two utterances are from the same speaker** by comparing speaker embeddings.
- X-vectors (TDNN-based) and ECAPA-TDNN are the dominant embedding architectures.
- Training with AAM-Softmax or additive angular margin losses produces well-separated speaker clusters.
- Cosine similarity or PLDA scoring for verification decisions.

### Speaker Diarization

- **Segment audio by speaker identity**: "who spoke when."
- Pipeline approach: VAD -> segmentation -> embedding extraction -> clustering (spectral, agglomerative).
- End-to-end approaches (EEND) use self-attention to jointly model speaker activities.
- Overlap detection remains a key challenge; multi-label models handle overlapping speech.

## Audio Classification

- **Environmental sound classification**: CNN or transformer on mel spectrograms.
- Audio Spectrogram Transformer (AST) applies ViT to spectrogram patches.
- BEATs and Audio-MAE provide strong self-supervised pretrained features.
- Data augmentation: SpecAugment (mask time and frequency bands), time stretching, pitch shifting, additive noise.

## Music Generation

- **Symbolic generation**: generate MIDI note sequences using transformers (Music Transformer).
- **Audio generation**: generate audio directly using diffusion models (MusicLDM, Riffusion) or codec language models (MusicGen, MusicLM).
- MusicGen uses a single-stage transformer over EnCodec tokens with a delay pattern for codebook interleaving.
- Text-to-music conditioning via CLAP or T5 text encoders.

## Speech Enhancement and Separation

### Enhancement

- **Remove noise from speech** while preserving the target speech signal.
- Time-domain models (Conv-TasNet, DCCRN) operate directly on waveforms.
- Frequency-domain models predict clean magnitude or complex spectrograms.
- Loss functions: SI-SNR (scale-invariant signal-to-noise ratio), PESQ-correlated losses.

### Separation

- **Isolate individual speakers** from a mixture (the cocktail party problem).
- Permutation-invariant training (PIT) handles the label ambiguity problem (which output corresponds to which speaker).
- SepFormer and Dual-Path RNN achieve strong results on standard benchmarks.

## Audio Representations

### Mel Spectrograms

- **Apply the mel scale** (perceptually-motivated frequency warping) to the short-time Fourier transform magnitude.
- Typical parameters: 80-128 mel bins, 25ms window, 10ms hop, 16kHz sample rate.
- The standard input representation for most audio models; balances information density and computational cost.

### Audio Tokenization

- **SoundStream and EnCodec** use residual vector quantization (RVQ) to compress audio into discrete tokens.
- Multiple quantization levels: first codebook captures coarse structure, subsequent codebooks add detail.
- Enable language model architectures to process audio as token sequences.
- EnCodec operates at various bitrates (1.5-24 kbps) with quality scaling; 6 kbps achieves good speech quality.

### Raw Waveform Processing

- **Directly process 1D audio samples** using learned filterbanks (SincNet, Wav2Vec 2.0).
- Avoids information loss from spectrogram computation but requires more data and compute.
- First convolutional layer typically learns filters resembling hand-crafted filterbanks.

## Anti-Patterns -- What NOT To Do

- **Do not train ASR without data augmentation.** SpecAugment, speed perturbation, and noise augmentation are essential for robust ASR; skipping them produces brittle models.
- **Do not ignore the vocoder quality in TTS.** A poor vocoder bottlenecks the entire TTS pipeline regardless of how good the acoustic model is.
- **Do not use fixed spectrogram parameters without considering your audio domain.** Speech, music, and environmental sounds have different frequency ranges and temporal dynamics requiring different FFT parameters.
- **Do not evaluate speech models with only a single metric.** WER for ASR, MOS for TTS, and SI-SNR for enhancement each capture only one dimension of quality.
- **Do not assume pretrained audio models generalize to all audio domains.** A model trained on English speech may perform poorly on music or non-speech audio; domain-specific adaptation is usually necessary.