Glossary

106 terms from Speech and Audio AI, in plain words, each with the chapter that teaches it.

Search the glossary

A

Aliasing: False tones that appear when you lower the sample rate without filtering out the high frequencies first. A proper resampler like soxr filters them out before dropping samples.
argparse: Python's standard tool for reading command-line arguments. The project's `pipeline.py` uses it to accept a folder plus the `--api` and `--no-audio` switches.
ASR (automatic speech recognition): Software that turns spoken audio into written text, such as Whisper transcribing a Kettle & Crane voicemail.
async generator (async for): A generator you loop over with `async for` inside an `async def` function, run with `asyncio.run`. Kokoro's `create_stream` is one.
Audio file formats (WAV, FLAC, MP3, AAC, Opus): WAV is uncompressed PCM (a small header plus samples; size = sample rate × bytes per sample × channels × seconds). FLAC is lossless at half to two-thirds the size; MP3 and AAC (`.m4a`) are lossy and common for music and phone apps; Opus (`.opus`, `.webm`) is lossy with the best quality per byte for speech. soundfile can't read `.m4a` or `.webm`, so convert them with ffmpeg first.
Audio tokens: The units in which speech models read and write sound, billed like text tokens. The Realtime API reports roughly 10 audio tokens per second in and 20 out.

B

Barge-in: When a caller starts talking while a voice agent is speaking. The agent should stop playback at once and tell the model how much of its reply was actually heard.
base64: A way of writing bytes as plain text. The Realtime API sends audio inside JSON messages this way.
Beam search: A decoding method that keeps the few likeliest partial transcripts at each step and picks the best at the end. faster-whisper's default `beam_size=5` is slower than 1 but usually a little more accurate.
Bit depth: How precisely each audio sample is stored. 16-bit audio stores each sample as a whole number from −32,768 to 32,767.
Bit rate: How many bits per second a compressed audio file uses. In ffmpeg, `-b:a 16k` sets it, and lower values give smaller, lossier files.

C

Cascade: A design that chains separate models, such as transcribing then translating with an LLM, or speech-to-text, LLM and text-to-speech for a voice agent. It lets you inspect and log every step's text.
CCPA (California Consumer Privacy Act): A California privacy law that gives customers the right to know about, and delete, the data a business holds on them.
CER (character error rate): Like WER but counted in characters instead of words. It is used for languages written without spaces, such as Japanese and Chinese.
Channels: The separate signals in a recording: mono has one, stereo has two (left and right). A stereo array has shape `(frames, 2)`.
Clipping: Samples above full scale get cut off at the limit, which sounds like crackling distortion. Keep peaks below 1.0 to avoid it.
Clustering: The last diarization step, which groups pieces of speech with similar speaker embeddings into one speaker. You give a speaker count or let a distance threshold decide.
Codec: The method used to compress audio. Lossless codecs such as FLAC keep every sample exactly; lossy ones such as MP3 and Opus discard detail.
Consent to record: Rules on whether a call may be recorded. Some US states need only one party's consent, others (such as Washington and California) need everyone's, so announce recording at the start of every recorded call.
CTranslate2: The inference engine faster-whisper uses to run Whisper models, up to four times quicker on a CPU than OpenAI's own package.
CUDA: NVIDIA's platform for running computations on its graphics cards. In faster-whisper you set `device="cuda"` and `compute_type="float16"` to use one.

D

Data retention and redaction: Retention is how long you keep audio and transcripts before deleting them; redaction removes personal details before sharing. Keep the minimum, and know where hosted APIs send your audio.
dBFS: Decibels measured against digital full scale (a sample of 1.0), so levels are zero or negative. −6 dBFS is a peak of about 0.5.
Decibel (dB): A logarithmic unit of level. Every 6 dB doubles the amplitude, and a power ratio is 10 × log10 while an amplitude ratio is 20 × log10.
DER (diarization error rate): The fraction of time in a recording that speakers are labeled wrongly. Overlaps and very short turns make it worse.
Diarization: Working out who spoke when in a recording. It labels speakers 0, 1, 2 without names, as in Sam and Kendra's call.

E

Edit distance: The fewest word substitutions, deletions and insertions needed to turn one sequence into another. WER is built on it.
Encoder, decoder: Whisper's two model parts. The encoder reads a whole 30-second window and turns it into vectors, and the decoder writes the text tokens one at a time.
Entity accuracy: Whether the key facts, such as names and order numbers, are transcribed correctly. A low WER can still hide one wrong digit, like Fiona's order number.
espeak-ng: A free library that converts words to phonemes. Many open TTS models, including Kokoro, use it.

F

faster-whisper: A Python package that runs Whisper through the CTranslate2 engine, faster than OpenAI's own package on a CPU. It decodes audio with PyAV and can quantize the model.
FCC and TCPA: In 2024 the US Federal Communications Commission ruled that AI-generated voices count as artificial voices under the Telephone Consumer Protection Act. Automated calls using them therefore need the called person's prior consent.
ffmpeg: A free command-line program that converts almost any audio or video format, for example to 16 kHz mono WAV for speech models. Its companion ffprobe describes a file.

G

Gated model: A model on Hugging Face that you can only download after accepting its conditions and passing an access token, like pyannote's pipeline.
Generator: A sequence that produces items only as you loop over it. faster-whisper's `transcribe()` returns segments as a generator, so nothing is transcribed until you loop.
Grounding: Giving a voice agent real facts and tools so it doesn't make things up, such as the catering price in the instructions.

H

Hallucination: When Whisper writes text that isn't in the audio, typically on silence, music or noise, such as "Thank you for watching". VAD is the reliable defense.
Hertz (Hz): The unit of sample rate and frequency, in cycles or samples per second. 16 kHz means 16,000 samples each second.
Hugging Face: A website that hosts open models. faster-whisper downloads Whisper models from it into `~/.cache/huggingface`.

I

Insertion, deletion, substitution: The three kinds of word error counted in WER: a wrong word in place of a right one (substitution), a missed word (deletion), and an added word (insertion).
Intent: The category of what a caller wants, such as `wholesale_order` or `spam`, chosen from a fixed list so routing code can act on it.
io.BytesIO: A file-like object from Python's `io` module that holds its bytes in memory instead of on disk. It lets code such as an audio chunk written with `soundfile` be uploaded to an API without saving a temporary file.

J

jiwer: A Python package that computes WER and shows the alignment of substitutions, insertions and deletions.
JSONL (JSON Lines): A file with one JSON object per line, like `voicemails.jsonl` and `transcripts.jsonl`.

K

Kokoro: A small open text-to-speech model with 54 voices that makes 24 kHz audio. The `kokoro-onnx` package runs it locally.

L

Latency budget: The total delay a voice agent can afford before the caller thinks no one is there. Each stage, such as turn end, speech-to-text, LLM and TTS, spends part of it.
Log-Mel spectrogram: A picture of the energy at each pitch over time, on a scale matching human hearing. Whisper converts each audio window into one before encoding it.
Log-probability: The natural log of a probability: 0 means certain, and more negative means less sure. Whisper's `avg_logprob` averages it over a segment's tokens.
Lookaround: A regular-expression check of the character before or after a position without replacing it, such as `(?<=\d)`.
Lossy and lossless: Lossless codecs keep every sample exactly; lossy ones throw away detail that ears notice least, so each lossy step loses a little.
Loudness normalization: Scaling audio by a gain in decibels so its loudness hits a target such as −23 LUFS, without letting peaks pass the clipping limit.
loudnorm: ffmpeg's filter that normalizes loudness while limiting peaks. It reaches a target like −16 LUFS without clipping, unlike plain gain.
LUFS: A broadcast loudness standard weighted to how ears hear. Podcasts aim for about −16 LUFS and TV for −23 or −24.

M

Messages API: Anthropic's API for calling Claude with text, images and documents (not audio). `messages.parse` also validates the reply against a Pydantic class.
MOS (mean opinion score): The average of listeners' 1-to-5 ratings, the standard measure of how natural TTS sounds.

N

Nyquist frequency: Half the sample rate, the highest frequency that rate can capture. 16 kHz audio can represent up to 8 kHz.

O

ONNX Runtime: A fast engine for running models saved in the portable ONNX format. sherpa-onnx and Kokoro use it, so no PyTorch is needed.

P

p95 (95th percentile): The time that 95% of requests beat. Reporting it shows the slow calls that users remember.
PCM (pulse-code modulation): Uncompressed audio stored as plain sample numbers. WAV normally holds PCM, and every other format is PCM or a way of compressing it.
Peak: The largest sample in a recording, which shows how close it is to clipping. A peak of 0.5 is −6 dBFS.
Phoneme: A sound unit of a language. A TTS system converts words to phonemes ("coffee" is /ˈkɔfi/) before making audio.
Prompt and keyword hints: Extra input that steers a transcription model toward your vocabulary: Whisper's `initial_prompt` and `hotwords`, and the hosted API's `prompt`, `keywords` and `languages`. Keep them short and factual; they are not instructions.
pyannote.audio: The best-known open diarization toolkit. Its pipeline is gated and needs PyTorch, while sherpa-onnx uses only its segmentation model.
PyAV: Python bindings for ffmpeg's libraries. faster-whisper uses it to decode MP3 and M4A directly.
Pydantic: A Python library for describing data as classes and validating it. It can define a record such as the `Voicemail` that Claude fills in, with `Literal` limiting a field like `intent` to a fixed menu of values.
pyloudnorm: A Python package that measures LUFS loudness. It needs at least 0.4 seconds of audio.
pytest and assert: pytest is a tool that runs every function named `test_...` and reports failures. `assert` is the Python statement that fails a test if its condition is False.
PyTorch: A deep-learning library for building and running neural networks. OpenAI's `openai-whisper` package and the pyannote.audio toolkit require it, while tools such as faster-whisper and sherpa-onnx run models without it.

Q

Quantization: Storing a model's weights with fewer bits, such as 8-bit integers instead of 32-bit floats. It saves memory and time at almost no cost in accuracy.

R

Real-time factor (RTF): Processing time divided by audio duration. Below 1.0 means faster than real time.
Realtime API: OpenAI's API for live speech-to-speech models. It keeps a WebSocket open and exchanges JSON events carrying audio, transcripts and status.
Reference, hypothesis: The reference is the correct transcript, and the hypothesis is the model's transcript scored against it.
Regular expression: A pattern language for matching and replacing text, used in Python through the `re` module. It is useful for tasks such as stripping punctuation, finding numbers and validating phone numbers.
Resampling: Changing the sample rate. It must filter first to prevent aliasing, so you can't just keep every other sample.
RMS (root mean square): The square root of the average squared sample, a measure of a signal's average level.

S

Sample: One measurement of the sound's air pressure at one instant. A recording is a long list of them.
Sample rate: How many samples are taken per second, in hertz. Phone voicemail is 8,000 Hz, while most speech models expect 16,000 Hz.
Segmentation: The first diarization step, where a small network finds where speech is and where the speaker changes.
Semantic VAD: Deciding from the words, not just silence, whether the caller has finished. "My number is five oh three..." isn't finished, but "Thanks, bye" is. The Realtime API's plain alternative is `server_vad`, based on silence.
sherpa-onnx: A Python package that runs speech models, including speaker diarization, through ONNX Runtime. It needs neither PyTorch nor an access token.
Silero VAD: The small voice activity detection model that faster-whisper includes and uses for `vad_filter`.
SIP (Session Initiation Protocol): The standard protocol for internet phone calls. A call can reach the Realtime API over SIP, and the SDK can accept, transfer or hang it up.
SNR (signal-to-noise ratio): Speech power divided by noise power, in dB. 20 dB is a quiet room, 10 dB a busy café, and at 0 dB the noise is as strong as the voice.
soundfile: A Python package that reads and writes WAV, FLAC, MP3 and Ogg files as NumPy arrays of floats.
soxr: A Python package for high-quality, fast resampling.
Speaker embeddings: A vector from a neural network that is similar for the same voice. Diarization clusters these vectors to decide who is who.
Speech translation: Turning speech in one language into text or speech in another, either directly (Whisper into English) or through a cascade with an LLM.
Speech-to-speech model: One model that hears audio and answers with audio, giving the lowest latency and natural turn-taking, but with less inspectability than a cascade.
SRT: A subtitle file format. Each block has a number, a `00:00:01,500 --> 00:00:03,900` time line, the text and a blank line.
SSE (server-sent events): A way for a server to push a stream of events to your program. OpenAI's speech endpoint can use it so the final event reports token usage.
SSML (Speech Synthesis Markup Language): An XML markup standard for controlling TTS pauses, emphasis and how numbers are read. Google, Azure and Polly support it, but Kokoro and OpenAI don't.
Streaming: Receiving results in pieces while the model is still working. For transcription it arrives as `transcript.text.delta` events and a final `done` event.
Structured output: Having an LLM return data matching a fixed schema instead of free text, such as Claude returning a validated `Voicemail` record.

T

Temperature: How randomly a model picks tokens. Whisper raises it when it is unsure and retries, so output on hard audio can differ between runs.
Text normalization: Rewriting text into a standard form, for scoring (lowercasing, joining spoken digits, removing punctuation) or for speaking (turning "$4.50" into "four fifty").
TTS (text-to-speech): Generating spoken audio from text, in stages: text normalization, phonemes, then an acoustic model that makes the audio.
Turn detection: Deciding when the caller has stopped talking. The usual method is VAD with a silence threshold, where too short interrupts people and too long adds delay.

V

VAD (voice activity detection): A small model that finds the stretches of audio containing speech. It is used to skip noise, split long audio at pauses and detect turn ends.
Voice agent: An assistant you talk to: it listens, decides what to say, perhaps after looking up an order, and answers aloud.
Voice cloning: Making a TTS model speak in a particular person's voice from a short sample. It should be done only with the speaker's written, informed consent.

W

WebSocket: A connection that stays open so both sides can send small messages at any time. The Realtime API uses it to stream audio and events.
Weights: The learned numbers that make up a model. Whisper's are free to download, so audio can stay on your machine.
WER (word error rate): (substitutions + deletions + insertions) divided by the number of reference words. Lower is better, and it can exceed 100%.
Whisper: OpenAI's open speech-recognition model, released in 2022 under the MIT license. It resamples audio to 16 kHz, cuts it into 30-second windows, encodes them and decodes text, and comes in sizes from `tiny` to `large-v3`.
Word timestamps: With `word_timestamps=True`, each word gets a start, end and probability. Low probabilities flag words worth checking, and the times let you match words to speakers.

#

μ-law (mu-law): The phone network's lossy 8-bit format, at 8 kHz and one byte per sample. The Realtime API accepts it directly as `audio/pcmu`.