Skip to content

Glossary: Embeddings and Vector Search

token

Also written: tokens, tokenizer, tokenizing, tokenize

A token is a piece of a word, and a model's tokenizer splits text into them. In keyword search, tokenizing means splitting text into words, which must keep codes like KC-MUG-GRN whole.

Where the book teaches it

Related terms

truncationBM25