Glossary: Embeddings and Vector Search
token
Also written: tokens, tokenizer, tokenizing, tokenize
A token is a piece of a word, and a model's tokenizer splits text into them. In keyword search, tokenizing means splitting text into words, which must keep codes like KC-MUG-GRN whole.