Tokenizer

Also written: tokenizers, tokenization

The component that turns text into token IDs a model understands. Always use the one that came with the model. The `tokenizers` library also lets you train your own.