Tokenizer

Also written: tokenization

The tool that splits text into tokens and turns them into ids and back. A model must always be used with the tokenizer it was trained with.