Glossary: NLP with Transformers

Normalizer, pre-tokenizer, post-processor

Also written: normalizer, pre-tokenizer, post-processor

Three tokenizer parts. The normalizer cleans text, the pre-tokenizer splits it into words, and the post-processor adds special tokens such as `[CLS]` and `[SEP]`.