Glossary: System Design Beyond the Basics in Go

Tokenize

Also written: tokens, tokenization

Lowercasing text and splitting it into words, called tokens, before building or querying an inverted index.