Skip to content

Glossary: Transformers Basics

Token

Also written: tokens

A piece of text the model handles as one unit, such as a character, a word or part of a word, each with an integer id.

Where the book teaches it

Related terms

TokenizerVocabulary