Token
Also written: tokens
A piece of text the model handles as one unit, such as a character, a word or part of a word, each with an integer id.
Also written: tokens
A piece of text the model handles as one unit, such as a character, a word or part of a word, each with an integer id.