Glossary: NLP with Transformers
Subword
Also written: subword tokens, word pieces, ##
A piece of a word that a tokenizer uses as a token. Common words stay whole, and rarer ones split into pieces, like `es ##press ##o`.
Glossary: NLP with Transformers
Also written: subword tokens, word pieces, ##
A piece of a word that a tokenizer uses as a token. Common words stay whole, and rarer ones split into pieces, like `es ##press ##o`.