Glossary: Building with LLMs Basics
Tokenizer
The rules a model family uses to split text into tokens. The same email was 189 tokens for Claude and 123 for GPT, so compare prices using real counts.
Glossary: Building with LLMs Basics
The rules a model family uses to split text into tokens. The same email was 189 tokens for Claude and 123 for GPT, so compare prices using real counts.