Tokenizer
The code that splits text into a model's tokens. Using the embedding model's own tokenizer lets you check that chunks fit under its 256-token limit.
The code that splits text into a model's tokens. Using the embedding model's own tokenizer lets you check that chunks fit under its 256-token limit.