Pre-splitting
Also written: pre-tokenization
Splitting text at spaces and punctuation before BPE merging, so no token crosses a word boundary. Our simple tokenizer lacks it, which is why 'one ships' got split oddly.
Also written: pre-tokenization
Splitting text at spaces and punctuation before BPE merging, so no token crosses a word boundary. Our simple tokenizer lacks it, which is why 'one ships' got split oddly.