Compute-optimal
Training a model with the best balance of size and data for a fixed compute budget, about 20 tokens per parameter by Chinchilla's rule. Labs often train smaller models past this point because they are cheaper to serve.
Training a model with the best balance of size and data for a fixed compute budget, about 20 tokens per parameter by Chinchilla's rule. Labs often train smaller models past this point because they are cheaper to serve.