Batching

Also written: batch

Processing several requests in the same model step. A decode step then reads the weights once and produces a token for every request in the batch.