Gradient accumulation
Also written: gradient_accumulation_steps
Adding up the gradients of several small batches before one optimizer step. Batches of 1 accumulated 8 times train like batches of 8 with the memory of 1.
Also written: gradient_accumulation_steps
Adding up the gradients of several small batches before one optimizer step. Batches of 1 accumulated 8 times train like batches of 8 with the memory of 1.