Activations
The intermediate results a model saves during the forward pass so the backward pass can use them. Their memory grows with batch size and sequence length, on top of the weights and optimizer memory.
The intermediate results a model saves during the forward pass so the backward pass can use them. Their memory grows with batch size and sequence length, on top of the weights and optimizer memory.