Mixed precision

Also written: torch.autocast, autocast

Training that does most of the math in 16-bit numbers, which can be about twice as fast on modern GPUs and halves memory use. It mainly pays off for big models, and with float16 it uses a gradient scaler to keep tiny gradients from rounding to zero.