Optimizer states
Also written: optimizer state
AdamW's two running averages for each trained parameter, taking 8 bytes per parameter in float32. LoRA avoids most of this cost by freezing the base weights.
Also written: optimizer state
AdamW's two running averages for each trained parameter, taking 8 bytes per parameter in float32. LoRA avoids most of this cost by freezing the base weights.