AdamW
A widely used optimizer for training neural networks. It keeps two running averages (optimizer states) for every trained parameter, which is a big part of full fine-tuning's memory cost.
A widely used optimizer for training neural networks. It keeps two running averages (optimizer states) for every trained parameter, which is a big part of full fine-tuning's memory cost.