Weight decay

Also written: weight_decay

Shrinks every weight a little toward zero at each step, so only weights the data keeps pushing on stay large. It is the network version of the penalty in ML Basics' logistic regression.