Glossary: Transformers Basics

Cosine schedule

Also written: CosineAnnealingLR, learning rate schedule

A learning-rate schedule that starts high and decreases smoothly toward 0 along a cosine curve over the training steps. It allows fast early learning and fine adjustments near the end, and is often paired with AdamW.