Scaled dot-product attention
Also written: scaling the scores
Attention where scores are the query–key dot products divided by √d_k, so large vectors don't make softmax put nearly all weight on one token. The formula is softmax(QKᵀ/√d_k)V.
Also written: scaling the scores
Attention where scores are the query–key dot products divided by √d_k, so large vectors don't make softmax put nearly all weight on one token. The formula is softmax(QKᵀ/√d_k)V.