Parallelism (data, pipeline, tensor)

Also written: Data parallelism, pipeline parallelism, tensor parallelism, --data-parallel-size, --tensor-parallel-size, --pipeline-parallel-size

Ways to use several GPUs. Data parallelism runs whole copies of the model; pipeline parallelism gives each GPU a block of layers; tensor parallelism splits every layer across GPUs in one machine and needs a fast link like NVLink.