pytorch mutliple gpus

https://glassboxmedicine.com/2020/03/04/multi-gpu-training-in-pytorch-data-and-model-parallelism/

Multi-GPU Training in Pytorch: Data and Model Parallelism

Training on One GPU

Let’s say you have 3 GPUs available and you want to train a model on one of them. You can tell Pytorch which GPU to use by specifying the device:

• device = torch.device(‘cuda:0’) for GPU 0
• device = torch.device(‘cuda:1’) for GPU 1
• device = torch.device(‘cuda:2’) for GPU 2

Training on Multiple GPUs

To allow Pytorch to “see” all available GPUs, use:

device = torch.device(‘cuda’)