torch.nn.DistributedDataParallel vs DataParallel

[hw-ju]

Hi!
I see almost all MONAI multigpu tutorial scripts use torch.nn.DistributedDataParallel. Does torch.nn.DistributedDataParallel always give better performance than torch.nn.DataParallel when using MONAI for multigpu training?

[Nic-Ma]

Hi @hw-ju ,

It's recommended by PyTorch to use torch.nn.DistributedDataParallel to get better performance, I think it's common practice in PyTorch projects.

Thanks.

[hw-ju]

Thanks @Nic-Ma! That's what I read from PyTorch doc, just need to confirm with you for MONAI. :)