Concept
FSDP2
FSDP2 is the second-generation Fully Sharded Data Parallel API in PyTorch, a technique for training models too large to fit on one GPU by sharding parameters, gradients, and optimizer states across ranks. Like standard data parallelism (DDP), except instead of…
The rest of “FSDP2” is a premium feature: every concept in the library gets a precise, practitioner-focused write-up like this one, cross-linked straight from the paper summaries that use it.
Log in to unlock→