Concept
Tensor-parallel
Tensor-parallel (a form of model parallelism) splits the math inside a single layer across multiple GPUs, so no one device holds the full weight matrix. This is distinct from data parallelism, where every GPU has a complete copy of the model and processes different…
The rest of “Tensor-parallel” is a premium feature: every concept in the library gets a precise, practitioner-focused write-up like this one, cross-linked straight from the paper summaries that use it.
Log in to unlock→