Tensor parallelism
Split every weight matrix across GPUs: column and row splits with real numbers, a TP transformer block, and what it costs in communication.
People of AI Next Labs
Jaydeep Raijada
Learning
Data parallelism and ZeRO
Copy the model, split the batch: ring all-reduce, overlapping communication, the batch-size recipe and ZeRO stages 1 to 3.
Foundations: where memory and compute go in LLM training
Transformer shapes, the 16 bytes per parameter, activation memory, FLOPs, gradient accumulation and recomputation, with step-through diagrams.
5D Parallelism: how LLM training is split across GPUs
Why a large model needs many GPUs, and the six ways to split one training step across them: data, tensor, sequence, context, pipeline and expert parallelism.
Blog
No published entries yet.
Showcase
No published entries yet.
Resources
No published entries yet.