Tensor parallelism
Split every weight matrix across GPUs: column and row splits with real numbers, a TP transformer block, and what it costs in communication.
AI Next Labs / Learning
Learning
Interactive guides to the concepts behind our work.
Interactive lesson
How RAG finds an answer
Turn a document into searchable chunks. Find the right facts. Give them to the AI to help it answer.
Data parallelism and ZeRO
Copy the model, split the batch: ring all-reduce, overlapping communication, the batch-size recipe and ZeRO stages 1 to 3.
Foundations: where memory and compute go in LLM training
Transformer shapes, the 16 bytes per parameter, activation memory, FLOPs, gradient accumulation and recomputation, with step-through diagrams.
5D Parallelism: how LLM training is split across GPUs
Why a large model needs many GPUs, and the six ways to split one training step across them: data, tensor, sequence, context, pipeline and expert parallelism.
How RAG finds an answer
Turn a document into searchable chunks. Find the right facts. Give them to the AI to help it answer.