Memory
This guide collects the main knobs that affect GPU memory use.
First-line levers
- Reduce
sequence_len - Reduce
per_device_train_batch_size - Increase
gradient_accumulation_steps(amortizes optimizer step)
Framework knobs
- Recomputation (
recompute_*) - Chunking (
lmhead_chunks,attn_bwd_chunks) - Offloading (
offload_*) - Quantization / QLoRA (
qlora_*)