LoRA: Low-Rank Adaptation
Freeze the big model and train only small low-rank matrices, cutting fine-tuning cost dramatically.
10,000×fewer trainable parameters for GPT-3 175B
Why it matters
Made it possible for ordinary people to fine-tune big models; most open-source fine-tunes since rely on it.
Led to
Source: arxiv.org ↗