InstructGPT: alignment with RLHF
Used reinforcement learning from human feedback (RLHF) to teach the model to follow instructions and give the answers people want.
1.3B > 175Bafter RLHF, a small model was preferred over the original GPT-3
Why it matters
The key recipe behind ChatGPT: the model didn't need to get bigger, just better behaved.
Costs and controversy
Human labeling is costly, and models may learn to please rather than to be right.
Built on
Led to
Source: arxiv.org ↗