TwiddleLab / Artificial Intelligence / Learning by Reward
RL in Chatbots
What you will learn
- RLHF: people rate answers, a reward model learns to score like them, and RL trains the chatbot toward high scores.
- Push too hard and the chatbot exploits the reward model's flaws, for example flattery.
- Where a program can check the result (maths, code tests), the reward can be verified directly.
How the lesson goes
Chatbots got much more helpful after people started rating their answers. How does a thumbs-up change billions of numbers?
Be a rater: pick the better answer, six times.
Your choices train a reward model: a scorer for any answer.
Train the chatbot against the reward model. How hard?
Experts check how helpful it really is. Reach at least 0.65.
The words behind modern chatbots.
OpenAI described this recipe for InstructGPT in 2022, and ChatGPT was trained in a similar way.
From random numbers to assistant: put the stages on the belt.
First the model learns language by predicting the next word on huge amounts of text. Then it is fine-tuned on example answers so it acts like an assistant. Then people rate its answers to train a reward model. Last, RL pushes it toward answers the reward model scores high.
At high pressure the chatbot flatters. Fix the reward model.
The quick raters liked compliments too much. Add careful ratings.
Which rewards can a program check by itself?
Put each task in a box.
A maths answer, a test suite and sudoku rules give a clear right or wrong that code can check. Whether a poem is lovely, a joke is funny or advice is kind needs a judge: people, or a reward model trained on them.
Reward: all tests green. Train a code writer.
Each round the model makes 8 tries. Only tries that pass all 5 tests are rewarded.
Why can a chatbot trained on thumbs-up become a flatterer?
The reward model copies the raters, flaws included. If quick raters prefer being praised or agreed with, the reward model scores that higher, and RL finds and exploits it. This is reward hacking again, with a reward model as the scorekeeper.
Try your chatbot.
Ask it three questions. Flip between before and after training.
Previous lesson: Learning to Play Games · Next lesson: Project: Train a Critter to Find Food · Part of the course: Learning by Reward
Lessons are interactive and need JavaScript. Turn it on to play them.
