Fine-tuning LLMs on Human Feedback (RLHF + DPO)

🤝 Want your team maximizing Claude? I run 1:1 and team AI workshops for companies doing $1M+ per year: https://aibuilder.academy/yt/bbVoDXoPrPM Here, I discuss how to use reinforcement learning to fine-tune LLMs on human feedback (i.e. RLHF) and a more efficient reformulation of it (i.e. DPO) 📰 Read more: https://medium.com/@shawhin/fine-tuning-llms-on-human-feedback-rlhf-dpo-1c693dbc4cbf?sk=b8b24748abb1a0792f4881982589e848 Example code: https://github.com/ShawhinT/YouTube-Blog/tree/main/LLMs/dpo 🤗 Dataset: https://huggingface.co/datasets/shawhin/youtube-titles-dpo 🤗 Fine-tuned Model: https://huggingface.co/shawhin/Qwen2.5-0.5B-DPO References [1] arXiv:2407.21783 [cs.AI] [2] arXiv:2203.02155 [cs.CL] [3] arXiv:1707.06347 [cs.LG] [4] https://youtu.be/7xTGNNLPyMI [5] arXiv:2305.18290 [cs.LG] Intro - 0:00 Base Models - 0:25 InstructGPT - 2:20 RL from Human Feedback (RLHF) - 5:18 Proximal Policy Optimization (PPO) - 9:20 Limitations of RLHF - 10:30 Direct Policy Optimization (DPO) - 11:50 Example: Fine-tuning Qwen on Title Preferences - 14:29 Step 1: Curate preference data - 17:49 Step 2: Fine-tuning with DPO - 20:53 Step 3: Evaluate fine-tuning model - 25:27