LLM in a Loop: Automate feedback with evals

🤝 Want your team maximizing Claude? I run 1:1 and team AI workshops for companies doing $1M+ per year: https://aibuilder.academy/yt/6VcXukJuGA0 This is the 4th video in a series on AI agents. Here, I discuss how to improve LLM systems with automated feedback loops and share example code for an Upwork profile rewriter using OpenAI's Responses API. 📰 Blog: https://shawhin.medium.com/llm-in-a-loop-improving-outputs-with-evals-5620e00f7258?sk=95956863ff584b8d1fd3664b0ec8a6bc 💻 Code: https://github.com/ShawhinT/YouTube-Blog/tree/main/agents ▶️ Series Playlist: https://www.youtube.com/playlist?list=PLz-ep5RbHosU02OKABBkbsQrYWmQfoZMH References [1] arXiv:2402.06627 [cs.LG] [2] https://youtu.be/-sL7QzDFW-4 Intro - 0:00 Motivation - 0:16 LLM in a Loop - 1:57 3 Types of Evals - 4:42 Type 1: Rule-based - 5:32 Type 2: LLM-based - 7:32 Type 3: Real-world - 9:00 Example: Upwork Profile Rewriter - 10:38 Limitations - 22:59