How to Improve LLM Apps with Error Analysis

🤝 Want your team maximizing Claude? I run 1:1 and team AI workshops for companies doing $1M+ per year: https://aibuilder.academy/yt/982V2ituTdc Building software on top of LLMs can feel more like praying to the AI gods than engineering. In this video, I discuss how error analysis can resolve the guesswork by systematically identifying and improving the most significant failures of an LLM system. 📰 Read more: https://shawhin.medium.com/how-to-improve-ai-apps-with-error-analysis-4af5f163a1d1?source=friends_link&sk=e6c1ecbff95224d6ed2ee6350e28c1ec 💻 Example Code: https://github.com/ShawhinT/YouTube-Blog/tree/main/LLMs/error-analysis References [1] https://youtu.be/JoAxZsdw_3w?si=5idc0Qq18IF1TRNi [2] https://youtu.be/qH1dZ8JLLdU?si=Qx8S1o16kXfN1EzX [3] https://hamel.dev/blog/posts/field-guide/index.html [4] https://maven.com/parlance-labs/evals Introduction - 0:00 LLMs are Probabilistic - 0:20 Error Analysis - 1:50 LLM Error Analysis - 5:26 Example: Error Analysis of LinkedIn Ghostwriter - 6:40 What's Next? - 15:57