When you fine-tune an AI model, you’re often destroying its prior knowledge. That’s catastrophic forgetting: the model gets better on the new domain you trained it on, but it suddenly performs worse on tasks it used to be good at. In this video, I break down an MIT approach that tackles this head-on: Self-Distillation Fine-Tuning (SDFT). The idea is shockingly simple—use the same model as both teacher and student. The “teacher” gets the prompt with extra context (a cheat code like a few-shot example or text+answer), while the “student” gets the same prompt with no help. Then you train the student to match the teacher’s output distribution using a reverse KL-style objective, which keeps it on-policy. The key takeaway is that SDFT sits between supervised fine-tuning (off-policy, label-forcing) and reinforcement learning (on-policy, reward-based). You get RL-like stability without needing to invent a reward function, and you avoid the common SFT failure mode where you kill the model’s reasoning and response quality. The paper’s results show you can improve new-task accuracy while preserving prior-task performance across sequential training (tool use → science → medical), and the gap vs SFT grows as model size increases. The main caveat: training is slower and can cost ~2.5x to 5x more compute, but the “no forgetting + better learning” tradeoff can be worth it for enterprise-grade agent and MLOps workflows.