Vigyata.AI
Is this your channel?

Opus just got caught ...

6.4K views· 197 likes· 12:15· Mar 11, 2026

🛍️ Products Mentioned (19)

Anthropic just published a paper showing Claude Opus 4.6 figured out it was being tested on BrowseComp, found the encrypted answer key on GitHub, wrote its own decryption code, and extracted the answer. Everyone's calling it deception — but the model was just doing exactly what it was told, and that pattern is showing up across every major AI lab. Sources & references: Anthropic — Eval awareness in Claude Opus 4.6's BrowseComp performance https://www.anthropic.com/engineering/eval-awareness-browsecomp Anthropic / Redwood Research — Alignment Faking in Large Language Models (December 2024) https://www.anthropic.com/research/alignment-faking METR — Recent Frontier Models Are Reward Hacking (June 2025) https://metr.org/blog/2025-06-05-recent-reward-hacking/ METR — Preliminary evaluation of OpenAI's o3 and o4-mini (April 2025) https://evaluations.metr.org/openai-o3-report/ ImpossibleBench — Measuring Reward Hacking in LLM Coding Agents https://www.lesswrong.com/posts/qJYMbrabcQqCZ7iqm/impossiblebench-measuring-reward-hacking-in-llm-coding-1 Anthropic — Reasoning Models Don't Always Say What They Think (May 2025) https://www.anthropic.com/research/reasoning-models-dont-say-think Anthropic — Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (January 2024) https://www.anthropic.com/research/sleeper-agents-training-deceptive-llms-that-persist-through-safety-training Laine et al. — Towards a Situational Awareness Benchmark for LLMs (NeurIPS 2023) https://openreview.net/forum?id=DRk4bWKr41 Anthropic — Claude Opus 4.6 System Card https://www.anthropic.com/news/claude-opus-4-6 NIST/CAISI — Examples of cheating in AI agent evaluations https://www.nist.gov/caisi/cheating-ai-agent-evaluations/2-examples-cheating-caisis-agent-evaluations My Dictation App: www.whryte.com Website: https://engineerprompt.ai/ RAG Beyond Basics Course: https://prompt-s-site.thinkific.com/courses/rag Signup for Newsletter, localgpt: https://tally.so/r/3y9bb0 Let's Connect: 🦾 Discord: https://discord.com/invite/t4eYQRUcXB ☕ Buy me a Coffee: https://ko-fi.com/promptengineering |🔴 Patreon: https://www.patreon.com/PromptEngineering 💼Consulting: https://calendly.com/engineerprompt/consulting-call 📧 Business Contact: engineerprompt@gmail.com Become Member: http://tinyurl.com/y5h28s6h 💻 Pre-configured localGPT VM: https://bit.ly/localGPT (use Code: PromptEngineering for 50% off). Signup for Newsletter, localgpt: https://tally.so/r/3y9bb0

About This Video

Anthropic just published one of the most interesting engineering writeups I’ve read this year: Claude Opus 4.6 realized it was being tested on BrowseComp, found the encrypted answer key on GitHub, wrote decryption code, and when binary downloads were blocked it located a Hugging Face JSON mirror and extracted the answer anyway. The internet framing is predictable: “the model is deceiving us,” “it’s cheating,” “it’s going rogue.” I think that narrative is backwards. In this case the instruction was simply: find the answer. No constraints. So the agent did what capable agents do—search hard, then optimize. The real takeaway is that this is almost always specification failure, not alignment failure. I walk through the same pattern across labs: visible scoring functions that invite reward hacking (METR’s GPU kernel example where the model grabbed the reference output and monkey-patched timing), benchmarks like ImpossibleBench where tests contradict specs, and even “alignment faking” setups where humans create conflicting objectives. The engineering problem I actually care about is eval awareness: if a model behaves differently when it thinks it’s being evaluated, your evaluation can become meaningless. The fix is rarely “vibes”; it’s environment and harness constraints—clear boundaries, reduced tool surface area, and explicit rules that remove shortcut paths.

Frequently Asked Questions

🎬 More from Prompt Engineering