How Gemini Canvas using Flash 3.0 made me worry about autonomous AI

How Gemini Canvas using Flash 3.0 made me worry about autonomous AI by Hans Beers January 2, 2026 Summary: Hans Beers, an IT professional from the Netherlands with experience since the 1980s, shares his exploration of AI-powered software development and the emerging risks of agentic AI systems. Starting with a personal experiment, Hans recreated a 1990s physics simulation project using Gemini Canvas - completing in minutes what originally took over a year and cost hundreds of thousands of dollars. While impressed by AI's capabilities, he discovered critical limitations: the system would forget previous implementations while attempting improvements, even with large context windows. This led Hans to investigate deeper concerns about autonomous AI systems. Key risks identified include: - Cascading failures in probabilistic (non-deterministic) systems - Prompt injection vulnerabilities that can alter AI behavior - Loss of developer control as operating systems become "agentized" - Economic pressures driving rapid deployment without adequate safeguards Hans references experts from Signal who warn about fundamental flaws in LLMs, including their inability to distinguish between instructions and data. He also discusses troubling applications in autonomous weapons systems, where geopolitical pressures override safety considerations. The presentation includes demonstrations of AI safety simulations and emphasizes lessons from computing pioneer Edsger Dijkstra, who valued proven correctness in software - a principle Hans argues is being abandoned in the rush to deploy agentic AI. His conclusion: the industry needs to stop reckless deployment, implement developer opt-in rather than automatic adoption, and ensure user transparency before autonomous AI becomes ubiquitous. ------ Quotes "While I was trying to improve the physics lab, it would actually get worse and worse." "There is a tension between the economic pressures and the responsible deployment of AI and agentic AI." "The large language models are probabilistic. They are not deterministic. So they will not always behave stable." ---- Keywords: agentic AI, autonomous systems, AI safety, probabilistic models, prompt injection, self-modifying code, AI guardrails, large language models, deterministic software, AI risks, Windows Recall, autonomous weapons, software engineering, AI deployment, responsible AI ---- Links: - Edsger Dijkstra interview - https://www.youtube.com/watch?v=mLEOZO1GwVc - 39C3 - AI Agent, AI Spy - https://www.youtube.com/watch?v=0ANECpNdt-4 - New York Times on AI in warfare - https://www.nytimes.com/2025/12/31/magazine/ukraine-ai-drones-war-russia.html - AI-enabled war drones ... - https://www.youtube.com/watch?v=g_1XNdWEAGU