How Smart Are LLMs? Mario Knows!

Fun weekend project testing different LLMs within a Gaming Agent. Seems like I learn the most doing things that really have no applicability in the world. So why is this an interesting project and test for LLMs and Agents? - Problem Solving and Reasoning: Tests the capabilities of LLMs ability to reason and problem solve within dynamic environments. You can see from my below screenshot... my agent is a bit of an idiot. Speed is an issue from input to execution. Could likely speed it up, but I'm already running up my API tab with Anthropic playing around - Multimodal Skills: What's this thing doing under the hood? It's taking screenshots, embedding for context and then reasoning through the next action (vision-to-language-to-action). It's a very slow process, so if I want any sort of positive value, then I should reduce the speed at which my NES emulator is running. and a VERY big thank you to Codeium's cascade feature in Windsurf with helping me get this working. Reddit Post Discussion: https://www.reddit.com/r/singularity/comments/1j1pxru/playing_super_mario_with_llms_as_a_benchmark_by/ GitHub: https://github.com/lmgame-org/GamingAgent —————— Follow me: https://www.instagram.com/wesjh_/ X: https://twitter.com/wesjh_  @chubbies, @chubbiesshorts Subscribe to the podcast! Apple: https://podcasts.apple.com/us/podcast/wes-henderson-podcast/i d1570344602 Spotify: https://open.spotify.com/show/2WPDYKlvgoYbZWLPMMSxHa?si=O3ZEC-lORS2FLAs0e_xx6w Support the Channel with my affiliate links! https://havenathletic.com/WES-HENDERSON. Code WES-HENDERSON Shop Lululemon: https://creatoriq.cc/3sKggwz https://www.patreon.com/weshenderson Best active eyewear and triathlon gear in the game! Visit ROKA! https://www.avantlink.com/click.php?tt=cl&merchant_id=cca349d7-b121-43c4-a8a4-56bc2f084318&website_id=04d4bb09-ade8-466e-8f50-d02b5ef94363&url=https%3A%2F%2Fwww.roka.com%2F Thorne: Most science backed, and clinically validated supplements on the market - Get 20% off your first order with my link: https://thor.ne/qOtrs My choice for CBD - cbdMD - 15% OFF - https://cbdmd.t7c9v8.net/ZkMEk Badass Coffee Company - NVR Rest - Use ‘WES’ to get 10% off - https://nvrrest.com/ Shop on Amazon! https://www.amazon.com/?&_encoding=UTF8&tag=wesjh-20&linkCode=ur2&linkId=70b15611910ce468775b76425895bbae&camp=1789&creative=9325 To get your own Eight Sleep mattress or pod pro cover use the following link and code 'WES' for $100 Off at checkout: https://eight-sleep.ioym.net/qnANPb