Uff, this chapter won’t be conventional, as we’re setting aside the usual history of agents. Originally, we had that chapter prepared, but a couple of recent developments – Google’s Project Jarvis and the π0 model from Physical Intelligence – gave us a fresh angle. So, today’s agentic episode will dive into the iconic Just A Rather Very Intelligent System, aka J.A.R.V.I.S. – a brilliant (and positive!) example of Capable AI, but also… a butler.

In today’s episode, we’ll explore:

Let’s dive in!

The idea of J.A.R.V.I.S. – and the vision of a tech-driven superhero – deeply inspired my colleague Alyona, so much so that she chose to pursue engineering, focusing on aircraft control systems and autopilot programming. She wanted to learn the advanced tech needed to create intelligent systems like J.A.R.V.I.S. Alas, this was back in 2015, and personal assistant technology was far from ready to meet that dream.

J.A.R.V.I.S. made its cinematic debut in 2008’s Iron Man, voiced by Paul Bettany, but if we rewind to the original Marvel comics, Jarvis wasn’t AI at all. He was Edwin Jarvis, a loyal, very human butler who managed Tony Stark’s home and kept the Avengers’ mansion in order – something many of us would happily delegate to AI today.

Then, in 2008, Marvel decided to merge the idea of Edwin Jarvis with a lesser-known comic character: H.O.M.E.R. (Heuristically Operative Matrix Emulation Rostrum), an early, prototype-style digital assistant. While H.O.M.E.R. didn’t have J.A.R.V.I.S.’s interactive flair, it laid the groundwork for Stark’s evolving AI systems by handling data processing and basic operations. Practical but limited, H.O.M.E.R. was a task-oriented system that worked on strictly functional levels – without the learning and interaction that J.A.R.V.I.S. would later embody.

Basically, H.O.M.E.R. is what we have now.

J.A.R.V.I.S. is what we’re still trying to build.

But what we’re also looking forward to is for AI to become our butler, or simply a physical helper, right?

Zuckerberg’s Jarvis and Google’s Project Jarvis

Meta AI

Not that many people remember it, but in 2016, Facebook CEO Mark Zuckerberg embarked on a personal project to develop an AI assistant for his home, inspired by the same fictional J.A.R.V.I.S. from the Iron Man series. His AI, also named Jarvis, was designed to control various household functions, including lighting, temperature, music, and security systems. It could recognize guests at the door, manage appliances, and even entertain his daughter, Max. Zuckerberg utilized programming languages like Python, PHP, and Objective-C, employing AI techniques like natural language processing and facial recognition to build this system. He spent about 100 hours (!) building Jarvis that year. Today, with so many coding assistants, that time would be reduced to… a day? If you're still wondering whether AI can be helpful, that's the right comparison to keep in mind. Overall, his blog is a fascinating read in retrospect.

In 2023, that Jarvis became an inspiration and basis for Meta AI agents.

Google’s Project Jarvis

The most recent Jarvis-inspired project is Google’s upcoming AI agent (also called Jarvis), designed to handle consumer tasks by interacting directly with a user’s web browser, primarily Chrome. Unlike other digital AI agents that are more business or code-oriented, Google’s Jarvis simplifies everyday web-based tasks for consumers, such as booking flights, purchasing items, and managing returns. Powered by Google’s Gemini LLM, Jarvis interprets browser screenshots to navigate pages and perform actions like clicking buttons and entering text. According to The Information, it currently operates at a slower pace as it carefully processes each action. Initial releases are expected to reach a limited group for testing its usability and privacy handling, particularly since it requires access to sensitive information to complete tasks.

Getting Back to Being a Butler: Robot Models

With so many digital AI agents currently swarming the space, many people are beginning to wonder why AI handles all the creative work (writing poetry, drawing) but still can’t fold laundry or wash dishes. That’s why a tweet from Sergey Levine caught my eye: