The last cup on the table
Clearing breakfast looks simple until a robot tries it. One cup reveals what physical AI still has to learn about helping in a changing home.
Robotics and AI1 / 4
You are about to leave the house, but breakfast is still on the table. You stack two plates, carry a cup to the sink and leave the half-finished coffee where it is. Your mind is already on the day ahead. You probably will not remember doing any of this by lunchtime.
Now imagine handing that small job to a robot. Suddenly the things you never needed to say become the difficult part. Which cup is finished? Can the plate carry another plate? Is the cloth caught underneath? This is what draws me to physical AI: the distance between understanding a request and quietly taking a little work out of someone’s day.
We leave most of the instructions unsaid
“Clear the table” sounds like a complete instruction because we share so much background. We know a notebook should stay dry and that someone might come back for their coffee. The robot gets the words without having lived the morning. Recognising every object would still leave it with a question about what we intended.
Then there is the cup itself. It may look almost identical to yesterday’s cup while its handle faces the wrong way or a cloth lies beneath it. For us, changing the grip barely interrupts the job. For a machine, the change reaches all the way from the camera image to the pressure at its fingertips. The world gets a vote in whether the plan works.
In the scene below, moving the cup changes what the arm has room to do. That small adjustment is easy to overlook in a polished demonstration. At home, it is the ordinary course of things. Nobody wants to prepare the kitchen for the robot before the robot can prepare the kitchen for them.
Same instruction, a different table
Choose what happens on the table. The arm and the explanation change together.
The first attempt has a clear approach.
- What changed?
- The cup is in the expected place.
- In this example
- The arm approaches along the familiar route.
- What it means for the work
- A useful starting test, before anything changes.
Repeating the old approach is no longer enough.
- What changed?
- The cup moves and an object narrows the approach.
- In this example
- The illustrated arm turns toward the changed position.
- What it means for the work
- The task needs a fresh observation and a new approach.
A retreat can keep an unfinished task manageable.
- What changed?
- The approach is not suitable for continuing.
- In this example
- The arm withdraws instead of forcing the movement.
- What it means for the work
- The person can resolve a specific difficulty before another attempt.
A robot needs more than one kitchen’s worth of experience
This is where the research becomes interesting. RT-2 explores how knowledge learned from images and language can help a robot choose actions. Open X-Embodiment brings together experience from different robot bodies. Both push against the same frustrating limit: a skill can look impressive where it was taught and become fragile a few steps away. [1] [2]
DROID approaches that limit by gathering demonstrations in varied real settings. I like what that choice says about the problem. A home is not a set of objects against a neutral background. The lighting changes, the working surface fills up, and people put things where there happens to be space. Experience has to include some of that untidiness. [3]
Physical Intelligence’s π0.5 work takes the question into household tasks in new homes. Its reported successes are encouraging precisely because a different home changes many details at once. The failures matter too: they help us see how far a learned skill travels. Reading this work, I find myself thinking less about a spectacular movement and more about the next unfamiliar room. [4]
What happens when the cup slips?
Back at the table, suppose the grip misses. The cup stays where it was; the arm reaches the end of its movement. For a moment, the machine and the person watching it have different accounts of the job. The person sees a cup that still needs clearing. A system that only follows its planned motion may already be preparing the next step.
That is why another look belongs inside the skill. The robot needs to notice that the scene did not change as expected before the mistake grows into something else. A small slip need not ruin the whole task. It becomes exhausting when a person has to watch every movement just to catch what the machine missed.
A request for help can fit naturally here: the handle is out of reach, or the cloth needs moving first. We help one another this way all the time. What matters is that the request points to the actual difficulty, so a person can make one small intervention and return to what they were doing. The loop below leaves room for that second look.
Following the cup through the task
A closed observation-and-action cycle
- Observe
Keep the scene and the instruction together.
- Choose and act
Use a movement that fits the current conditions.
- Check and recover
Observe again; continue, correct or ask for help.
The morning we would like to get back
It is tempting to judge a robot by the most difficult thing it can do. For help at home, I keep coming back to a quieter question: what can the person stop worrying about? If they can leave the room, make a phone call or get ready without wondering whether the next cup will cause trouble, the machine has begun to give something back.
The studies above do not promise that morning yet. They do make its ingredients easier to see: experience beyond one setup, a connection between understanding and movement, and the ability to recover when an ordinary detail changes. None is especially glamorous on its own. Together they decide whether the help survives after the camera crew leaves.
The last cup is a modest place to begin, which is why I like it. Everyone can understand what remains to be done. The ambition behind it is much larger: a machine that learns enough about the changing world to take a little responsibility for the work we would gladly put down.
The research behind the story
| Further reading | The useful idea | The next question |
|---|---|---|
| RT-2 · 2023 | Connects visual and language knowledge to robotic action prediction. | Which physical variations does that semantic transfer actually survive? |
| Open X-Embodiment · 2023 | Brings experience from different robot embodiments into a common research resource. | What still needs adaptation for a new robot and task? |
| DROID · 2024 | Expands manipulation data across varied real-world scenes and tasks. | Does diversity improve the behaviours that matter in our intended setting? |
Sources
- RT-2: Vision-Language-Action Models robotics-transformer2.github.io
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models robotics-transformer-x.github.io
- DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset droid-dataset.github.io
- Physical Intelligence: π0.5, a VLA with Open-World Generalization pi.website