Bayram Eker
Menu
EN
CV Start a conversation

Home / Writing / Research

The last cup on the table

Clearing breakfast looks simple until a robot tries it. One cup reveals what physical AI still has to learn about helping in a changing home.

Robotics and AI1 / 4

A camera-equipped robot gripper pauses above a folded cloth, with a cup and wooden block on a workshop table.
The cup is easy to recognise. Lifting it without pulling the cloth is a different problem. Concept illustration.

You are about to leave the house, but breakfast is still on the table. You stack two plates, carry a cup to the sink and leave the half-finished coffee where it is. Your mind is already on the day ahead. You probably will not remember doing any of this by lunchtime.

Now imagine handing that small job to a robot. Suddenly the things you never needed to say become the difficult part. Which cup is finished? Can the plate carry another plate? Is the cloth caught underneath? This is what draws me to physical AI: the distance between understanding a request and quietly taking a little work out of someone’s day.

We leave most of the instructions unsaid

“Clear the table” sounds like a complete instruction because we share so much background. We know a notebook should stay dry and that someone might come back for their coffee. The robot gets the words without having lived the morning. Recognising every object would still leave it with a question about what we intended.

Then there is the cup itself. It may look almost identical to yesterday’s cup while its handle faces the wrong way or a cloth lies beneath it. For us, changing the grip barely interrupts the job. For a machine, the change reaches all the way from the camera image to the pressure at its fingertips. The world gets a vote in whether the plan works.

In the scene below, moving the cup changes what the arm has room to do. That small adjustment is easy to overlook in a polished demonstration. At home, it is the ordinary course of things. Nobody wants to prepare the kitchen for the robot before the robot can prepare the kitchen for them.

Figure 1

Same instruction, a different table

Choose what happens on the table. The arm and the explanation change together.

Choose a situation
  1. ObserveObject, contact and available space
  2. ActChoose a reachable movement
  3. CheckCompare the result with the task

The first attempt has a clear approach.

What changed?
The cup is in the expected place.
In this example
The arm approaches along the familiar route.
What it means for the work
A useful starting test, before anything changes.

Repeating the old approach is no longer enough.

What changed?
The cup moves and an object narrows the approach.
In this example
The illustrated arm turns toward the changed position.
What it means for the work
The task needs a fresh observation and a new approach.

A retreat can keep an unfinished task manageable.

What changed?
The approach is not suitable for continuing.
In this example
The arm withdraws instead of forcing the movement.
What it means for the work
The person can resolve a specific difficulty before another attempt.
An explanatory animation of the table example, not a robot performance measurement.

A robot needs more than one kitchen’s worth of experience

This is where the research becomes interesting. RT-2 explores how knowledge learned from images and language can help a robot choose actions. Open X-Embodiment brings together experience from different robot bodies. Both push against the same frustrating limit: a skill can look impressive where it was taught and become fragile a few steps away. [1] [2]

DROID approaches that limit by gathering demonstrations in varied real settings. I like what that choice says about the problem. A home is not a set of objects against a neutral background. The lighting changes, the working surface fills up, and people put things where there happens to be space. Experience has to include some of that untidiness. [3]

Physical Intelligence’s π0.5 work takes the question into household tasks in new homes. Its reported successes are encouraging precisely because a different home changes many details at once. The failures matter too: they help us see how far a learned skill travels. Reading this work, I find myself thinking less about a spectacular movement and more about the next unfamiliar room. [4]

What happens when the cup slips?

Back at the table, suppose the grip misses. The cup stays where it was; the arm reaches the end of its movement. For a moment, the machine and the person watching it have different accounts of the job. The person sees a cup that still needs clearing. A system that only follows its planned motion may already be preparing the next step.

That is why another look belongs inside the skill. The robot needs to notice that the scene did not change as expected before the mistake grows into something else. A small slip need not ruin the whole task. It becomes exhausting when a person has to watch every movement just to catch what the machine missed.

A request for help can fit naturally here: the handle is out of reach, or the cloth needs moving first. We help one another this way all the time. What matters is that the request points to the actual difficulty, so a person can make one small intervention and return to what they were doing. The loop below leaves room for that second look.

Figure 2

Following the cup through the task

A closed observation-and-action cycle

  1. Observe

    Keep the scene and the instruction together.

  2. Choose and act

    Use a movement that fits the current conditions.

  3. Check and recover

    Observe again; continue, correct or ask for help.

The next observation closes the loop. A completed movement is not automatically a completed task.

The morning we would like to get back

It is tempting to judge a robot by the most difficult thing it can do. For help at home, I keep coming back to a quieter question: what can the person stop worrying about? If they can leave the room, make a phone call or get ready without wondering whether the next cup will cause trouble, the machine has begun to give something back.

The studies above do not promise that morning yet. They do make its ingredients easier to see: experience beyond one setup, a connection between understanding and movement, and the ability to recover when an ordinary detail changes. None is especially glamorous on its own. Together they decide whether the help survives after the camera crew leaves.

The last cup is a modest place to begin, which is why I like it. Everyone can understand what remains to be done. The ambition behind it is much larger: a machine that learns enough about the changing world to take a little responsibility for the work we would gladly put down.

Figure 3

The research behind the story

Further readingThe useful ideaThe next question
RT-2 · 2023

Connects visual and language knowledge to robotic action prediction.

Which physical variations does that semantic transfer actually survive?

Open X-Embodiment · 2023

Brings experience from different robot embodiments into a common research resource.

What still needs adaptation for a new robot and task?

DROID · 2024

Expands manipulation data across varied real-world scenes and tasks.

Does diversity improve the behaviours that matter in our intended setting?

The original studies discussed in the essay, with a short guide to what each one explores.

Sources

  1. RT-2: Vision-Language-Action Models robotics-transformer2.github.io
  2. Open X-Embodiment: Robotic Learning Datasets and RT-X Models robotics-transformer-x.github.io
  3. DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset droid-dataset.github.io
  4. Physical Intelligence: π0.5, a VLA with Open-World Generalization pi.website

Written by Bayram Yüksel Eker. Corrections and disagreement are welcome at bayram@neuraparse.com.