Why is clearing a kitchen so hard for robots?

After breakfast, the table is hardly a dramatic scene: a cup, a plate, a cloth and a few things nobody put away. We clear it while thinking about something else. For a robot, that ordinary moment brings several difficult questions together. What should move? Where should it go? Can it be picked up without disturbing everything around it?

This is why household work is such an interesting test of physical AI. The goal is easy to explain, but the scene refuses to stay the same. Looking at one cup helps us understand the larger challenge: turning a learned movement into a useful habit that survives an ordinary, changing room.

“Pick up the cup” leaves a lot to the robot

A picture can tell the robot that there is a cup. It does not automatically tell it whether the handle is reachable, whether the cup contains liquid or whether the cloth will move with it. The arm has to turn an interpretation into contact with an object. A small error in that contact can undo a perfectly sensible plan.

We also leave the purpose unstated. “Clear the table” might mean put dishes in the sink, leave someone’s unfinished drink alone and keep a notebook dry. People fill in those details from experience. A useful robot needs to connect the instruction to this particular room, rather than treat every cup as the same pick-and-place exercise.

In the interactive example below, the instruction stays the same while the cup and the approach change. Watch the arm, then read what changed in the task. The useful question is not whether the motion looks smooth. It is whether the robot has noticed that its old plan no longer fits.

Figure 1

Same instruction, a different table

Choose what happens on the table. The arm and the explanation change together.

Choose a situation

Familiar layoutThe table changesPull back and reassess

1. ObserveObject, contact and available space
2. ActChoose a reachable movement
3. CheckCompare the result with the task

The first attempt has a clear approach.

What changed?

The cup is in the expected place.

In this example

The arm approaches along the familiar route.

What it means for the work

A useful starting test, before anything changes.

Repeating the old approach is no longer enough.

What changed?

The cup moves and an object narrows the approach.

In this example

The illustrated arm turns toward the changed position.

What it means for the work

The task needs a fresh observation and a new approach.

A retreat can keep an unfinished task manageable.

What changed?

The approach is not suitable for continuing.

In this example

The arm withdraws instead of forcing the movement.

What it means for the work

The person can resolve a specific difficulty before another attempt.

Where experience comes from

Robot-learning research is trying to make experience less tied to one table. RT-2 connects visual and language knowledge to action prediction. Open X-Embodiment pools experience from different robots. These are complementary ideas: understanding more about an object and learning from more than one machine can both help when a familiar instruction meets an unfamiliar scene. ^{[1] [2]}

DROID addresses another part of the problem by collecting manipulation demonstrations across varied real settings. The point is not simply to make the dataset bigger. If the robot sees only tidy arrangements, it gets little practice with the small irregularities that make a kitchen difficult. Varied experience gives us a better starting point for testing those changes. ^[3]

Physical Intelligence’s π0.5 work makes this question tangible through household tasks in new homes. Its reported experiments explore generalisation beyond the training setting, while acknowledging failures. I read that as progress toward the problem, not its completion. A kitchen contains a sequence of small decisions; a few successful skills still have to work together. ^[4]

The next attempt is where learning becomes visible

Return to our table. Someone has put the cup closer to the cloth. If the robot repeats the previous trajectory, we learn that it can reproduce a movement. If it looks again and chooses a different approach, we begin to learn something about adaptation. The distinction is small on a video and large in everyday use.

This changes how I would run a test. I would keep ordinary variations out of training, then describe exactly what the robot faced: a different position, another camera angle, an obstacle in the approach. I would record help between attempts too. Quietly straightening the cloth can make the machine appear more independent than it was.

A missed grip is worth reading carefully, rather than putting it in one failure column. Did the robot see the object? Did it reach it? Did it notice the slip? Each answer points to different work. The loop below shows why a new observation belongs after the movement: finishing the motion does not tell us that the cup arrived.

Figure 2

The cup has to arrive, not just the arm

A closed observation-and-action cycle

1. 01Observe

   Keep the scene and the instruction together.

   ↓
2. 02Choose and act

   Use a movement that fits the current conditions.

   ↓
3. 03Check and recover

   Observe again; continue, correct or ask for help.

The useful robot leaves less work for the person

The practical goal is a machine whose help remains worthwhile after we count supervision, resets and recovery. If clearing the table means standing beside the arm and arranging every object first, the impressive part has not yet become a useful service. The person should gain time or effort, not a new job as the robot’s full-time assistant.

Sometimes a controlled retreat or a clear request for help is the right contribution. It lets the person fix one specific difficulty rather than take over the whole task. Recovery time and the amount of help deserve to sit beside the success rate, because they tell us what living with the robot might actually involve.

That is the promise of physical AI I find compelling: ordinary help that survives ordinary variation. The cup is a useful beginning because we can see what went wrong and what would count as improvement. Once that behaviour is learned, the next question arrives. Can the robot still offer that help when it leaves the workbench and loses its connection?

Figure 3

The research behind the story

Further reading	The useful idea	The next question
RT-2 · 2023	The useful idea Connects visual and language knowledge to robotic action prediction.	The next question Which physical variations does that semantic transfer actually survive?
Open X-Embodiment · 2023	The useful idea Brings experience from different robot embodiments into a common research resource.	The next question What still needs adaptation for a new robot and task?
DROID · 2024	The useful idea Expands manipulation data across varied real-world scenes and tasks.	The next question Does diversity improve the behaviours that matter in our intended setting?

Sources

1. RT-2: Vision-Language-Action Models robotics-transformer2.github.io
2. Open X-Embodiment: Robotic Learning Datasets and RT-X Models robotics-transformer-x.github.io
3. DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset droid-dataset.github.io
4. Physical Intelligence: π0.5, a VLA with Open-World Generalization pi.website