Robotics in 2025: Where We Are and Where I Think We're Going

Over the last year and a half, there has been very interesting progress in robot control using end-to-end deep learning techniques.

Traditionally, many robot applications have relied on carefully programmed motions and rules, sometimes augmented with computer vision models. These systems can work extremely well with precise hardware in controlled environments. But they can also be brittle when things change, and they are still far from dealing with the dynamics and variability of the real world.

We are now seeing a different approach: learning to control robots directly from what they observe.

## From observations to actions

What do we mean by end-to-end control?

The idea is to take inputs from the robot's sensors and feed them into a model that produces actions. Those inputs, usually called observations, might include:

- **Images from cameras:** what the robot sees.

- **Joint positions:** typically the angle of each joint. For example, a robot arm with six rotational degrees of freedom (DOF) has six joint angles.

- **Inertial measurements:** gyroscopes and accelerometers in an inertial measurement unit (IMU) help estimate the robot's orientation and motion.

- **Force or tactile measurements:** for example, how much pressure a gripper or hand is applying to an object.

The model also needs to know the task goal, which could be provided as a language instruction or another kind of input.

The output, or action space, might be the target joint angles. For our six-DOF arm, that would mean deciding what the six angles should be next, based on the observations and the task. Other systems might output joint velocities or a target position and orientation for the gripper, which a lower-level controller translates into joint movements.

A broader system could also produce other outputs, such as speech: what the robot should say while interacting with someone.

As with other machine learning problems, we are mapping inputs to outputs. The black box in the middle is a learned model, often called a policy in robotics.

Transformers are a natural fit for many of these models. They have proven useful across a wide variety of domains, particularly for modeling sequences, and robotics is no exception. Here, the sequences contain observations and actions rather than just words. A model might use recent camera images and joint positions to predict the next sequence of movements.

Transformers aren't the only option, though. For example, [diffusion policies](https://diffusion-policy.cs.columbia.edu/) can generate action sequences using convolutional or transformer architectures.

## Where does the training data come from?

If you are a machine learning engineer, or an AI engineer in today's terms, this is probably your next question.

One thing is clear: we don't have the same abundance of training data that we have for text and images on the internet.

There is no comparable, freely available source that continuously records what a human sees, what their skin feels, where their limbs are, and exactly how they move. And even if we had all of that, translating human movements into commands for a particular robot would still be a challenge.

So where can we get useful data?

### Internet videos

Yes, we have an abundance of videos on YouTube and elsewhere. But there are a few problems:

- Many videos show a third-person perspective, rather than what a robot's cameras would see.

- Many show people talking, with little physical interaction relevant to the tasks we want robots to learn.

- Videos don't directly record joint movements, forces, or contact information. We can estimate some movements with pose estimation models, but that is different from having the actual measurements.

Researchers are exploring ways to turn these videos into more useful training data. These include estimating human poses and movements, mapping them to robot motions, and editing videos to replace human limbs with robots. For example, [Phantom](https://phantom-human-videos.github.io/) removes the human arm through inpainting and overlays a rendered robot.

Still, making a video look like a robot performed a task doesn't automatically give us the correct robot actions or the physics behind them. There is useful information in these videos, but extracting it is not straightforward.

### Simulation

Another approach is to create synthetic data in simulation.

Instead of recording every interaction in the real world, we build a virtual environment where a robot can perform tasks. We need physics that captures what matters for the task and, if the policy uses cameras, useful visual observations. That doesn't necessarily mean every scene needs to look perfectly photorealistic.

Two important tools here are [NVIDIA's Isaac Sim](https://developer.nvidia.com/isaac/sim) and [Google DeepMind's MuJoCo](https://mujoco.org/). They have different strengths: Isaac Sim provides tools for robotics simulation, rendering, and synthetic data generation, while MuJoCo focuses on fast physics simulation.

The challenge is building the scenes and recreating the tasks. We can construct environments manually, reconstruct parts of them from real-world recordings, or teleoperate simulated robots to collect demonstrations. We can also train policies through reinforcement learning, where the robot learns by trying actions and receiving feedback from a reward function.

One very nice advantage of simulation is that, once we have a scene, we can generate variations relatively easily. We can change the objects to pick up, their positions, the surrounding furniture, lighting, textures, or the robot's appearance.

We also know the exact state of the simulated world: the joint positions, object locations, and what each simulated camera sees. We can configure the number of cameras, where they are placed, and their properties, including their field of view.

The catch is that the simulated world is still an approximation. A policy that works there may struggle on a real robot because friction, contact, lighting, or other details behave differently. This is the sim-to-real gap.

### Teleoperation

With teleoperation, we put a real robot in an environment and have a human control it to perform a task. We record the robot's observations and the actions taken during that demonstration.

Think of it as playing a video game, except you are moving a physical robot. You might be in the same room, looking directly at it, or somewhere else, controlling it over the internet through its cameras.

This gives us data from the actual hardware interacting with the actual world. But it comes with some challenges:

- We need access to the robot and the environment, which can be expensive to set up and maintain.

- Collecting demonstrations takes human time. Each attempt, including resetting the scene, takes time in the real world.

By comparison, automated simulations can often run faster than real time and in many parallel instances. They still require compute and engineering work, and teleoperating a simulated robot still takes human time. But once data collection or learning is automated, simulation can scale in ways that physical demonstrations cannot.

## How I think this will evolve

My guess is that we will need a combination of these approaches.

- **We can't assume robotics will follow exactly the same path as LLMs.** Robot foundation models are already being developed, with [π₀](https://www.physicalintelligence.company/blog/pi0) being one example. But the data problem is different. We don't have an internet-sized collection of robot observations paired with actions, freely available and ready to use. We need to think differently about how that data gets created.

- **Training will probably combine multiple sources of data.** Internet images and videos, simulated experience, and real robot demonstrations each provide something useful. I don't expect any one source to solve the whole problem.

- **I expect learning through interaction to play a major role.** Pretrained language and vision-language models give us a useful starting point, but knowing what an object is or describing a task is different from physically performing it. My guess is that reinforcement learning and other forms of post-training will play an even larger role in robotics than they did in the early development of LLMs and VLMs.

- **Robots should improve from experience over time.** As they perform tasks, they can collect data about what worked and what failed. That data can then be used to train and evaluate better policies. This won't happen automatically just because a robot is running; we still need a process for turning experience into reliable improvements.

## Robot form factors

The other question is what these robots should actually look like.

I'd start by saying that I think there is too much obsession and hype around humanoids.

By a humanoid, I mean a robot built roughly like a human: two legs, two arms, a torso, and a head, often with hands designed to resemble our own.

There are several reasonable arguments for this approach:

- **The world has been designed for humans.** A robot with a similar body should be able to use many of the same spaces, tools, and objects.

- **Humanoids could fit into existing workplaces.** In principle, we could introduce them without redesigning the entire environment. The appeal is a robot that can step into a job with relatively few changes around it.

- **We know how humans perform these tasks.** That gives us a reference for teaching robots and evaluating their behavior, even if transferring those movements is not straightforward.

- **The human body is versatile.** We know this form factor can perform a very wide range of tasks that matter to us.

- **A common form factor could help the field build on its progress.** Rather than developing everything separately for every kind of robot, we could reuse hardware designs, data, and models across similar platforms.

I think many of these arguments make sense. But they don't settle the question.

### Efficiency, cost, and safety

The humanoid form factor is not necessarily the most efficient way to perform a task.

Take a robot working on a flat floor in a single-story building. Why does it need legs? Wheels can be faster, more energy efficient, and mechanically simpler. A stable wheeled base also doesn't need to actively balance just to remain upright when it loses power.

Legs have advantages when dealing with stairs, obstacles, or uneven terrain. But not every application needs those capabilities, and we shouldn't assume their benefits justify the complexity in every case.

The same applies to hands. Five fingers might be useful for some tasks, but a simpler gripper may be enough for others.

Every extra capability has a cost: more components, more control problems, more potential failures, and often more weight and expense. A humanoid is not automatically the cheapest or safest design. That depends on the task and the environment.

### Cultural biases

I also think some of our enthusiasm for humanoids comes from culture.

Science fiction has spent decades showing us robots that look like people. It is easy to absorb the idea that this is what an advanced robot is supposed to look like.

We also seem drawn to machines that resemble us. I don't dismiss this. In applications such as hospitality or healthcare, a familiar appearance might help some people feel more comfortable interacting with a robot.

We are hackable animals. Even when we know something is a machine, part of us may still respond to its face, voice, or gestures as if it were a person. I don't necessarily see that as a defect. It can be a useful feature of the interaction.

But making a robot feel familiar to us is a different design goal from making it efficient at a physical task. Sometimes those goals overlap. Sometimes they don't.

### Biological biases

Nature gives us an impressive set of designs to learn from, but it doesn't define the limits of what we can build.

Wheels are the obvious example. They are one of humanity's most useful inventions, yet animals don't move around on wheels. Biological evolution works within constraints that robot designers don't necessarily share.

So the fact that animals use legs doesn't mean legs are the best choice for robots, particularly in environments such as homes, factories, and other buildings with flat floors.

### The limits of our imagination

After seeing so many humanoids in fiction and now in real demonstrations, it can be hard to imagine alternatives.

What would a robot look like if we designed it around the task, without starting from the human body? Would it need a head? Two arms? Could a completely different, potentially weird-looking form factor do the job better?

I think there is a lot of room to explore those questions. Humanoids may be the right answer for many applications. But I'd like to see us give other designs the same attention, including designs that look nothing like us.