[AI Robotics Field Notes (2)] What Role Should Reinforcement Learning Play in Robots?

After AlphaGo, reinforcement learning looked like an algorithm that could learn anything. The picture was that if you repeated the cycle of trying, being evaluated, and finding a better answer indefinitely, any task could be mastered.

Applied to robots, the results were different. Training took far too long, the training process frequently collapsed into instability, and a training recipe that worked on one task did not carry over to another. Papers say "do it like we did," but because they start from highly specialized setups, they rarely reproduce as-is.

So we redefined the role of reinforcement learning. Not the lead that builds a policy from scratch, but the corrector that refines an already-built policy to fit the field.

1. Reinforcement learning learns by trial and error

Boil reinforcement learning down to a phrase and it is trial and error. Try an action, get the result evaluated, and repeat to find better actions. It is a loop in which the robot observes a state, acts, and receives a reward.

There is one point here that is frequently misunderstood. The reward is not something the environment gives. The real world does not hand out scores, so the reward signal is ultimately designed by people. Anything that can score consistently — an algorithm or a human observer — can serve as the reward, and the quality of that design determines what gets learned.

When reward design goes wrong, the robot repeats behaviors that collect points instead of achieving the goal — because it optimizes the reward itself, not the intent of whoever built it.

2. The field does not honor reinforcement learning's assumptions

Reinforcement learning theory stands on three assumptions: that knowing the current state is enough to predict the future, that the robot can observe the true state of the environment in full, and that the rules of the environment do not change over time.

A factory guarantees none of the three. Cameras and sensors see only part of the environment. Lighting changes, tooling wears, material lots vary. What makes it worse is that the math training the policy does not know this. A trained policy operates on the assumption that the world it sees is the whole world and that the rules never change.

So this gap has to be closed with architecture and data, not with the algorithm. Missing observations are compensated for by adding modalities — force-torque sensors, tactile sensors, depth cameras — and a changing environment can only be taught with data that captures the change.

3. So we don't make it learn from scratch

Learning by trial and error from the ground up is hard to make work in the field. It takes too long, and the trial and error cannot be absorbed on a real production line.

The structure we adopt splits the policy into two layers. The lower layer, the base policy, handles large motions — moving, grasping, releasing. Built from demonstration data, it is the skeleton of the behavior. The upper layer, the residual policy, handles the small, precise, domain-specific motions the base cannot manage.

Reinforcement learning goes in the upper layer. With the large motions already learned, reinforcement learning refines the last few millimeters and few newtons on top of them. As the space to be explored narrows, training time and instability shrink together.

4. How we are using it at AIVEX

From this perspective, here are two directions AIVEX's AI Robotics group has experimented with.

One is fine-tuning a pre-trained policy with reinforcement learning. We inject learnable noise into a flow-based policy trained on demonstration data, letting the policy find for itself the action space worth exploring. It is not exploration that wanders at random, but exploration that looks for better answers around motions the policy already knows how to perform.

The other leaves the policy body untouched. We overlay an optimized transformation only on the model's intermediate internal representation to adjust the output motion. Since the body does not need to be retrained, performance can be raised while preserving an already-validated policy.

Both directions also revealed a limit. The adjusted policy improved near its training conditions, but when conditions shifted — such as a large change in the workpiece's position — that improvement did not hold. It was a reminder that the corrector only refines the policy; generalization ultimately falls to the base policy and the data.

In closing

The direction we are pushing is "learning that adapts to the field as quickly as possible, from as few demonstrations as possible." Within that, the place of reinforcement learning is clear.

• Reinforcement learning is not the lead that builds a policy from scratch, but the corrector that fits a built policy to the field.

• Instead of exploring from the ground up, we start from a good base policy and make it better.

• The gap between theoretical assumptions and the field is closed with sensor modalities and data, not with the algorithm.

• The next challenge is data, not the algorithm. We are looking at curriculum datasets that break tasks into sub-tasks and collect them in stages, and at ways to obtain a base policy better than the demonstrations without online learning.