Teaching Robots to Walk: PPO and Imitation Learning

Source codeem-ech/rl-ppo-imitation-learning

Two simulated robots, a two-legged walker and a four-legged ant, learn to move in the MuJoCo physics engine. A PPO expert learns each task from scratch, behavioral cloning and DAgger distil the expert into a student, and the student then continues training on its own.

PyTorchStable-Baselines3MuJoCoPPODAgger

6,208

DAgger student on the walker

6,043

its PPO teacher

4.0x

gain from normalizing observations

Problem

Understanding the challenge and user needs

Reinforcement learning from scratch spends millions of steps rediscovering how to stand. Imitation offers a shortcut, but a student that only copies demonstrations fails as soon as it drifts into situations the expert never showed it.

Learning curves for PPO from scratch and two warm-started variants on the walker — Warm-started curves climb toward the expert's level within the first half million steps while the from-scratch agent stays low.

Approach

Strategy and methods used to address the problem

A five-person team trained PPO experts for 8 million and 10 million steps, cloned them with behavioral cloning and with DAgger, and then used each clone as a warm start for further PPO training. Ablations isolated noisy experts and observation normalization, and an off-policy SAC expert provided a second reference.

Side by side frames of the expert walker and the cloned student — The PPO expert, left, and the student cloned from its demonstrations, right, walk the same course.

Line chart of DAgger return by iteration on the walker — DAgger improves with each round of expert corrections and reaches the expert's return by the twelfth.

Line chart of student performance as expert action noise increases — A noisy teacher ruins the walker student almost at once; the ant student holds up through moderate noise.

Line chart of return against number of expert episodes — More demonstrations help the walker steadily, with wide variation across five seeds.

Final Solution

The final design and outcome

DAgger students matched or exceeded their teachers: 6,208 against 6,043 on the walker and 6,564 against 6,293 on the ant. After 1.5 million steps, a warm-started walker scored about 5,700 where the same algorithm from scratch scored about 1,126. Observation normalization alone changed the walker's return fourfold.

Grouped bar chart of return at 1.5 million steps for three training strategies — At 1.5 million steps, both warm-started agents far outscore PPO trained from scratch on the walker.

Bar chart of return with normalized and raw observations — Normalizing observations quadruples the cloned walker's return and barely changes the ant's.

Four ant robots side by side, one per policy — Four ant policies run side by side: the PPO expert, the cloned student, DAgger, and SAC.

Average return over 20 evaluation episodes

Experiment	Walker2d	Ant
PPO expert	6,043	6,293
Behavioral cloning	5,719 (95%)	6,237 (99%)
DAgger	6,208	6,564
PPO from scratch at 1.5M steps	about 1,126	about 4,965
Imitation then PPO at 1.5M steps	about 5,700	6,600 to 6,900
Cloning with and without normalized observations	4,654 and 1,163	5,679 and 5,946

A SAC expert reached 7,295 on the ant in 3 million steps against PPO's 6,293 in 10 million.

Impact & Validation

Results, learnings, and outcomes

The results give a practical rule for training budgets: demonstrations buy most of the performance for a fraction of the steps, and the remaining steps are better spent refining than rediscovering.

Presentation

The deck, slide by slide

Title slide: Learning to Move, in pixel-art video game style. Five pixel-art character avatars stand on grass, labelled Marco, Em, JuanJo, Lea and Matteo, next to a green pipe. Bottom right: Press Start.

Slide 1 of 30