Lunar Lander: Three Agents, One Landing

Three reinforcement learning agents learn to land a spacecraft between two flags in the LunarLander environment. Each one comes from a different family of methods, and all three face the same task, the same scoring, and the same 30 test landings.

PyTorchDQNDouble DQNPPOGymnasium

135,816

steps for Double DQN to solve

239

DQN test average

80%

landing rate, DQN and PPO

PPO configurations tested

Problem

Understanding the challenge and user needs

Algorithm comparisons in textbooks rarely share conditions. Whether value-based or policy-gradient methods learn faster, land more reliably, or overestimate their own performance depends on measuring them side by side.

Approach

Strategy and methods used to address the problem

DQN, Double DQN, and PPO each trained until solving the task, with environment steps counted to the point of solution. A sensitivity study varied nine PPO configurations, and a separate analysis tracked how far each value-based agent overestimated its returns.

Training reward curves for DQN, Double DQN, and PPO — Training reward for each agent, with the solved threshold of 200 marked.

Return against environment steps for the three agents — Plotted against environment steps, the two value-based agents reach the threshold sooner than PPO.

Predicted action values for DQN and Double DQN over training — DQN's predicted action values run above Double DQN's, the overestimation that the second network corrects.

Three small charts of PPO return, each varying one setting — The sensitivity study varies one PPO setting at a time across nine configurations.

Final Solution

The final design and outcome

Double DQN solved the task soonest, at 135,816 environment steps, against 144,933 for DQN and 225,280 for PPO. On 30 test landings DQN averaged 239 points and PPO 216, each landing successfully 80 percent of the time, while Double DQN averaged 167 and landed 47 percent.

The trained DQN agent lands between the flags.

The Double DQN agent flies the same approach.

The PPO agent descends under a policy learned without a value table.

Bar charts of test reward and landing rate for the three agents — Test reward and landing rate over 30 episodes separate the agents more than their training curves do.

Three agents on the same task

Agent	Solved at	Steps to solve	Test average	Landing rate
DQN	Episode 270	144,933	239.2	80%
Double DQN	Episode 256	135,816	166.5	47%
PPO	Iteration 110	225,280	215.7	80%

Test figures cover 30 landings per agent.

Impact & Validation

Results, learnings, and outcomes

Fastest to train did not mean most reliable to fly. The comparison separates sample efficiency from test performance, a distinction that matters whenever a model moves from training to use.