Balancing a Pole: From Q-Tables to Deep Q-Networks

Three agents learn to keep a pole upright on a moving cart. The first two store values in a table over a discretized state space; the third replaces the table with a neural network.

TensorFlowDQNDouble Q-LearningHyperparameter Sweeps

230

episodes for the DQN to solve

349

average over 100 test episodes

1,396

episodes, single Q-Learning

3,290

episodes, Double Q-Learning

Problem

Understanding the challenge and user needs

CartPole's state is continuous, so a table needs bins, and the choice of bins, learning rate, and discount decides whether the agent learns at all.

Learning curves for three state discretizations — The number of bins decides the outcome: a coarse grid solves the task, a fine one stalls.

Approach

Strategy and methods used to address the problem

A sweep over learning rates and discount factors mapped which settings solve the task and how long each takes. Double Q-Learning tested whether two tables reduce overestimation, and a Deep Q-Network removed the bins entirely.

Heat maps of final reward and episodes to solve across learning rate and discount — Final reward and episodes to solve across learning rates and discount factors. The lowest learning rate finishes far below the others.

Learning curves for three exploration schedules — The exploration schedule alone separates a run that solves the task from one that never comes close.

Grid of nine learning curves from the hyperparameter sweep — The nine learning curves behind the heat maps.

Final Solution

The final design and outcome

The Deep Q-Network solved the task at episode 230 and averaged 349 points over 100 greedy evaluation episodes. Double Q-Learning needed 3,290 episodes on average against 1,396 for single Q-Learning.

Learning curves for single and Double Q-Learning with shaded variation — Single Q-Learning reaches the solved line well before Double Q-Learning.

Training reward and evaluation histogram for the final tuned run — The final tuned run: training reward at left, the spread of 100 greedy evaluation episodes at right.

Episodes needed to solve the task

Agent	Episodes to solve
Deep Q-Network	230
Tabular Q-Learning	1,396 plus or minus 404
Tabular Double Q-Learning	3,290 plus or minus 1,220

The final DQN averaged 349.10 plus or minus 21.77 over 100 greedy episodes.

Impact & Validation

Results, learnings, and outcomes

The more careful estimator learned more slowly here, a reminder that a fix for one failure carries a cost of its own.