The Safe Path and the Short Path: SARSA and Q-Learning

SARSA and Q-Learning learn the same two tasks: delivering a taxi passenger and walking along a cliff edge. The update rules differ in a single term, and that term changes how each agent treats risk.

Reinforcement LearningSARSAQ-LearningGymnasium

steps on SARSA's safe route

steps on Q-Learning's route

-24.6

SARSA training reward

-53.1

Q-Learning training reward

Problem

Understanding the challenge and user needs

An agent that explores will sometimes step off the cliff. Whether its learning accounts for those missteps determines the route it prefers.

Approach

Strategy and methods used to address the problem

Both agents trained on Taxi and Cliff Walking, with rolling training reward and greedy evaluation recorded for each.

Learning curves for SARSA and Q-Learning on Taxi — On Taxi, where no step is catastrophic, the two learning curves nearly coincide.

Arrow grids of the greedy policies learned on Taxi — The greedy Taxi policies for one pickup and destination pair.

Final Solution

The final design and outcome

On Taxi the two finished close together, 7.89 for SARSA and 8.14 for Q-Learning against negative 765 for random play. On the cliff, SARSA averaged negative 24.6 over its last 100 training episodes while Q-Learning averaged negative 53.1.

Two grid diagrams of the routes learned by SARSA and Q-Learning along the cliff — SARSA climbs away from the edge and takes 17 steps. Q-Learning hugs the cliff and takes 13.

Frame of the SARSA agent walking along the top of the Cliff Walking grid — The SARSA agent walks the top row, far from the drop.

Frame of the Q-Learning agent walking beside the cliff edge — The Q-Learning agent walks the row beside the edge.

Training reward curves for SARSA and Q-Learning on Cliff Walking — During training, Q-Learning's shorter route costs it more reward because exploration keeps stepping off the edge.

Two tasks, two agents

Task and measure	SARSA	Q-Learning
Taxi, greedy evaluation reward	7.89	8.14
Cliff Walking, last 100 training episodes	-24.6	-53.1
Cliff Walking, length of learned route	17 steps	13 steps

Random play on Taxi averages -765.

Impact & Validation

Results, learnings, and outcomes

Q-Learning learns the shortest route and pays for it while exploring; SARSA learns the route that survives its own mistakes. The right choice depends on whether errors during learning cost anything.