Learning by Trial: Tabular Reinforcement Learning

Monte Carlo, TD(0) SARSA, n-step TD, and SARSA(lambda) each learn to cross a frozen lake without falling through the ice. Every algorithm runs from first principles in NumPy on Gymnasium environments.

Reinforcement LearningMonte CarloTD LearningGymnasium

66%

TD(0) success, 8 by 8 slippery

24%

Monte Carlo, same lake

75%

three-step TD, 4 by 4 slippery

algorithms from scratch

Problem

Understanding the challenge and user needs

On a slippery lake an agent moves in its intended direction only part of the time. Methods that wait until the end of an episode to learn struggle when most episodes end in a hole.

The eight by eight FrozenLake environment with the agent near the top right — The eight-by-eight lake, with the agent partway to the goal.

Approach

Strategy and methods used to address the problem

Each algorithm trained on four-by-four and eight-by-eight lakes, with and without slipping, and on the Taxi environment. Heat maps of the learned value tables and greedy policies show what each agent came to believe about every square.

Heat maps of learned values with arrows for the greedy policy on four lakes — Learned values and greedy policies on four lakes. Arrows show the action each square prefers; H marks a hole and G the goal.

Rolling success rate over training on four lakes — Rolling success during training. Slippery lakes take far longer to learn than fixed ones.

A custom grid world with lava squares, a viewpoint, and a goal — A custom Volcano grid adds lava and a scenic viewpoint worth more than the goal.

Learned values and policies on the Volcano grid under four conditions — Learned values and greedy policies on the Volcano grid, with and without slipping.

The Taxi environment with a taxi, a passenger location, and a destination — The Taxi task: pick up a passenger and deliver them to the right corner.

Final Solution

The final design and outcome

On the eight-by-eight slippery lake, TD(0) reached the goal in 66 percent of evaluation episodes against 24 percent for Monte Carlo. On the four-by-four slippery lake, three-step TD performed best at 75 percent.

Bar chart of success rate against the number of steps n — Success rate by number of look-ahead steps. Three steps does best; long look-aheads collapse.

Bar chart of success rate against the trace decay lambda — SARSA(lambda) holds steady across most trace-decay values and fails only at the extreme.

Success rate on slippery lakes

Lake	Method	Evaluation success
8 by 8	Monte Carlo	24%
8 by 8	TD(0) SARSA	66%
4 by 4	n-step TD, n = 3	75%

Impact & Validation

Results, learnings, and outcomes

Learning from each step rather than each finished episode nearly tripled the success rate under uncertainty, the core idea behind every deep reinforcement learning method that follows.