Q-learning in a maze.
Reinforcement learning strips decision-making to its core: an agent, an environment, a reward. No labels, no gradients on a loss you chose - just trial, error, and a value function that slowly crystallises into a policy. This page trains one live, on a maze that did not exist until you loaded it.
The setup
The maze is a Markov Decision Process: states are cells, actions are the four moves, walls bounce you back, every step costs −1 and reaching the goal pays +50. Costing each step makes the optimal policy the shortest path - the discounted value of a state decays with every extra step it takes to reach the goal. What the agent is really estimating is the optimal action-value function, defined by the Bellman optimality equation:
Dynamic programming solves this exactly - but it needs the full transition model. The interesting case is when the agent must learn from experience only.
The algorithm
Q-learning turns the Bellman equation into a stochastic update applied along the agent's trajectory:
The bracket is the temporal-difference error - surprise, in one number. Exploration uses an ε-greedy policy: act randomly with probability ε (decayed over episodes from 1.0 to 0.05), greedily otherwise. Because the update bootstraps on the greedy next value rather than the action actually taken, Q-learning is off-policy: it learns the optimal policy even while behaving exploratorily. That is exactly why it wins on this problem - in the lab comparisons (MC control, SARSA, expected SARSA on cliff-world and blackjack), Q-learning converged to the shortest path most reliably, where on-policy SARSA learns the safe path its own exploration noise forces it to take.
What you are watching
The amber dot is the agent, moving one action at a time: you watch it actually walk the maze, bumping into walls and wandering while ε is high, then beelining once it has learned. Drop steps / frame to 1 to follow every move, or push it up to fast-forward the training. The heatmap is : value flows backwards from the goal like ink in water - the discount makes each cell worth a fraction of its best neighbour. The faint white line is the current greedy policy rolled out from the start; early on it dead-ends or loops, then snaps onto the optimal route (compare its length to the BFS optimum). The learning curve below shows the classic RL signature: hundreds of wasted steps per episode while ε is high, collapsing to near-optimal as the value function fills in and exploration anneals.
Hit New maze and everything resets: a fresh maze is generated (recursive backtracker, plus a few knocked-down walls so there are genuinely competing routes) and the agent starts from zero knowledge. Everything upstream of this - multi-armed bandits and the exploration-exploitation dilemma, policy iteration with full dynamic programming, Monte Carlo against TD prediction - is what the maze quietly assumes; it is where all of it lands in one picture.