Gradient Descent and the Learning Rate

Everything that learns from data does it by taking small steps downhill. The only real question is how big a step to take, and the answer is stranger than "as big as you dare": there is a rate that arrives in one move, and a rate past which the thing never arrives at all.

Worked example — Gradient Descent and the Learning Rate

Step multiplier r	0.9000
What this rate does	Converges directly
Convergence stops at	2.000
One-step rate	1.000
Position after 20 steps	0.3647
Loss there	0.0665
Fraction of the starting loss	0.0148
Steps to bring \|x\| under 10⁻⁶	142

These are the values the calculator shows when the page opens. Change any input above to recompute them.

Loading interactive simulation...

Lesson

The theory — Gradient Descent and the Learning Rate

Gradient descent is one line repeated: stand somewhere, measure the slope, and step downhill in proportion to it. x ← x − η f′(x). The step size η is not part of the problem — the loss curve is given, the gradient is computed, and η is the one number a person chooses. On a quadratic of curvature a the arithmetic is exactly solvable, which is why this page uses one: every later claim about learning rates is a claim about 1 − ηa.

What each symbol means

η: the learning rate, the only thing you choose
a: the curvature of the loss, which the problem chooses for you

Assumes: One variable, a fixed step size and an exactly known gradient. Real training breaks all three at once — millions of parameters, a step size on a schedule, and a gradient estimated from a small batch — and the one thing that survives intact is the boundary at 2/a, because it comes from the curvature in a single direction and a real loss surface has one of those per eigenvector.

The edge belongs to the surface, not to your run 🖖

Drag the starting point anywhere you like, out to 10 or across to the other side of the valley, and 2/a does not move. Nor does it care how many steps you take. The threshold is fixed by the curvature alone, because the step multiplier r = 1 − aη contains no x₀ and no step count at all. The whole run is x₀ multiplied by r as many times as you asked, so whether it shrinks depends on r by itself. Starting closer to the bottom buys you no extra headroom, it only delays the point at which a doomed rate becomes obvious. Set the curvature to 1.6 and the rate to 1.3, then sweep the starting point across its whole range: the table says Diverges at every position, including the ones half a unit from the minimum.

Overshooting costs nothing by itself 🖖

Set the curvature to 1, which puts the one-step rate at η = 1. Now try 0.9, and then 1.1. Both bring |x| under 10⁻⁶ in seven steps. One of them creeps down a single wall of the valley; the other jumps clean over the bottom on every step and lands on alternating sides. They cost the same because the multiplier is 1 − aη in one case and its negative in the other, and a negative multiplier shrinks a number exactly as fast as its positive twin. The symmetry is not approximate. Try 0.5 and 1.5: twenty-two steps each. So a zig-zagging path tells you which side of the best rate you are standing on and nothing whatever about whether you have gone too far. For that, the only number that matters is 2/a.

Why real learning rates are still guessed 🖖

This page has one curvature because it has one dimension. A real network has millions of parameters and its loss surface curves by a different amount in every direction, but the reasoning survives with a replaced by the largest curvature anywhere in the surface, the top eigenvalue of the Hessian. One learning rate has to be safe in every direction at once, so the steepest direction decides when the whole thing blows up. The flattest direction decides how long you wait, since progress along it is governed by its own small curvature. The ratio between the two is the condition number, and it explains why a rate tuned on one problem transfers so badly to the next. Nothing about the loss announces either eigenvalue in advance, and computing the largest one for a hundred million parameters is not a thing anybody does before breakfast. Momentum, and per-parameter methods like Adam, exist to widen the gap between what is stable and what is quick.

Problem solved in full

Your learning rate of 0.4 is working. Find the exact curvature at which it stops working, and say what the failure looks like on either side of that point. Start from the tool at η = 0.4, a = 1, x₀ = 8, where the Step multiplier row reads 0.6000 and the path walks quietly down one wall.

Open this problem in the tool
1. Every question on this page is a question about |r| < 1, where r = 1 − aη. The rate is fixed at 0.4 and the curvature is what you are allowed to move, so write the condition in terms of a rather than η.
  
  $x_{n + 1} = x_{n} - η f^{'} (x_{n}) = (1 - a η) x_{n} ⟹ r = 1 - a η$
2. |1 − 0.4a| < 1 unfolds to −1 < 1 − 0.4a < 1, then to 0 < 0.4a < 2, so a < 5. The rate survives every curvature below 5 and none above it. This is the familiar η < 2/a read the other way round, as a < 2/η. Same inequality, and the second arrangement is the one that answers what was asked.
  
  $∣1 - 0.4 a ∣ < 1 ⟺ 0 < 0.4 a < 2 ⟺ a < 5 = \frac{2}{η}$
3. Set a = 5 exactly and read the table. r = −1.0000, and Position after 20 steps prints 8.000, the number it started at. The fraction of the starting loss is 1.000. This is the boundary itself: each step lands as far from the minimum as the last one did, on the opposite side, so the path bounces between 8 and −8 for as long as you leave it running.
  
  $a = 5 : r = - 1, x_{20} = x_{0} = 8$
4. Now nudge to a = 5.1. r = −1.0400, and after twenty steps x is 17.529 instead of 8. That is real divergence, and it is worth seeing how mild it looks: twenty steps roughly doubled the distance. A run just this side of the boundary can pass for merely noisy for a long time before the numbers turn obviously absurd.
  
  $a = 5.1 : r = - 1.04, x_{20} = 17.529$
5. Back the other way, a = 4.9 does still converge. r = −0.9600, |r| is under 1, and the table says Overshoots, converges. But Steps to bring |x| under 10⁻⁶ now reads 390, against 32 at a = 4. Almost-stable is close to useless.
  
  $a 14 4.9 5 5.1 r 0.6000 - 0.6000 - 0.9600 - 1.0000 - 1.0400 behaviour direct overshoots overshoots neither diverges steps to 1 0^{- 6} 3232390 \infty \infty$
Answer

a = 5, which is 2/η. Below it the rate converges, at it the loss never changes, above it the loss grows without limit. The expensive part is the approach. Near the boundary |r| is 1 − a·g, writing g for the gap 2/a − η, so log|r| ≈ −a·g and the step count is roughly log(|x₀|/tolerance) divided by a·g. It goes as one over the distance to the edge: halve the gap and you double the work. At a = 5 from x₀ = 8, gaps of 0.05, 0.025 and 0.0125 need 56, 120 and 247 steps, and multiplying each count by its own gap gives 2.8, 3.0 and 3.1, closing on log(8 × 10⁶)/5 = 3.18. That is why a tuned learning rate never sits at the edge of stability. The last safe sliver before the boundary is also among the slowest ground on the whole axis, and the fastest rate of all is the one in the middle at 1/a.

$a^{*} = \frac{2}{η} = 5$

Learning path

Fitting a line to data

References (3)

Where the method comes from. Two pages, and the idea is already complete: follow the gradient downhill in steps. A.-L. Cauchy, “Méthode générale pour la résolution des systèmes d’équations simultanées.” Comptes Rendus de l’Académie des Sciences 25, 536–538, 1847.
The tool's central claim, in the form it takes for a real network: with a as the largest eigenvalue of the Hessian, the fastest single learning rate is 1/a and anything past 2/a diverges. This is the source for insight block 3. Y. LeCun, L. Bottou, G. B. Orr and K.-R. Müller, “Efficient BackProp.” In Neural Networks: Tricks of the Trade, Lecture Notes in Computer Science, 9–50. Springer, 1998.
The convergence analysis behind the step count: why the work needed grows with the ratio of the largest curvature to the smallest, which is the condition number named in insight block 3. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press, 2004.

By the 42Math team · Reviewed against published sources. Corrections welcome.