Skip to the content ForoHackers
Log inJoin

A learning-rate experiment you can run in 20 lines

Deep learningMachine learningOptimization
#1084 · gradient_descent_hiker · · Report

New here. I like small experiments that show why a knob matters. Learning rate, on the simplest possible function: f(x) = 10 x^2, whose gradient is 20 x.

def run(lr, steps=50):
    x = 5.0
    for _ in range(steps):
        x -= lr * 20 * x
    return x

for lr in (0.001, 0.01, 0.05, 0.1, 0.11):
    print(lr, run(lr))

Each step multiplies x by (1 - 20 * lr). That one factor predicts everything:

  • lr = 0.001: factor 0.98, it creeps toward 0 and is still far after 50 steps.
  • lr = 0.01: factor 0.8, converges nicely.
  • lr = 0.05: factor 0, lands exactly on 0 in the first step.
  • lr = 0.1: factor -1, bounces between 5 and -5 forever.
  • lr = 0.11: factor -1.2, each step overshoots more than the last. Divergence.

The stable range is 0 < lr < 2 / curvature, here 2 / 20 = 0.1. Real models have thousands of directions with different curvatures, and the sharpest one sets the limit for all the others. That is the intuition behind why training is touchy about learning rate, and why decaying it helps near the end.

Try it before you read the output, and write down your prediction. Measuring your own surprise is the useful part.

#1680 · lambda_llama · · Report

This is the same bound as in my parabola thread: there the curvature was 2, so the limit was lr < 1. Same maths, different constants. The factor (1 - lr * curvature) is the whole story for a quadratic.

#1681 · rubber_duck_supervisor · · Report

Before running it: what do you expect run(0.1) to print, and does the answer depend on whether steps is odd or even?

#1682 · off_by_two · · Report

Even steps gives 5.0, odd gives -5.0. A lovely trap: a training loop that "does not improve" can be perfectly stable and oscillating at the edge.

#1683 · gradient_descent_hiker · · Report

Correct on both. Next thing I want to test is what happens with two directions of very different curvature, say f(x, y) = 10 x^2 + 0.1 y^2. I expect the large one to dictate the learning rate and the small one to crawl.

Log in or join to reply.