Saltar al contenido ForoHackers

A learning-rate experiment you can run in 20 lines

Aprendizaje profundoAprendizaje automáticoOptimización
#1084 · gradient_descent_hiker · · Denunciar

New here. I like small experiments that show why a knob matters. Learning rate, on the simplest possible function: f(x) = 10 x^2, whose gradient is 20 x.

def run(lr, steps=50):
    x = 5.0
    for _ in range(steps):
        x -= lr * 20 * x
    return x

for lr in (0.001, 0.01, 0.05, 0.1, 0.11):
    print(lr, run(lr))

Each step multiplies x by (1 - 20 * lr). That one factor predicts everything:

  • lr = 0.001: factor 0.98, it creeps toward 0 and is still far after 50 steps.
  • lr = 0.01: factor 0.8, converges nicely.
  • lr = 0.05: factor 0, lands exactly on 0 in the first step.
  • lr = 0.1: factor -1, bounces between 5 and -5 forever.
  • lr = 0.11: factor -1.2, each step overshoots more than the last. Divergence.

The stable range is 0 < lr < 2 / curvature, here 2 / 20 = 0.1. Real models have thousands of directions with different curvatures, and the sharpest one sets the limit for all the others. That is the intuition behind why training is touchy about learning rate, and why decaying it helps near the end.

Try it before you read the output, and write down your prediction. Measuring your own surprise is the useful part.

#1680 · lambda_llama · · Denunciar

This is the same bound as in my parabola thread: there the curvature was 2, so the limit was lr < 1. Same maths, different constants. The factor (1 - lr * curvature) is the whole story for a quadratic.

#1681 · rubber_duck_supervisor · · Denunciar

Before running it: what do you expect run(0.1) to print, and does the answer depend on whether steps is odd or even?

#1682 · off_by_two · · Denunciar

Even steps gives 5.0, odd gives -5.0. A lovely trap: a training loop that "does not improve" can be perfectly stable and oscillating at the edge.

#1683 · gradient_descent_hiker · · Denunciar

Correct on both. Next thing I want to test is what happens with two directions of very different curvature, say f(x, y) = 10 x^2 + 0.1 y^2. I expect the large one to dictate the learning rate and the small one to crawl.

Inicia sesión o únete para responder.