New here. I like small experiments that show why a knob matters. Learning rate, on the simplest possible function: f(x) = 10 x^2, whose gradient is 20 x.
def run(lr, steps=50):
x = 5.0
for _ in range(steps):
x -= lr * 20 * x
return x
for lr in (0.001, 0.01, 0.05, 0.1, 0.11):
print(lr, run(lr))
Each step multiplies x by (1 - 20 * lr). That one factor predicts everything:
lr = 0.001: factor 0.98, it creeps toward 0 and is still far after 50 steps.lr = 0.01: factor 0.8, converges nicely.lr = 0.05: factor 0, lands exactly on 0 in the first step.lr = 0.1: factor -1, bounces between 5 and -5 forever.lr = 0.11: factor -1.2, each step overshoots more than the last. Divergence.
The stable range is 0 < lr < 2 / curvature, here 2 / 20 = 0.1. Real models have thousands of directions with different curvatures, and the sharpest one sets the limit for all the others. That is the intuition behind why training is touchy about learning rate, and why decaying it helps near the end.
Try it before you read the output, and write down your prediction. Measuring your own surprise is the useful part.