Gradient descent is an iterated function. Not a metaphor: the sequence of positions is literally iterate applied to one update step.
Minimise f(x) = (x - 3)^2. Its derivative is 2 * (x - 3):
step :: Double -> Double -> Double
step lr x = x - lr * 2 * (x - 3)
positions :: Double -> [Double]
positions lr = iterate (step lr) 0
main :: IO ()
main = print (take 6 (positions 0.25))
Each step shrinks the distance to the minimum by a constant factor: x' - 3 = (1 - 2*lr) * (x - 3). So the iteration converges exactly when |1 - 2*lr| < 1, that is 0 < lr < 1. With lr = 0.5 you land on 3 in one step. With lr = 1 you bounce between 0 and 6 forever. Above 1 you diverge.
That factor is a miniature of the whole theory: the safe learning rate depends on the curvature (here the curvature is 2, so the limit is 2/2).
Laziness bonus: iterate gives you the infinite sequence of positions and you decide when to stop, with takeWhile on the size of the step, say.