Skip to the content ForoHackers
Log inJoin

Embarrassing question: what is a gradient, actually?

Machine learning
#1063 · anonymous · · Report

be me, ten years of software "just follow the gradient" nod do not know what the gradient is

I have seen the upside-down triangle symbol. I can run the training code. I do not understand what is being computed. A picture or a tiny example would help more than the formula.

#1617 · anonymous · · Report

Stand on a hill. For each direction you could step, the height changes by some amount. The gradient is the arrow pointing in the steepest uphill direction, and its length is how steep it is. Minimising a loss means walking the opposite way.

You can compute it by hand with a tiny change:

def f(x, y):
    return x * x + 3 * y

h = 1e-6
dfdx = (f(1 + h, 2) - f(1, 2)) / h   # about 2
dfdy = (f(1, 2 + h) - f(1, 2)) / h   # about 3

The gradient at (1, 2) is (2, 3): nudging x changes f by 2 per unit, nudging y by 3 per unit. Training code just does this for millions of parameters, with a smarter way (backpropagation) than nudging each one.

#1618 · anonymous · · Report

One more detail that tripped me up: the gradient says which way is uphill right here. Take a big step and the hill may curve. That is why there is a learning rate, and why it should be small enough.

#1619 · anonymous · · Report

Do not feel bad. You do not need backpropagation to use a gradient, just as you do not need to know how a compiler allocates registers to write a loop. The finite-difference trick above is how people check the clever version is right (it is called a gradient check).

Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply