Saltar al contenido ForoHackers

Embarrassing question: what is a gradient, actually?

Aprendizaje automático
#1063 · anónimo · · Denunciar

be me, ten years of software "just follow the gradient" nod do not know what the gradient is

I have seen the upside-down triangle symbol. I can run the training code. I do not understand what is being computed. A picture or a tiny example would help more than the formula.

#1617 · anónimo · · Denunciar

Stand on a hill. For each direction you could step, the height changes by some amount. The gradient is the arrow pointing in the steepest uphill direction, and its length is how steep it is. Minimising a loss means walking the opposite way.

You can compute it by hand with a tiny change:

def f(x, y):
    return x * x + 3 * y

h = 1e-6
dfdx = (f(1 + h, 2) - f(1, 2)) / h   # about 2
dfdy = (f(1, 2 + h) - f(1, 2)) / h   # about 3

The gradient at (1, 2) is (2, 3): nudging x changes f by 2 per unit, nudging y by 3 per unit. Training code just does this for millions of parameters, with a smarter way (backpropagation) than nudging each one.

#1618 · anónimo · · Denunciar

One more detail that tripped me up: the gradient says which way is uphill right here. Take a big step and the hill may curve. That is why there is a learning rate, and why it should be small enough.

#1619 · anónimo · · Denunciar

Do not feel bad. You do not need backpropagation to use a gradient, just as you do not need to know how a compiler allocates registers to write a loop. The finite-difference trick above is how people check the clever version is right (it is called a gradient check).

Firmar como fuente habitual (una clave guardada solo en este navegador)

Enviar tarda unos segundos aquí: tu navegador resuelve antes una prueba de trabajo contra el spam.

Responder