Unit 6: Downhill

A different kind of question

Until now the answer was a word: apple or orange. Now the answer is a number.

Bags of sweets cost the same each, plus a fee for the box. Four shopping trips are written down. What would 5 bags cost?

Python

data = [[1, 5], [2, 8], [3, 11], [4, 14]]

for bags, cost in data:
    print(bags, "bags cost", cost)

It prints

1 bags cost 5
2 bags cost 8
3 bags cost 11
4 bags cost 14

The model is a straight line

Guess a price per bag w and a box fee b, and the prediction is w * bags + b. Two numbers again — the same kind of model as the neuron, without the firing.

Here is a bad guess: 1 each and no fee.

Python

data = [[1, 5], [2, 8], [3, 11], [4, 14]]
w = 1
b = 0

for bags, cost in data:
    guess = w * bags + b
    print(guess, "should be", cost, "- out by", cost - guess)

It prints

1 should be 5 - out by 4
2 should be 8 - out by 6
3 should be 11 - out by 8
4 should be 14 - out by 10

Squeeze all those mistakes into one number

Square each mistake and take the average. That single number is the loss, and the whole of training is one idea: make the loss smaller.

Squaring does two jobs. It throws away minus signs, so being 4 under is as bad as 4 over. And it punishes big mistakes far more than small ones — being out by 10 is not twice as bad as being out by 5, it is four times as bad.

Python

data = [[1, 5], [2, 8], [3, 11], [4, 14]]

def loss(w, b):
    total = 0
    for bags, cost in data:
        guess = w * bags + b
        total = total + (guess - cost) ** 2
    return total / len(data)

print(loss(1, 0))
print(loss(2, 2))
print(loss(3, 2))

It prints

54.0
7.5
0.0

Zero means perfect

A loss of 0.0 means every prediction was exactly right — 3 per bag and a 2 box fee. Real data is never this tidy, and a real loss stops somewhere above zero. That is fine. Smaller is the goal, not zero.

The loss is a valley

Keep the fee at 2 and try prices from 0 to 6. The loss falls, touches the bottom, and climbs again. Training is standing somewhere on this valley in the dark and trying to walk down.

Python

data = [[1, 5], [2, 8], [3, 11], [4, 14]]

def loss(w, b):
    total = 0
    for bags, cost in data:
        total = total + (w * bags + b - cost) ** 2
    return total / len(data)

for w in range(7):
    print(w, loss(w, 2))

It prints

0 67.5
1 30.0
2 7.5
3 0.0
4 7.5
5 30.0
6 67.5

Trying every value will not scale

That table worked because there was one number to try. A real model has millions of them, and trying every combination of a million numbers would take longer than the universe has lasted.

So instead of looking everywhere, feel which way the ground tilts under your feet and take one step that way.

Find the tilt by nudging

Move w by a tiny amount h, and see how much the loss changed. Divide the change by h and you have the slope: how much the loss goes up for every 1 you add to w.

Python

data = [[1, 5], [2, 8], [3, 11], [4, 14]]

def loss(w, b):
    total = 0
    for bags, cost in data:
        total = total + (w * bags + b - cost) ** 2
    return total / len(data)

h = 0.001
for w in range(6):
    slope = (loss(w + h, 2) - loss(w, 2)) / h
    print(w, round(slope, 2))

It prints

0 -44.99
1 -29.99
2 -14.99
3 0.01
4 15.01
5 30.01

Read the signs

A negative slope means the loss goes down as w goes up, so w should get bigger. A positive slope means the opposite. Near the bottom the slope is almost nothing, and that is how a program knows it has arrived.

The answer at the very bottom is 0.01 rather than 0.00 because nudging measures the tilt over a tiny step instead of at the exact point. It is an excellent approximation, not a perfect one.

One line of training

w = w - rate * slope

That is gradient descent, and it trains almost every AI in the world. The minus sign is the whole idea: the slope points uphill, and you want to go the other way.

The rate decides how big a step to take. Every number in the model gets its own slope and its own step, all at the same time.

Both numbers, a thousand steps

Start with a price of 0 and a fee of 0 — a model that thinks sweets are free. Each step measures the tilt in both directions and takes a small step downhill in each.

Python

data = [[1, 5], [2, 8], [3, 11], [4, 14]]

def loss(w, b):
    total = 0
    for bags, cost in data:
        total = total + (w * bags + b - cost) ** 2
    return total / len(data)

h = 0.001
rate = 0.05
w = 0.0
b = 0.0

for step in range(1000):
    slope_w = (loss(w + h, b) - loss(w, b)) / h
    slope_b = (loss(w, b + h) - loss(w, b)) / h
    w = w - rate * slope_w
    b = b - rate * slope_b
    if step % 200 == 0:
        print(step, round(w, 2), round(b, 2), round(loss(w, b), 4))

It prints

0 2.75 0.95 2.8873
200 3.01 1.96 0.0002
400 3.0 2.0 0.0
600 3.0 2.0 0.0
800 3.0 2.0 0.0

It found the price

3 per bag and a fee of 2 — which is exactly the pattern in the table, discovered by nothing but rolling downhill.

And now it can answer a question it was never shown.

Python

data = [[1, 5], [2, 8], [3, 11], [4, 14]]

def loss(w, b):
    total = 0
    for bags, cost in data:
        total = total + (w * bags + b - cost) ** 2
    return total / len(data)

h = 0.001
w = 0.0
b = 0.0
for step in range(1000):
    slope_w = (loss(w + h, b) - loss(w, b)) / h
    slope_b = (loss(w, b + h) - loss(w, b)) / h
    w = w - 0.05 * slope_w
    b = b - 0.05 * slope_b

print("5 bags:", round(w * 5 + b, 2))

It prints

5 bags: 16.99

16.99, not 17

Look closely — it is a penny out. Gradient descent walks towards the answer and stops when the steps stop mattering, so it lands *near* the bottom rather than exactly on it.

That is true of every trained model, including the enormous ones. They are all a very good approximation that stopped somewhere sensible.

Too small, and you crawl

The same code with a rate of 0.001, given a hundred steps. It is heading the right way — 2.68 is on its way to 3 — but at this speed it will be there next week.

Python

data = [[1, 5], [2, 8], [3, 11], [4, 14]]

def loss(w, b):
    total = 0
    for bags, cost in data:
        total = total + (w * bags + b - cost) ** 2
    return total / len(data)

h = 0.001
w = 0.0
b = 0.0
for step in range(100):
    slope_w = (loss(w + h, b) - loss(w, b)) / h
    slope_b = (loss(w, b + h) - loss(w, b)) / h
    w = w - 0.001 * slope_w
    b = b - 0.001 * slope_b

print(round(w, 2), round(b, 2), round(loss(w, b), 2))

It prints

2.68 0.94 3.62

Too big, and you fly off the mountain

A rate of 0.2 overshoots the bottom, lands higher up the far side, measures a steeper slope, and leaps further still. Ten steps in, the loss is over two billion.

Nothing is broken. The steps are simply longer than the valley is wide.

Python

data = [[1, 5], [2, 8], [3, 11], [4, 14]]

def loss(w, b):
    total = 0
    for bags, cost in data:
        total = total + (w * bags + b - cost) ** 2
    return total / len(data)

h = 0.001
w = 0.0
b = 0.0
for step in range(5):
    slope_w = (loss(w + h, b) - loss(w, b)) / h
    slope_b = (loss(w, b + h) - loss(w, b)) / h
    w = w - 0.2 * slope_w
    b = b - 0.2 * slope_b
    print(step + 1, round(loss(w, b), 2))

It prints

1 555.04
2 3039.43
3 16642.88
4 91141.45
5 499099.71

The learning rate is the number people fiddle with most

It is not learned. Somebody picks it, watches the loss, and picks again. A number chosen by hand rather than learned from data is called a hyperparameter, and every real AI has a pile of them.

Watching the loss go up is the sign to make it smaller. A loss that barely moves is the sign to make it bigger.