Unit 5: The Neuron

Named after a brain cell, but much simpler

An artificial neuron does three things and stops:

1. multiply each input by its own weight
2. add the results together, plus one extra number called the bias
3. if the total is above zero, fire a 1 — otherwise a 0

That is all. Everything in the rest of this track is made of these.

Python

def neuron(a, b, w1, w2, bias):
    total = a * w1 + b * w2 + bias
    if total > 0:
        return 1
    return 0

print(neuron(1, 1, 0.2, 0.1, -0.2))
print(neuron(0, 1, 0.2, 0.1, -0.2))

It prints

1
0

What the three numbers mean

A weight is how much that input matters. A big weight means "listen to this one". A negative weight means "this input is evidence *against* firing".

The bias is how eager the neuron is in the first place. A bias of -0.2 means the inputs have to add up to more than 0.2 before anything happens at all.

Python

def total(a, b, w1, w2, bias):
    return a * w1 + b * w2 + bias

print(round(total(1, 1, 0.2, 0.1, -0.2), 2))
print(round(total(1, 0, 0.2, 0.1, -0.2), 2))
print(round(total(0, 0, 0.2, 0.1, -0.2), 2))

It prints

0.1
0.0
-0.2

Change the numbers, change the rule

The code never changes. Only the three numbers do — and those three numbers are the entire model. Saving an AI means saving its weights, and nothing else.

Nobody hands you the weights

So far you chose them. Now the neuron will find them, using a rule so short it fits in one line of thinking:

*If I should have fired and did not, turn up the weights of the inputs that were on. If I fired and should not have, turn them down.*

Write the mistake as error = target - output. It is 1 when the neuron was too shy, -1 when it was too eager, and 0 when it was right — so when it is right, nothing changes at all.

Nudge, do not leap

Each weight moves by error times the input, times a small number called the learning rate. Multiplying by the input matters: an input that was 0 had no say in the mistake, so its weight must not move.

A learning rate of 0.1 means "move a tenth of the way". You will meet it again in every unit from here on.

Python

w1 = 0.5
w2 = 0.5
bias = 0.0
error = -1
a = 1
b = 0

w1 = w1 + 0.1 * error * a
w2 = w2 + 0.1 * error * b
bias = bias + 0.1 * error

print(round(w1, 2), round(w2, 2), round(bias, 2))

It prints

0.4 0.5 -0.1

Teaching it "and"

Here is the whole training loop. Four examples, six passes over them, weights starting at zero. Watch the numbers wander and then stop moving — when it stops moving, it has stopped being wrong.

Python

data = [[0, 0, 0], [0, 1, 0], [1, 0, 0], [1, 1, 1]]

w1 = 0.0
w2 = 0.0
bias = 0.0

def fire(a, b):
    if a * w1 + b * w2 + bias > 0:
        return 1
    return 0

for step in range(6):
    for a, b, target in data:
        error = target - fire(a, b)
        w1 = w1 + 0.1 * error * a
        w2 = w2 + 0.1 * error * b
        bias = bias + 0.1 * error
    print(step + 1, round(w1, 2), round(w2, 2), round(bias, 2))

It prints

1 0.1 0.1 0.1
2 0.2 0.1 0.0
3 0.2 0.1 -0.1
4 0.2 0.2 -0.1
5 0.2 0.1 -0.2
6 0.2 0.1 -0.2

It works

Weights 0.2 and 0.1 with a bias of -0.2. Nobody chose those. The neuron found them by being wrong a few times, and they are a perfect and.

Python

w1 = 0.2
w2 = 0.1
bias = -0.2

for a in [0, 1]:
    for b in [0, 1]:
        total = a * w1 + b * w2 + bias
        print(a, b, 1 if total > 0 else 0)

It prints

0 0 0
0 1 0
1 0 0
1 1 1

Where does it change its mind?

The neuron fires when a*w1 + b*w2 + bias is above zero, and does not when it is below. So the interesting place is where the total is exactly zero — and that place is a straight line across the grid.

On one side of the line the answer is 1. On the other it is 0. A neuron is a line.

The line it found for "and"

With weights 0.2 and 0.1 and bias -0.2, the total is zero along a line that passes just under the corner at 1, 1 — leaving that one corner on the firing side and the other three below.

Python

w1 = 0.2
w2 = 0.1
bias = -0.2

# The boundary is where a*w1 + b*w2 + bias == 0,
# so b = (0 - a*w1 - bias) / w2
for a in [0, 0.5, 1]:
    b = (0 - a * w1 - bias) / w2
    print(a, round(b, 2))

It prints

0 2.0
0.5 1.0
1 0.0

Back to the fruit

Fruit is a grid too, so the same neuron works on it — with the features scaled to 0 and 1 first, exactly as in unit 2. The labels become 0 for apple and 1 for orange, because a neuron only speaks in numbers.

Python

fruits = [
    [150, 2, 0],
    [170, 3, 0],
    [140, 1, 0],
    [180, 7, 1],
    [165, 8, 1],
    [190, 9, 1],
]

def scale(f):
    return [(f[0] - 140) / 50, (f[1] - 1) / 8]

w1 = 0.0
w2 = 0.0
bias = 0.0

def fire(x):
    if x[0] * w1 + x[1] * w2 + bias > 0:
        return 1
    return 0

for step in range(20):
    for f in fruits:
        x = scale(f)
        error = f[2] - fire(x)
        w1 = w1 + 0.1 * error * x[0]
        w2 = w2 + 0.1 * error * x[1]
        bias = bias + 0.1 * error

print(round(w1, 2), round(w2, 2), round(bias, 2))
for f in fruits:
    print(f[2], fire(scale(f)))

It prints

0.06 0.1 -0.1
0 0
0 0
0 0
1 1
1 1
1 1

Notice which weight won

Bumpiness ended up with the bigger weight, 0.1 against 0.06. Nobody told the neuron that bumpiness was the useful feature — it discovered that by being corrected. That is the first time in this track a program has found something out on its own.

One or the other, but not both

You have met this one before. In the Chips track it was Xor: true when the two inputs are different.

Let us ask the neuron to learn it. Same rule, same code, only the target column changes.

Python

data = [[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 0]]

for a, b, target in data:
    print(a, b, target)

It prints

0 0 0
0 1 1
1 0 1
1 1 0

Twenty passes, and it never improves

Two out of four, pass after pass. It is not close, it is not slowly getting there, and waiting longer will not help — a thousand passes give exactly the same two.

Python

data = [[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 0]]

w1 = 0.0
w2 = 0.0
bias = 0.0

def fire(a, b):
    if a * w1 + b * w2 + bias > 0:
        return 1
    return 0

for step in range(20):
    for a, b, target in data:
        error = target - fire(a, b)
        w1 = w1 + 0.1 * error * a
        w2 = w2 + 0.1 * error * b
        bias = bias + 0.1 * error
    if step % 5 == 4:
        right = 0
        for a, b, target in data:
            if fire(a, b) == target:
                right = right + 1
        print("pass", step + 1, "-", right, "out of 4")

It prints

pass 5 - 2 out of 4
pass 10 - 2 out of 4
pass 15 - 2 out of 4
pass 20 - 2 out of 4

It is not the code. It is impossible.

Draw the four corners on paper. The two that should fire are top-left and bottom-right. The two that should not are bottom-left and top-right.

Now try to draw one straight line with the firing pair on one side and the other pair on the other. You cannot. They are diagonally opposite, so any line that catches both catches one of the others too.

A single neuron is a single line, so a single neuron can never do this. No learning rate, no amount of patience.

You already know the way out

In the Chips track, Xor was not one gate either — you built it out of several. The answer here is the same: use more than one neuron, and feed the output of the first ones into another.

That stack is called a network, and the neurons in the middle are a hidden layer. Unit 5 builds one, and it solves this exact problem.