Unit 7: Layers

The hard step has no downhill

Unit 4 trained by feeling which way the ground tilts. Now try that on the neuron from unit 3: nudge a weight a tiny bit, and the answer is still 1. Nudge it again — still 1. Then all at once it flips to 0.

The loss is flat everywhere and a cliff in one place. There is no slope to follow, so gradient descent has nothing to walk down.

Squash instead of snap

The sigmoid takes any number at all and squeezes it into the range 0 to 1 — smoothly, with no jump. Big positive numbers come out near 1, big negative ones near 0, and 0 comes out at exactly one half.

Python

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

for x in [-4, -2, -1, 0, 1, 2, 4]:
    print(x, round(sigmoid(x), 3))

It prints

-4 0.018
-2 0.119
-1 0.269
0 0.5
1 0.731
2 0.881
4 0.982

Now the answer means something

A 0.99 is "almost certainly yes". A 0.5 is "no idea". A 0.52 is a shrug. The old neuron could only shout 1 or 0, so it could never tell you it was unsure.

And because the curve slides instead of jumping, a small change to a weight makes a small change to the answer — which is exactly what gradient descent needs.

You have solved this before

In the Chips track, Xor was not one gate. You built it out of three:

Xor = And( Or(a, b), Nand(a, b) )

"At least one is on" and "not both are on". Neurons can do each of those three jobs, so a stack of three neurons can do Xor.

Three neurons, nine numbers

Two neurons look at the inputs. The third looks only at what those two said. Big weights like 20 make the sigmoid behave almost like a switch, so each neuron acts like the gate it is copying.

Python

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def network(a, b):
    h1 = sigmoid(20 * a + 20 * b - 10)     # like Or
    h2 = sigmoid(-20 * a - 20 * b + 30)    # like Nand
    return sigmoid(20 * h1 + 20 * h2 - 30)  # like And

print(round(network(0, 0), 2))
print(round(network(0, 1), 2))
print(round(network(1, 0), 2))
print(round(network(1, 1), 2))

It prints

0.0
1.0
1.0
0.0

Look at the middle

Printing h1 and h2 shows the two hidden neurons doing their separate jobs. On the row 1 1, h1 says yes (at least one is on) but h2 says no (they are both on) — so the final And refuses. That is the row a single neuron could never get right.

Python

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

for a in [0, 1]:
    for b in [0, 1]:
        h1 = sigmoid(20 * a + 20 * b - 10)
        h2 = sigmoid(-20 * a - 20 * b + 30)
        out = sigmoid(20 * h1 + 20 * h2 - 30)
        print(a, b, round(h1, 2), round(h2, 2), round(out, 2))

It prints

0 0 0.0 1.0 0.0
0 1 1.0 1.0 1.0
1 0 1.0 1.0 1.0
1 1 1.0 0.0 0.0

Two lines, not one

Unit 3 proved one neuron cannot separate XOR, because one neuron is one straight line. This network has two neurons in the middle, so it draws two lines — and the output neuron keeps only the strip between them.

More hidden neurons means more lines, and enough lines can fence off any shape at all.

The proper names

The numbers you feed in are the input layer. The neurons in the middle are a hidden layer — hidden because nobody outside the network ever sees what they say. The last neuron or neurons are the output layer.

A network with several hidden layers is called deep, and that is the whole of the phrase "deep learning". It means "quite a lot of layers".

Counting the numbers

Each neuron holds one weight per input, plus one bias. Our XOR network has 2 inputs, 2 hidden neurons and 1 output:

- hidden layer: 2 neurons × (2 weights + 1 bias) = 6
- output layer: 1 neuron × (2 weights + 1 bias) = 3

Nine numbers. That is the entire model.

Python

def count(layers):
    total = 0
    for i in range(1, len(layers)):
        total = total + layers[i] * (layers[i - 1] + 1)
    return total

print(count([2, 2, 1]))
print(count([2, 3, 1]))
print(count([784, 100, 10]))

It prints

9
13
79510

That last one reads handwriting

A small network for recognising handwritten digits takes 784 inputs — one for every pixel in a 28 by 28 picture — and gives 10 answers, one per digit. About eighty thousand numbers.

The language models you have heard of are the same arithmetic with hundreds of billions of numbers. Nothing new gets added. There is just a great deal more of it.

Everything so far, in one picture

A neuron multiplies each input by a weight, adds them up with a bias, and squashes the total between 0 and 1.

A layer is several neurons looking at the same inputs.

A network is layers, where each layer reads what the layer before it said.

Running the numbers forwards through it like this is called a forward pass.

Written with lists, it works for any size

The same forward pass, written so the shape lives in the data instead of in the code. Now a bigger network needs bigger lists, not new lines.

Python

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def layer(inputs, weights, biases):
    out = []
    for n in range(len(biases)):
        total = biases[n]
        for i in range(len(inputs)):
            total = total + inputs[i] * weights[n][i]
        out.append(sigmoid(total))
    return out

hidden = layer([1, 0], [[20, 20], [-20, -20]], [-10, 30])
output = layer(hidden, [[20, 20]], [-30])
print([round(h, 2) for h in hidden])
print([round(o, 2) for o in output])

It prints

[1.0, 1.0]
[1.0]

But somebody chose those nine numbers

Every weight in this unit was picked by hand, by copying a circuit you already knew. That does not scale: nobody hand-picks eighty thousand numbers, let alone a billion.

Unit 6 throws the hand-picked numbers away, starts from nearly nothing, and lets the network find its own.