Unit 9: Seeing

There are no pictures inside a computer

There is a grid of numbers, one per pixel, saying how bright that dot is. A black and white picture 5 dots across and 5 down is 25 numbers, and that is the whole picture.

Python

picture = [
    [0, 0, 1, 0, 0],
    [0, 1, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 1, 1, 1, 0],
]

print(len(picture), "rows of", len(picture[0]))
print(picture[1])

It prints

5 rows of 5
[0, 1, 1, 0, 0]

Draw it and it stops being a list

Print a # for 1 and a . for 0 and the digit appears. Nothing changed — the numbers were always that shape. Seeing it is a thing your eyes do, not a thing the data does.

Python

picture = [
    [0, 0, 1, 0, 0],
    [0, 1, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 1, 1, 1, 0],
]

for row in picture:
    line = ""
    for pixel in row:
        line = line + ("#" if pixel == 1 else ".")
    print(line)

It prints

..#..
.##..
..#..
..#..
.###.

Real pictures are the same, only bigger

A photo on a phone is a few million pixels, each with three numbers for how much red, green and blue it holds. The famous handwriting dataset is 28 by 28 — 784 numbers per digit, which is exactly the input layer you counted in unit 5.

Features again. Just a great many of them.

Python

print(28 * 28)
print(1920 * 1080 * 3)

It prints

784
6220800

A tiny grid of weights

A filter is a little grid — often 3 by 3 — of weights. Lay it over a 3 by 3 patch of the picture, multiply each pixel by the weight on top of it, and add them all up.

That is the weighted sum from unit 3, with the inputs arranged in a square instead of a row.

Python

patch = [[0, 0, 1], [0, 1, 1], [0, 0, 1]]
edge = [[1, 0, -1], [1, 0, -1], [1, 0, -1]]

total = 0
for r in range(3):
    for c in range(3):
        total = total + patch[r][c] * edge[r][c]

print(total)

It prints

-3

Now do that everywhere

Slide the filter one step at a time across the picture and down it, and write each answer into a new grid. A 5 by 5 picture gives a 3 by 3 answer, because the filter cannot hang off the edge.

Sliding a filter over a picture like this is called a convolution, and it is the C in "convolutional network".

Python

bar = [
    [0, 0, 1, 1, 0],
    [0, 0, 1, 1, 0],
    [0, 0, 1, 1, 0],
    [0, 0, 1, 1, 0],
    [0, 0, 1, 1, 0],
]
edge = [[1, 0, -1], [1, 0, -1], [1, 0, -1]]

def apply_filter(image, f):
    out = []
    for r in range(len(image) - 2):
        row = []
        for c in range(len(image[0]) - 2):
            total = 0
            for i in range(3):
                for j in range(3):
                    total = total + image[r + i][c + j] * f[i][j]
            row.append(total)
        out.append(row)
    return out

for row in apply_filter(bar, edge):
    print(row)

It prints

[-3, -3, 3]
[-3, -3, 3]
[-3, -3, 3]

It has found the edges of the bar

A big negative number where the picture goes from dark to light, a big positive one where it goes back from light to dark, and 0 in the flat parts where left and right look the same.

The filter answers one question — *is there a vertical edge here?* — at every position at once.

A different filter, a different question

Turn the weights on their side and it looks for horizontal edges instead. On the upright bar it answers 0 everywhere: there is nothing horizontal to find.

That is what people mean by a filter "detecting" something. It is nine numbers that happen to add up big for one shape and to nothing for the rest.

Python

bar = [
    [0, 0, 1, 1, 0],
    [0, 0, 1, 1, 0],
    [0, 0, 1, 1, 0],
    [0, 0, 1, 1, 0],
    [0, 0, 1, 1, 0],
]
flat = [[1, 1, 1], [0, 0, 0], [-1, -1, -1]]

def apply_filter(image, f):
    out = []
    for r in range(len(image) - 2):
        row = []
        for c in range(len(image[0]) - 2):
            total = 0
            for i in range(3):
                for j in range(3):
                    total = total + image[r + i][c + j] * f[i][j]
            row.append(total)
        out.append(row)
    return out

for row in apply_filter(bar, flat):
    print(row)

It prints

[0, 0, 0]
[0, 0, 0]
[0, 0, 0]

Why not just use the network from unit 5?

You could. Feed all 784 pixels into a hidden layer of 100 neurons and it is the same maths — but count the weights first.

Python

print("one weight per pixel:", 28 * 28 * 100 + 100)
print("one filter:          ", 3 * 3)

It prints

one weight per pixel: 78500
one filter:           9

And the count is not even the worst of it

A network with one weight per pixel learns about *positions*. Teach it a cat in the middle of the picture and it has learned nothing whatever about a cat in the corner — those are different pixels, so they are different weights.

A filter is nine weights that get slid over every position, so whatever it learns, it has learned everywhere at once. That is the real reason pictures are done this way.

Shrink the answer: pooling

Take each 2 by 2 block and keep only the biggest number in it. The grid halves, the strongest findings survive, and a shape that moves by one pixel usually gives the same answer afterwards.

This is max pooling, and it is as simple as it sounds.

Python

grid = [
    [0, 1, 3, 0],
    [2, 0, 0, 1],
    [0, 0, 4, 0],
    [1, 5, 0, 0],
]

for r in range(0, 4, 2):
    row = []
    for c in range(0, 4, 2):
        best = grid[r][c]
        for i in range(2):
            for j in range(2):
                if grid[r + i][c + j] > best:
                    best = grid[r + i][c + j]
        row.append(best)
    print(row)

It prints

[2, 3]
[5, 4]

That is a convolutional network

Filter, pool, filter again on the result, pool again — early filters find edges, later ones find corners and curves made of those edges — and then hand what is left to an ordinary layer of neurons like unit 5's.

And the nine numbers in every filter are not written by a person. They start as junk and are trained by backpropagation, exactly as in unit 6. Nobody tells it to look for edges; looking for edges is what turns out to reduce the loss.

Two digits it has seen before

A 1 and a 7, drawn on a 5 by 5 grid. These are the training data — two examples, with labels.

Python

one = [
    [0, 0, 1, 0, 0],
    [0, 1, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 1, 1, 1, 0],
]
seven = [
    [1, 1, 1, 1, 1],
    [0, 0, 0, 1, 0],
    [0, 0, 1, 0, 0],
    [0, 1, 0, 0, 0],
    [0, 1, 0, 0, 0],
]

for picture in [one, seven]:
    for row in picture:
        print("".join("#" if p == 1 else "." for p in row))
    print("-----")

It prints

..#..
.##..
..#..
..#..
.###.
-----
#####
...#.
..#..
.#...
.#...
-----

Now somebody writes a wobbly one

No little flick at the top this time. It is not identical to either example, which is the whole difficulty of reading handwriting.

Python

mystery = [
    [0, 0, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 1, 1, 1, 0],
]

for row in mystery:
    print("".join("#" if p == 1 else "." for p in row))

It prints

..#..
..#..
..#..
..#..
.###.

Unit 2 already solved this

Count how many pixels differ from each example and pick the nearer one. That is nearest neighbour, with 25 features instead of 2 — the method has not changed at all, only the number of columns.

Python

one = [
    [0, 0, 1, 0, 0],
    [0, 1, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 1, 1, 1, 0],
]
seven = [
    [1, 1, 1, 1, 1],
    [0, 0, 0, 1, 0],
    [0, 0, 1, 0, 0],
    [0, 1, 0, 0, 0],
    [0, 1, 0, 0, 0],
]
mystery = [
    [0, 0, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 0, 1, 0, 0],
    [0, 1, 1, 1, 0],
]

def different(a, b):
    count = 0
    for r in range(5):
        for c in range(5):
            if a[r][c] != b[r][c]:
                count = count + 1
    return count

print("pixels unlike the 1:", different(mystery, one))
print("pixels unlike the 7:", different(mystery, seven))
print("guess:", "1" if different(mystery, one) < different(mystery, seven) else "7")

It prints

pixels unlike the 1: 1
pixels unlike the 7: 10
guess: 1

Where this falls down

Slide that 1 one column to the right and it stops matching almost anywhere, because every pixel it touches has changed. A filter would not care, which is exactly the problem filters were invented to fix.

A real handwriting reader is filters and pooling and a couple of ordinary layers, trained on tens of thousands of digits by the method in unit 6. Every piece of it is now something you have written.