Unit 1: Rules or Examples

Every program so far did exactly what you said

You have written rules: if, else, while. The computer never decided anything — you decided, and it obeyed. Here is a rule for spotting a cat.

Python

def is_cat(legs, sound):
    if legs == 4 and sound == "meow":
        return True
    return False

print(is_cat(4, "meow"))
print(is_cat(2, "tweet"))

It prints

True
False

Now meet a cat that purrs

The rule was fine until the world got bigger. You can patch it — or sound == "purr" — but that is another line *you* had to think of. Real cats will always find a way through your list.

Python

def is_cat(legs, sound):
    if legs == 4 and sound == "meow":
        return True
    return False

print(is_cat(4, "purr"))

It prints

False

Machine learning turns it around

Instead of writing the rule, you collect examples that somebody has already labelled — a thousand photos with "cat" or "not cat" written on the back. Then a program searches for the rule that gets the most of them right.

You bring the examples. The computer brings the rule.

A computer cannot see a fruit

It can only add up numbers. So the first job in every AI is turning a real thing into a list of numbers. Each number is a feature.

Here each fruit is two features: its weight in grams, and how bumpy its skin is from 0 to 10.

Python

fruits = [
    [150, 2, "apple"],
    [170, 3, "apple"],
    [180, 7, "orange"],
    [165, 8, "orange"],
]

for weight, bumpiness, name in fruits:
    print(name, weight, bumpiness)

It prints

apple 150 2
apple 170 3
orange 180 7
orange 165 8

The word on the end is the answer

That last word is the label — what a human already knew this fruit was. Features go in, and the label is what we want the computer to say back.

A pile of examples with labels on them is called the training data.

Python

fruits = [
    [150, 2, "apple"],
    [170, 3, "apple"],
    [180, 7, "orange"],
    [165, 8, "orange"],
]

features = [f[0:2] for f in fruits]
labels = [f[2] for f in fruits]

print(features[0], labels[0])
print(features[3], labels[3])

It prints

[150, 2] apple
[165, 8] orange

Some features are more useful than others

Look down the two columns. Bumpiness splits the fruit cleanly — apples 2 and 3, oranges 7 and 8. Weight does not: a heavy apple weighs more than a light orange.

Picking features that actually separate the answers matters more than any clever maths later on.

Score the guesses

Before a computer can search for a good rule, it needs to know what "good" means. The simplest score is accuracy: how many guesses were right, out of how many there were.

Python

guesses = ["apple", "apple", "orange", "apple"]
answers = ["apple", "orange", "orange", "apple"]

right = 0
for i in range(len(answers)):
    if guesses[i] == answers[i]:
        right = right + 1

print(right, "out of", len(answers))
print(right / len(answers))

It prints

3 out of 4
0.75

A high score can still be useless

Imagine 100 emails and only 3 are junk. A program that says "not junk" every single time scores 97%.

It has learned nothing. It never once found what you asked it to find. Always ask what the laziest possible answer would score, and beat *that*.

Python

answers = ["ok"] * 97 + ["junk"] * 3
lazy = ["ok"] * 100

right = 0
for i in range(100):
    if lazy[i] == answers[i]:
        right = right + 1

print(right / 100)

It prints

0.97

A rule with a number in it

Here is a rule shaped like a question: *if bumpiness is at least T, say orange*. Change T and you change the rule. Nobody has to rewrite the code — only that one number.

Python

fruits = [
    [150, 2, "apple"],
    [170, 3, "apple"],
    [180, 7, "orange"],
    [165, 8, "orange"],
]

def guess(bumpiness, t):
    if bumpiness >= t:
        return "orange"
    return "apple"

print(guess(3, 5), guess(8, 5))
print(guess(3, 2), guess(8, 2))

It prints

apple orange
orange orange

So let the computer try them all

The computer cannot think up a rule. But it can try every T from 0 to 10, score each one, and keep the best. That loop is a learning algorithm — and the number it keeps is the model.

Python

fruits = [
    [150, 2, "apple"],
    [170, 3, "apple"],
    [180, 7, "orange"],
    [165, 8, "orange"],
]

def score(t):
    right = 0
    for weight, bumpiness, label in fruits:
        guess = "orange" if bumpiness >= t else "apple"
        if guess == label:
            right = right + 1
    return right / len(fruits)

for t in range(0, 11, 2):
    print(t, score(t))

It prints

0 0.5
2 0.5
4 1.0
6 1.0
8 0.75
10 0.5

Keep the best one you have seen

Scan the scores, remember the highest, and you have learned a rule from data. Several values of T tie at 1.0 here, and this code keeps the first — that is a choice you are making, not a fact about the fruit.

Python

fruits = [
    [150, 2, "apple"],
    [170, 3, "apple"],
    [180, 7, "orange"],
    [165, 8, "orange"],
]

def score(t):
    right = 0
    for weight, bumpiness, label in fruits:
        guess = "orange" if bumpiness >= t else "apple"
        if guess == label:
            right = right + 1
    return right / len(fruits)

best_t = 0
best_score = 0
for t in range(11):
    s = score(t)
    if s > best_score:
        best_score = s
        best_t = t

print("learned T =", best_t)
print("accuracy", best_score)

It prints

learned T = 4
accuracy 1.0