Unit 4: Joining Programs Up

Every program has an out tray

A command does not really "print to the screen". It writes to something called standard output, and by default that happens to be pointed at your screen.

Point it somewhere else and the words go there instead. Nothing about the command changes. It never finds out.

Pointing it at a file

That is all > has ever been. You have been using it since unit 2.

And it works on every command, not just echo — because none of them know where their out tray leads.

Terminal

kid@alguni:~$ ls
a.txt
b.txt
c.txt
kid@alguni:~$ ls > list.txt
kid@alguni:~$ cat list.txt
a.txt
b.txt
c.txt
list.txt

The in tray

There is a tray on the way in as well: standard input. < fills it from a file.

wc -l < guests.txt and wc -l guests.txt count the same lines, but look at what they print. In the second, wc was handed a *name*, so it says the name. In the first it was handed the *words* and has no idea where they came from.

Terminal

kid@alguni:~$ wc -l guests.txt
3 guests.txt
kid@alguni:~$ wc -l < guests.txt
3

Out of one, into the next

If output can go to a file, it can go to another program. That is |, and it is the best idea in Unix.

The left command's out tray is plugged straight into the right command's in tray. No file in between, nothing saved to disk.

Terminal

kid@alguni:~$ ls
a.txt
b.txt
c.txt
d.txt
e
kid@alguni:~$ ls | wc -l
5

Why this is such a big deal

Nobody had to write a "count how many files" program. ls lists things and wc counts lines, and the pipe made a third thing out of them.

That is the whole design: lots of small programs that each do one thing, and a way to join them up. Every command in this course is one of those small programs.

Longer chains

You can keep going. Each | hands its answer on to the next one, left to right, like a relay.

Read this one out loud: list the files, keep the ones with "photo" in the name, count them.

Terminal

kid@alguni:~$ ls | grep photo
photo1.jpg
photo2.jpg
photo3.jpg
kid@alguni:~$ ls | grep photo | wc -l
3

Putting things in order

sort puts lines in order. By default it sorts them as words, which is not the same as sorting them as numbers.

As words, 100 comes before 9, because 1 comes before 9. -n says "these are numbers, treat them as numbers".

Terminal

kid@alguni:~$ sort sizes.txt
10
100
2
9
kid@alguni:~$ sort -n sizes.txt
2
9
10
100

Squashing repeats

uniq throws away a line that is the same as the one right before it. That "right before it" matters: it only ever looks at its neighbour, so you nearly always sort first.

uniq -c counts each group as it squashes it, and that is the trick behind almost every "which is most common?" question you will ever ask.

Terminal

kid@alguni:~$ sort fruit.txt
apple
apple
apple
fig
pear
pear
kid@alguni:~$ sort fruit.txt | uniq
apple
fig
pear
kid@alguni:~$ sort fruit.txt | uniq -c
      3 apple
      1 fig
      2 pear

The famous one

Put those together and you get the pipeline every Linux person knows by heart: sort, squash-and-count, sort by the count, take the top one.

Four small programs. No new program written. That is the point of the whole unit.

Terminal

kid@alguni:~$ sort fruit.txt | uniq -c | sort -n -r
      3 apple
      2 pear
      1 fig
kid@alguni:~$ sort fruit.txt | uniq -c | sort -n -r | head -n 1
      3 apple

A file that is really a table

Lots of files on a Linux machine are tables: one row per line, with the columns split by some character.

/etc/passwd lists everybody who can use this computer. The columns are split by colons, and the first one is the name.

Terminal

kid@alguni:~$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
kid:x:1000:1000:Kid:/home/kid:/bin/bash

Taking one column

cut takes columns out. -d says what splits them — the delimiter — and -f says which fields you want.

-f1 is the first. -f1,6 is the first and the sixth. -f3- is the third onwards.

Terminal

kid@alguni:~$ cut -d: -f1 /etc/passwd
root
daemon
nobody
kid
kid@alguni:~$ cut -d: -f1,6 /etc/passwd
root:/root
daemon:/usr/sbin
nobody:/nonexistent
kid:/home/kid

Cut, then everything else

cut is another small program, so of course it goes in a pipe. Take a column, sort it, count it.

At this point you are doing something a spreadsheet would take five minutes of clicking to do.

Terminal

kid@alguni:~$ cut -d: -f2 lunches.txt | sort | uniq -c
      3 apple
      1 pear

Two trays, not one

Every program has two out trays. Ordinary answers go to standard output. Complaints go to a second one called standard error.

Both land on your screen, which is why you never noticed. But > only catches the first — so an error stays on the screen even when everything else has gone into a file.

Terminal

kid@alguni:~$ cat there.txt missing.txt > out.txt
cat: missing.txt: No such file or directory
kid@alguni:~$ cat out.txt
hello

Why two?

Because otherwise every error message would end up buried in your results.

Imagine ls > list.txt on a folder with a problem in it. If the complaint went into list.txt, your list would quietly have a sentence of English in the middle. Keeping them apart is what makes pipes safe.

Catching the complaints too

2> redirects the second tray. The 2 is its number — output is 1, errors are 2.

And 2>&1 means "send tray 2 to wherever tray 1 is going", which is how you get both in one file.

Terminal

kid@alguni:~$ cat there.txt missing.txt 2> problems.txt
hello
kid@alguni:~$ cat problems.txt
cat: missing.txt: No such file or directory
kid@alguni:~$ cat there.txt missing.txt > all.txt 2>&1
kid@alguni:~$ cat all.txt
hello
cat: missing.txt: No such file or directory

The bin

There is a special file on every Linux machine called /dev/null. Anything written to it disappears, and reading it gives nothing.

So 2> /dev/null means "and be quiet about the errors". It is a real file in a real folder, and it is the tidiest way to say "I do not care".

Terminal

kid@alguni:~$ cat there.txt missing.txt 2> /dev/null
hello
kid@alguni:~$ cat /dev/null

One pipeline, five programs

You are going to answer a real question — which word does this book use most? — without writing a program.

Everything you need is already in your hands: tr, sort, uniq -c, sort -n -r, head. The only new one is tr, which swaps characters for other characters.

One word per line

tr translates characters. tr a-z A-Z shouts. tr -d deletes.

The trick for counting words is tr " " "\n": turn every space into a newline, and a paragraph becomes a list.

Terminal

kid@alguni:~$ cat line.txt | tr a-z A-Z
THE CAT AND THE HAT
kid@alguni:~$ cat line.txt | tr " " "\n"
the
cat
and
the
hat