Day 4: Thursday, January 19, 2017 - Theory of the Pipe

This is the last focused day on command line scripting before we move into the Python programming language – which, confusingly enough, has its own command line interpreter...why is why it’s good to be familiar with the very concept of an “interactive shell”.

Homework

Like the previous assignment, accept 12 times as many exercises:

Our dev server

By the end of class today, you should get an email from me with instructions on how to log on to our “dev server”.

The address is: dev.thrill.haus

And you SSH into it like:

$ ssh your_sunet_id@dev.thrill.haus

Note

note

This is my private server built for this class with all the software that is needed to do the homework assigned today. The fact that your username on my server is the same as it is on Stanford’s is deliberate (remember how naming things is a hard computer science problem?), but the server has no connection to Stanford’s servers at all.

How Doug McIlroy, inventor of the Unix pipe, summarized the Unix philosophy:

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

The Unix Pipe

(moved from the lecture notes on 2017-01-17)

This 27-minute film from Bell Labs, titled “The UNIX System: Making Computers More Productive”, is really worth watching if you enjoy 1980s-nostalgia. But here is a key excerpt about what made Unix significantly unique back then, explained by Bell Labs programmer Brian Kernighan (starts at 4:58):

What Kernighan said back in 1982 still applies today:

What you can do is to think of these Unix system programs as...building blocks with which you can create things. And the thing that distinguishes Unix system from many other systems is the degree to which those building blocks can be glued together in a variety of different ways. Not just obvious ways but in many cases, very unobvious ways to get different jobs done. The system is very flexible in that respect.

When he talks about how Unix works – as a framework that lets programs operate in igornance of each other – that’s how we should think of our own programs:

I think the notion of pipelining is the fundamental contribution...You can take a bunch of programs and stick them together end-to-end so that the data simply flows from the one on the left to the one on the right. And the system itself looks after all of the connections, all of the synchronization, making sure that the data goes from the one [program] into the other.

The programs themselves don’t know anything about the connection as far as they’re concerned. They’re just talking to the terminal.

Why we don’t Bash all the time

We’ve been executing commands in the Unix way through Bash, the Bourne Again Shell. We could have been doing proper programming in Bash, with variables and for loops and if statements, instead of doing it in Python.

There’s a lot of reasons why we’re using Python. One reason is how Pixar literally wiped out “Toy Story 2” because of a slightly errant Bash command; or, as Mental Floss puts it: How One Line of Text Nearly Killed ‘Toy Story 2’:

Writing in his book Creativity Inc, Pixar co-founder Ed Catmull recalled that in the winter of 1998, a year out from the release of Toy Story 2, somebody (he never reveals who in the book) entered the command ‘/bin/rm -r -f *‘ on the drives where the film’s files were kept.

The object of said command is to remove everything from a given location, and to remove it quickly. It did its job.

“First, Woody’s hat disappeared. Then his boots. Then he disappeared entirely,” recalls Catmull. “Whole sequences—poof!—were deleted from the drive.”

...The plug was pulled, but not in time—90% of the film was gone, erased “in a matter of seconds.”

Here’s a fun animated recap of that disaster: How Toy Story 2 Almost Got Deleted: Stories From Pixar Animation.

Every programming language has the power to do creative, and destructive things. For some fun legacy reasons, it’s almost always easier to be accidentally destructive in Bash. If even the tech geniuses at Pixar – or Valve, or basically everyone who has touched Unix – can have a disaster, so can we as students.