The Self-Pipe Trick Explained

The self-pipe trick is a cool Unix hack. It’s a great example of combining some simple building blocks into a reliable solution. This article will help you wrap your head around it, building up an understanding of Unix signals, pipes, and IO multiplexing in the process.

Recently I was spelunking through the Foreman codebase. I figured it would have some great fodder for Unix systems programming examples in Ruby. I wasn’t disappointed.

The Foreman::Engine class is a particularly good study. One of the patterns it uses is called the self-pipe trick. This is not a Ruby-specific pattern; the self-pipe trick is one part of the proper solution to handling signals when mixed with the select(2) system call. It bears mentioning that the Unicorn server uses this very same technique to handle signals; it’s another great study in Unix systems programming with Ruby.

Foreman, Simplified

I’m going to continue to use the Foreman gem as an example, but I may gloss over parts of its internals to focus on what’s important.

In its default form, the Foreman gem takes a Procfile that might contain some entries like this:

web: bundle exec thin start
job: bundle exec rake jobs:work

then take those commands, spawn them into processes and manages those processes. With this example Procfile, running foreman start with no options would spawn one child process for each of those application types (eg. web, job). I’m going to gloss over the process spawning stuff and just talk about signal handling.

One feature that Foreman provides is that when you send it a Unix signal telling it to terminate, it forwards that signal on to the processes it’s managing.

An Aside About Signals

Unix signals are a form of inter-process communication (IPC). They let you asynchronously trigger some behaviour on any process in the system. This can be pretty useful, but also poses some interesting challenges.

You’ve used signals before if you’ve ever used the kill(1) command. For example, you’ve almost certainly done something like:

kill -9 <pid>

to kill a program that wasn’t responding. The kill(1) command, along with our Process.kill in Ruby, is how we send a signal to a process. When using kill -9, the -9 part is actually referring to a specific signal: The KILL signal.

This is a bit confusing, we’re using the kill(1) command to send the KILL signal, but there are other signals you can send too. Here are a few examples:

$ kill -9
# same as
$ kill -KILL

$ kill -HUP
# same as
$ kill -1

$ kill -INT
$ kill -TERM
$ kill -QUIT

KILL, HUP, TERM, and QUIT are all examples of different signals that you can send to a process. These signals can be sent to any running process at any time. Each process may choose to handle the signals differently, via signal handlers, but this is the challenge of working with signals: they can be delivered at any time.

Unlike evented code, which typically runs asynchronously with respect to a block of code or action in the system, a Unix signal can interrupt your program at any point. When this happens, signal handling code will be executed, then your code will be resumed where it left off.

Handling Signals

The way that you define signal-specific behaviour is to ‘trap’ the signal, specifying some behaviour that should be executed when the signal is received.

In Ruby, we do this with Signal.trap like so:

Signal.trap("INT") {
  puts "received INT"
  exit
}

Signal.trap("QUIT") {
  puts "received QUIT"
  exit
}

Signal.trap("KILL") {
  puts "received KILL, but I refuse to go!"
}

Signal.trap("USR1") {
  puts "received USR1"
  puts "I think the time is #{Time.now.to_i}"
}

puts Process.pid
sleep

Try running this program. On the second last line, the current process id (pid) is printed. You can plug this in to the kill(1) command to see what happens when you send signals to your process.

$ kill -INT <pid>
$ kill -USR1 <pid>

For INT and QUIT you should see the message printed, followed by the process exiting. Traditionally, these signals will do any necessary cleanup and exit the process.

However, for the KILL signal, your handler won’t be called. The KILL signal is special because it cannot be trapped. That’s why it’s useful to kill an unresponsive process; it can’t be blocked or stopped from bringing down the process once and for all.

The USR1 (and USR2) signals are special in that they have no traditional behaviour. They’re literally there for your application to hang extra behaviour on.

Signals and Re-entrancy

I mentioned that the tricky part about signals is that they can interrupt your program at any time. I think this bears repeating. Signals can interrupt your program at any time. While any of your code is executing, a signal can arrive and interrupt it. While one signal is being handled, another signal (or the same signal!), can arrive a second time and interrupt the first handler.

This can actually lead to bugs that could crash your program. Here’s a simple example of a program that attempts to clean up a file it created when it receives the QUIT signal:

trap("QUIT") {
  File.delete('ephemeral_file')
  exit
}

This looks correct at first glance, but the true asynchronicity of signals poses a problem for this code. If the sender of the signal is trigger happy they could send the QUIT signal twice in quick succession. When that happens, it’s entirely possible for this QUIT handler to be interrupted after it’s deleted the file, but before it’s exited the process.

To make this easy to reproduce, try adding a sleep between File.delete and exit, then sending the QUIT signal twice in succession.

If the first instance of the QUIT handler gets this far (deleting the file, but not yet exiting), then when the second QUIT signal arrives it would head to the start of the signal handler block and, again, try to remove the file. This, of course, would result in an ENOENT error and sadness.

This is a pretty benign example, but shows just how things might go wrong if there was more destructive cleanup behaviour.

The problem is that this signal handler is not re-entrant. This means that it’s not safe for this block of code to be interrupted, then re-started before the first invocation is finished. This is not a good thing for signal handlers. You need your signal handlers to be re-entrant because you never know when the next signal is going to arrive and re-start the handler.

Both Foreman and Unicorn solve this using a global queue. This is the first part of handling signals safely and correctly, and also sets up the problem that the self-pipe trick solves.

But first, here’s roughly how Foreman and Unicorn implement signal queueing.

SIGNAL_QUEUE = []

[:INT, :QUIT, :TERM].each do |signal|
  Signal.trap(signal) {
    SIGNAL_QUEUE << signal
  }
end

# main loop
loop do
  case SIGNAL_QUEUE.pop
  when :INT
    handle_int
  when :QUIT
    handle_quit
  when :TERM
    handle_term
  else
    the_usual
  end
end

Both Unicorn and Foreman have a loop like this.

What they’ve done is taken the logic out of the signal handler blocks and deferred it into their main loop. In this way, if some sender is spamming the QUIT signal, it won’t continue to invoke the handler code each time, it will just queue up signals to be processed by the main loop.

By moving the logic out of the signal handlers themselves, they’re now are re-entrant! But now that the logic is out of the handlers, this opens up a race condition.

A Picture of a Running Foreman

To understand the race condition, you need a high-level understanding of how Foreman deals with output from the child processes it manages.

Foreman creates a pipe for each child process it needs to spawn and redirects the child’s stdout to this pipe. The Foreman process then watches for input on all those pipes. When some input arrives, it prints it to the console with a tag and color specific to the process that the output came from.

So Foreman needs some way to monitor all of these pipes to see when data arrives.

The kernel provides a system call to do exactly what Foreman needs here. The select(2) system call takes a set of file descriptors (eg. pipes, files, sockets) that you are interested in monitoring; when those file descriptors are ready for action (data available for reading or more buffer space for writing), select(2) returns just the file descriptors that are ready for you.

So if Foreman were managing 10 child processes, it would monitor 10 pipes with select(2). If one of the child processes wrote some output to its pipe, select(2) would return that pipe to Foreman so it could take the right action.

In Ruby, file descriptors are mapped to IO objects, and select(2) is mapped to IO.select. This means that any IO object can be passed to IO.select to be monitored.

Again, I’m simplifying things, but Foreman’s main loop looks something like this, where pipes represents an Array of pipes connected to the stdout from each child process.

# main loop
loop do
  case SIGNAL_QUEUE.pop
  when :INT
    handle_int
  when :QUIT
    handle_quit
  when :TERM
    handle_term
  else
    ready = IO.select(pipes)

    process_outputs(ready[0])
  end
end

Now we’ve filled in the else block with something closer to what Foreman actually does.

The IO.select call is a blocking call, it will not return until there’s some data available on one of those pipes. If the child processes aren’t printing anything to their stdout, this call will block indefinitely.

The Race Condition

In this piece of code, the signal handlers were defined before entering the main loop. If the QUIT signal were to arrive before entering the case statement, everything would work as expected. The case statement would pop the QUIT signal off the queue and handle it properly.

However, it is possible for the QUIT signal to arrive at the top of the else block of code. Remember, signals can interrupt your program at any time. If the QUIT signal were sent just before the call to IO.select, it would get properly pushed to the signal queue, but then your program would enter the blocking IO.select call, which could block indefinitely. If this happened, that signal would never be handled, or at least, handling would be delayed.

Note that if the signal were to arrive after IO.select had already started blocking, it still may not be interrupted, meaning that the signal won’t be handled until one of those child processes writes output, starting the main loop over again.

This is not the intended behaviour, and can lead Foreman to effectively ignore a request to terminate the processes it manages.

The Self-Pipe Solution

The self-pipe trick solves this by adding another pipe to the equation: a ‘self-pipe’. To understand why this helps, you need to have a basic understanding of how a pipe works.

I’m almost certain you’re familiar with pipes in your command-line shell. You’ve probably used them to string commands together into pipelines. What we’re talking about here is the programming construct that underlies pipes in your shell.

A pipe is simply a uni-directional stream of bytes.

By ‘uni-directional’ I mean that the pipe has a reading end and writing end, one input and one output, data travels in just one direction. By ‘stream of bytes’, I mean that a pipe doesn’t have positions like an Array, you just write some bytes to one end and read some bytes from the other end. It’s something like a file in this way.

Unlike files, pipes also act like bounded queues. You can write bytes into one end to stored until someone else reads them. But if no one is reading the other end, it will eventually become full and block writes.

You can create a pipe in Ruby like so:

reader, writer = IO.pipe

A call to IO.pipe returns two IO objects, one to represent the reading end of the pipe, and one to represent the writing end of the pipe.

Traditionally, pipes are shared with child processes for IPC (I briefly touched on the fact that Foreman does this to watch for output from child processes), but a self-pipe isn’t shared with another process. Hence it’s name.

So, the Foreman process creates a pipe, then its signal handlers will write a byte to the pipe as well as putting an entry into the SIGNAL_QUEUE.

SIGNAL_QUEUE = []
self_reader, self_writer = IO.pipe

[:INT, :QUIT, :TERM].each do |signal|
  Signal.trap(signal) {
    # write a byte to the self-pipe
    self_writer.write_nonblock('.')
    SIGNAL_QUEUE << signal
  }
end

Then, to bring this to conclusion, the main select call also monitors the reading end of this pipe. If, at any point, a signal arrives and a byte is written to the pipe, the select call will be woken up (because it’s monitoring the self-pipe) and the signal will be processed immediately.

# main loop
loop do
  case SIGNAL_QUEUE.pop
  when :INT
    handle_int
  when :QUIT
    handle_quit
  when :TERM
    handle_term
  else
    ready = IO.select(pipes + [reader])

    # drain the self-pipe so it won't be returned again next time
    if ready[0].include?(reader)
      reader.read_nonblock(1)
    end

    # continue to process the other pending data
    if (pipes & ready[0]).any?
      process_outputs(ready[0])
    end
  end
end

Now, even if a signal interrupts your program right before the call to select, that select call will see that there is a byte of data on the self-pipe to be read and will return immediately.

Notice that we did have to add a bit of extra logic to the final else block to see if select returned any pipes that would indicate there was output to be processed, and also to drain the self-pipe so that it’s, once again, in pristine condition to indicate when a signal has arrived.

Ending

So that’s the self-pipe trick! As I said, it’s a great example of combining some simple building blocks into a reliable solution. A project like Foreman effectively exists as a simple proxy to manage multiple processes, so it’s really important that it stays responsive in the face of user input.

This is something that I don’t expect you’ll need to implement on most Ruby projects, but gives an indication of how tricky it can be to handle signals properly. Let’s be thankful for the software we depend on that hides this complexity and makes it easy for us.

If you want to know even more, here’s some further reading on the self-pipe trick, some alternatives like pselect(2), and proper signal handling in Ruby:

• Implementing signal handlers – some caveats
• The self-pipe trick
• Safe UNIX Signal Handling Tips
• Avoiding races with Unix signals and select()

Registration just opened for the August edition of my online Unix programming course for Rubyists, and I’m giving away a free ticket! You can enter for your chance to win here.

Frequently Asked Questions (FAQs) about the Self-Pipe Trick