The self-pipe trick is a cool Unix hack. It’s a great example of combining some simple building blocks into a reliable solution. This article will help you wrap your head around it, building up an understanding of Unix signals, pipes, and IO multiplexing in the process.
Recently I was spelunking through the Foreman codebase. I figured it would have some great fodder for Unix systems programming examples in Ruby. I wasn’t disappointed.
The Foreman::Engine
class is a particularly good study. One of the patterns it uses is called the self-pipe trick. This is not a Ruby-specific pattern; the self-pipe trick is one part of the proper solution to handling signals when mixed with the select(2) system call. It bears mentioning that the Unicorn server uses this very same technique to handle signals; it’s another great study in Unix systems programming with Ruby.
Key Takeaways
- The self-pipe trick is a Unix programming technique that ensures safe and reliable signal handling by using a pipe to communicate signal occurrences asynchronously within the main program flow.
- This method helps avoid race conditions and re-entrancy issues common in traditional signal handling by integrating signal processing into the main event loop, rather than relying on flags or direct signal handler interventions.
- Foreman and Unicorn use the self-pipe trick to manage signals effectively, demonstrating its practical application in real-world software projects that require robust signal handling.
- The self-pipe trick involves creating a non-blocking pipe, where the signal handler writes a byte when a signal is received, which is then read by the main program loop, ensuring immediate and safe signal handling.
- While the self-pipe trick adds complexity and uses additional system resources, its benefits in creating a more stable and responsive application outweigh these potential drawbacks.
Foreman, Simplified
I’m going to continue to use the Foreman gem as an example, but I may gloss over parts of its internals to focus on what’s important.
In its default form, the Foreman gem takes a Procfile that might contain some entries like this:
web: bundle exec thin start
job: bundle exec rake jobs:work
then take those commands, spawn them into processes and manages those processes. With this example Procfile, running foreman start
with no options would spawn one child process for each of those application types (eg. web, job). I’m going to gloss over the process spawning stuff and just talk about signal handling.
One feature that Foreman provides is that when you send it a Unix signal telling it to terminate, it forwards that signal on to the processes it’s managing.
An Aside About Signals
Unix signals are a form of inter-process communication (IPC). They let you asynchronously trigger some behaviour on any process in the system. This can be pretty useful, but also poses some interesting challenges.
You’ve used signals before if you’ve ever used the kill(1) command. For example, you’ve almost certainly done something like:
kill -9 <pid>
to kill a program that wasn’t responding. The kill(1) command, along with our Process.kill
in Ruby, is how we send a signal to a process. When using kill -9
, the -9
part is actually referring to a specific signal: The KILL
signal.
This is a bit confusing, we’re using the kill(1) command to send the KILL
signal, but there are other signals you can send too. Here are a few examples:
$ kill -9
# same as
$ kill -KILL
$ kill -HUP
# same as
$ kill -1
$ kill -INT
$ kill -TERM
$ kill -QUIT
KILL
, HUP
, TERM
, and QUIT
are all examples of different signals that you can send to a process. These signals can be sent to any running process at any time. Each process may choose to handle the signals differently, via signal handlers, but this is the challenge of working with signals: they can be delivered at any time.
Unlike evented code, which typically runs asynchronously with respect to a block of code or action in the system, a Unix signal can interrupt your program at any point. When this happens, signal handling code will be executed, then your code will be resumed where it left off.
Handling Signals
The way that you define signal-specific behaviour is to ‘trap’ the signal, specifying some behaviour that should be executed when the signal is received.
In Ruby, we do this with Signal.trap
like so:
Signal.trap("INT") {
puts "received INT"
exit
}
Signal.trap("QUIT") {
puts "received QUIT"
exit
}
Signal.trap("KILL") {
puts "received KILL, but I refuse to go!"
}
Signal.trap("USR1") {
puts "received USR1"
puts "I think the time is #{Time.now.to_i}"
}
puts Process.pid
sleep
Try running this program. On the second last line, the current process id (pid) is printed. You can plug this in to the kill(1) command to see what happens when you send signals to your process.
$ kill -INT <pid>
$ kill -USR1 <pid>
For INT
and QUIT
you should see the message printed, followed by the process exiting. Traditionally, these signals will do any necessary cleanup and exit the process.
However, for the KILL
signal, your handler won’t be called. The KILL
signal is special because it cannot be trapped. That’s why it’s useful to kill an unresponsive process; it can’t be blocked or stopped from bringing down the process once and for all.
The USR1
(and USR2
) signals are special in that they have no traditional behaviour. They’re literally there for your application to hang extra behaviour on.
Signals and Re-entrancy
I mentioned that the tricky part about signals is that they can interrupt your program at any time. I think this bears repeating. Signals can interrupt your program at any time. While any of your code is executing, a signal can arrive and interrupt it. While one signal is being handled, another signal (or the same signal!), can arrive a second time and interrupt the first handler.
This can actually lead to bugs that could crash your program. Here’s a simple example of a program that attempts to clean up a file it created when it receives the QUIT
signal:
trap("QUIT") {
File.delete('ephemeral_file')
exit
}
This looks correct at first glance, but the true asynchronicity of signals poses a problem for this code. If the sender of the signal is trigger happy they could send the QUIT
signal twice in quick succession. When that happens, it’s entirely possible for this QUIT
handler to be interrupted after it’s deleted the file, but before it’s exited the process.
To make this easy to reproduce, try adding a sleep between File.delete
and exit
, then sending the QUIT
signal twice in succession.
If the first instance of the QUIT
handler gets this far (deleting the file, but not yet exiting), then when the second QUIT signal arrives it would head to the start of the signal handler block and, again, try to remove the file. This, of course, would result in an ENOENT
error and sadness.
This is a pretty benign example, but shows just how things might go wrong if there was more destructive cleanup behaviour.
The problem is that this signal handler is not re-entrant. This means that it’s not safe for this block of code to be interrupted, then re-started before the first invocation is finished. This is not a good thing for signal handlers. You need your signal handlers to be re-entrant because you never know when the next signal is going to arrive and re-start the handler.
Both Foreman and Unicorn solve this using a global queue. This is the first part of handling signals safely and correctly, and also sets up the problem that the self-pipe trick solves.
But first, here’s roughly how Foreman and Unicorn implement signal queueing.
SIGNAL_QUEUE = []
[:INT, :QUIT, :TERM].each do |signal|
Signal.trap(signal) {
SIGNAL_QUEUE << signal
}
end
# main loop
loop do
case SIGNAL_QUEUE.pop
when :INT
handle_int
when :QUIT
handle_quit
when :TERM
handle_term
else
the_usual
end
end
Both Unicorn and Foreman have a loop like this.
What they’ve done is taken the logic out of the signal handler blocks and deferred it into their main loop. In this way, if some sender is spamming the QUIT
signal, it won’t continue to invoke the handler code each time, it will just queue up signals to be processed by the main loop.
By moving the logic out of the signal handlers themselves, they’re now are re-entrant! But now that the logic is out of the handlers, this opens up a race condition.
A Picture of a Running Foreman
To understand the race condition, you need a high-level understanding of how Foreman deals with output from the child processes it manages.
Foreman creates a pipe for each child process it needs to spawn and redirects the child’s stdout to this pipe. The Foreman process then watches for input on all those pipes. When some input arrives, it prints it to the console with a tag and color specific to the process that the output came from.
So Foreman needs some way to monitor all of these pipes to see when data arrives.
The kernel provides a system call to do exactly what Foreman needs here. The select(2) system call takes a set of file descriptors (eg. pipes, files, sockets) that you are interested in monitoring; when those file descriptors are ready for action (data available for reading or more buffer space for writing), select(2) returns just the file descriptors that are ready for you.
So if Foreman were managing 10 child processes, it would monitor 10 pipes with select(2). If one of the child processes wrote some output to its pipe, select(2) would return that pipe to Foreman so it could take the right action.
In Ruby, file descriptors are mapped to IO
objects, and select(2) is mapped to IO.select
. This means that any IO
object can be passed to IO.select
to be monitored.
Again, I’m simplifying things, but Foreman’s main loop looks something like this, where pipes
represents an Array of pipes connected to the stdout from each child process.
# main loop
loop do
case SIGNAL_QUEUE.pop
when :INT
handle_int
when :QUIT
handle_quit
when :TERM
handle_term
else
ready = IO.select(pipes)
process_outputs(ready[0])
end
end
Now we’ve filled in the else
block with something closer to what Foreman actually does.
The IO.select
call is a blocking call, it will not return until there’s some data available on one of those pipes. If the child processes aren’t printing anything to their stdout, this call will block indefinitely.
The Race Condition
In this piece of code, the signal handlers were defined before entering the main loop. If the QUIT
signal were to arrive before entering the case
statement, everything would work as expected. The case statement would pop the QUIT
signal off the queue and handle it properly.
However, it is possible for the QUIT signal to arrive at the top of the else
block of code. Remember, signals can interrupt your program at any time. If the QUIT
signal were sent just before the call to IO.select
, it would get properly pushed to the signal queue, but then your program would enter the blocking IO.select
call, which could block indefinitely. If this happened, that signal would never be handled, or at least, handling would be delayed.
Note that if the signal were to arrive after IO.select
had already started blocking, it still may not be interrupted, meaning that the signal won’t be handled until one of those child processes writes output, starting the main loop over again.
This is not the intended behaviour, and can lead Foreman to effectively ignore a request to terminate the processes it manages.
The Self-Pipe Solution
The self-pipe trick solves this by adding another pipe to the equation: a ‘self-pipe’. To understand why this helps, you need to have a basic understanding of how a pipe works.
I’m almost certain you’re familiar with pipes in your command-line shell. You’ve probably used them to string commands together into pipelines. What we’re talking about here is the programming construct that underlies pipes in your shell.
A pipe is simply a uni-directional stream of bytes.
By ‘uni-directional’ I mean that the pipe has a reading end and writing end, one input and one output, data travels in just one direction. By ‘stream of bytes’, I mean that a pipe doesn’t have positions like an Array, you just write some bytes to one end and read some bytes from the other end. It’s something like a file in this way.
Unlike files, pipes also act like bounded queues. You can write bytes into one end to stored until someone else reads them. But if no one is reading the other end, it will eventually become full and block writes.
You can create a pipe in Ruby like so:
reader, writer = IO.pipe
A call to IO.pipe
returns two IO
objects, one to represent the reading end of the pipe, and one to represent the writing end of the pipe.
Traditionally, pipes are shared with child processes for IPC (I briefly touched on the fact that Foreman does this to watch for output from child processes), but a self-pipe isn’t shared with another process. Hence it’s name.
So, the Foreman process creates a pipe, then its signal handlers will write a byte to the pipe as well as putting an entry into the SIGNAL_QUEUE
.
SIGNAL_QUEUE = []
self_reader, self_writer = IO.pipe
[:INT, :QUIT, :TERM].each do |signal|
Signal.trap(signal) {
# write a byte to the self-pipe
self_writer.write_nonblock('.')
SIGNAL_QUEUE << signal
}
end
Then, to bring this to conclusion, the main select
call also monitors the reading end of this pipe. If, at any point, a signal arrives and a byte is written to the pipe, the select
call will be woken up (because it’s monitoring the self-pipe) and the signal will be processed immediately.
# main loop
loop do
case SIGNAL_QUEUE.pop
when :INT
handle_int
when :QUIT
handle_quit
when :TERM
handle_term
else
ready = IO.select(pipes + [reader])
# drain the self-pipe so it won't be returned again next time
if ready[0].include?(reader)
reader.read_nonblock(1)
end
# continue to process the other pending data
if (pipes & ready[0]).any?
process_outputs(ready[0])
end
end
end
Now, even if a signal interrupts your program right before the call to select
, that select
call will see that there is a byte of data on the self-pipe to be read and will return immediately.
Notice that we did have to add a bit of extra logic to the final else
block to see if select
returned any pipes that would indicate there was output to be processed, and also to drain the self-pipe so that it’s, once again, in pristine condition to indicate when a signal has arrived.
Ending
So that’s the self-pipe trick! As I said, it’s a great example of combining some simple building blocks into a reliable solution. A project like Foreman effectively exists as a simple proxy to manage multiple processes, so it’s really important that it stays responsive in the face of user input.
This is something that I don’t expect you’ll need to implement on most Ruby projects, but gives an indication of how tricky it can be to handle signals properly. Let’s be thankful for the software we depend on that hides this complexity and makes it easy for us.
If you want to know even more, here’s some further reading on the self-pipe trick, some alternatives like pselect(2), and proper signal handling in Ruby:
- Implementing signal handlers – some caveats
- The self-pipe trick
- Safe UNIX Signal Handling Tips
- Avoiding races with Unix signals and select()
Registration just opened for the August edition of my online Unix programming course for Rubyists, and I’m giving away a free ticket! You can enter for your chance to win here.
Frequently Asked Questions (FAQs) about the Self-Pipe Trick
What is the Self-Pipe Trick and how does it work?
The Self-Pipe Trick is a programming technique used to handle signals in a Unix-based system. It works by creating a pipe and making the read end non-blocking. When a signal arrives, the signal handler writes a byte into the pipe. The main program loop then reads from the pipe, effectively receiving the signal in a safe, asynchronous manner. This trick allows the program to handle signals without the risk of interrupting critical sections of code.
Why is the Self-Pipe Trick important in Unix programming?
The Self-Pipe Trick is crucial in Unix programming because it provides a safe way to handle signals. Signals are asynchronous events that can interrupt a program at any time, potentially causing data corruption or other issues. By using the Self-Pipe Trick, programmers can ensure that signals are handled properly and that their programs are robust and reliable.
How does the Self-Pipe Trick compare to traditional signal handling methods?
Traditional signal handling methods often involve setting a flag in the signal handler and then checking this flag at various points in the program. However, this approach can lead to race conditions and other issues. The Self-Pipe Trick avoids these problems by ensuring that signal handling is integrated into the main program flow. This makes it a more reliable and effective method for handling signals in Unix-based systems.
Can the Self-Pipe Trick be used in multi-threaded programs?
Yes, the Self-Pipe Trick can be used in multi-threaded programs. However, it’s important to note that each thread will need its own self-pipe to handle signals. This is because signals are delivered to individual threads, not to the process as a whole. Therefore, each thread must have a mechanism for handling signals independently.
What are the potential drawbacks of using the Self-Pipe Trick?
While the Self-Pipe Trick is a powerful tool for handling signals, it does have some potential drawbacks. For one, it can be more complex to implement than traditional signal handling methods. Additionally, it requires the use of a pipe, which can consume system resources. However, these drawbacks are generally outweighed by the benefits of using the Self-Pipe Trick.
Is the Self-Pipe Trick specific to Unix-based systems?
The Self-Pipe Trick is most commonly used in Unix-based systems, as these systems have a well-defined mechanism for handling signals. However, the basic concept of the Self-Pipe Trick could potentially be adapted for use in other types of systems.
How can I implement the Self-Pipe Trick in my own programs?
Implementing the Self-Pipe Trick involves creating a pipe, making the read end non-blocking, and setting up a signal handler to write to the pipe when a signal arrives. The main program loop then reads from the pipe to handle the signal. This process can be complex, so it’s important to have a good understanding of Unix programming and signal handling before attempting to implement the Self-Pipe Trick.
Can the Self-Pipe Trick handle multiple signals at once?
Yes, the Self-Pipe Trick can handle multiple signals at once. When multiple signals arrive, the signal handler writes multiple bytes into the pipe. The main program loop then reads these bytes, handling each signal in turn. This allows the Self-Pipe Trick to handle multiple signals in a safe and efficient manner.
What happens if a signal arrives while the program is reading from the pipe?
If a signal arrives while the program is reading from the pipe, the read operation will be interrupted and the signal handler will be invoked. The signal handler will then write a byte into the pipe, and the read operation will be restarted. This ensures that all signals are handled properly, even if they arrive during a read operation.
Can the Self-Pipe Trick be used with non-blocking I/O?
Yes, the Self-Pipe Trick can be used with non-blocking I/O. In fact, the Self-Pipe Trick relies on non-blocking I/O to function properly. By making the read end of the pipe non-blocking, the program can ensure that it never blocks while waiting for a signal to arrive. This allows the program to continue processing other tasks while waiting for signals.
Jesse Storimer is a programmer and author. Employed as a Senior Developer at Shopify, Inc., he also stays up late at night to self-publish books about system programming for Ruby developers. He writes a blog at jstorimer.com and can almost always be found spending time with his wife and two daughters when afk.