Handle Incoming Email with SendGrid

Key Takeaways

• SendGrid, a service for sending bulk emails, also offers a feature for handling incoming emails, which can be configured to manage all incoming messages for a given domain by pinging a specified URI.
• To create an email-to-post feature using SendGrid, users need to set up a webhook that acts on incoming mail, register a callback, and write a regular expression to parse multiple recipients.
• SendGrid’s incoming email parsing feature can be used for a variety of applications, such as sending alerts when an email has been received, transferring attachments to cloud storage services, allowing people to reply by email to notifications from bulletin boards, messaging systems, etc., and handling unsubscribe requests.
• SendGrid’s Inbound Parse Webhook can handle multiple email addresses using wildcard subdomains, secure data posted with HTTPS, handle file uploads with a limit of 20MB for the total size of all attachments, and troubleshoot issues through an Activity Feed in the dashboard.

If you’ve used blogging services such as Tumblr or the now-defunct Posterous, you may be aware that many of them let you bypass their web interface and create posts via email by sending a message to a specific email address.

In this article, I’m going to look at how you might implement an email-to-post feature, using SendGrid. SendGrid is a service for sending email – usually in bulk, but they also offer a less well-publicized feature for handling incoming email. SendGrid can be configured to handle all incoming messages for a given domain by pinging a URI of your choosing, and by implementing a simple webhook, you can act on the incoming mail accordingly.

Getting Started

I’ve based the example code for this article on this Slim Framework skeleton application.

To make the app easier to debug, we can implement file-based logging by adding the following to the require section of its composer.json file:

"slim/extras": "dev-develop",

Update the instantiation of the framework in include/services.php to configure the logger as follows:

$app = new Slim(array(
    'view' => new Twig(),
    'templates.path' => $c['config']['path.templates'],
    'log.writer' => new \Slim\Extras\Log\DateTimeFileWriter(array(
        'path' => dirname($c['config']['path.logs']),
        'name_format' => 'Y-m-d',
        'message_format' => '%label% - %date% - %message%'
    ))
));

Then, copy the example configuration file to config/config.php and set your own configuration values (such as your database connection details). Additionally, add the following lines to specify the directories that will hold log files and uploaded images:

'path.logs'    => $basedir . 'logs/',
'path.uploads' => $basedir . 'public/uploads/'

Create the directories and ensure they are both writeable by the web server.

Our application will provide registered users with an email alias. By matching the part before the @ in the recipient email address, we can determine the user who’s posting. In the real world, you’ll probably want to make the alias something much more difficult to guess, and perhaps restrict access to emails sent from a specific address (though of course this is fairly simple to spoof!).

The database schema defines two tables to hold the users and posts respectively:

CREATE TABLE users (
    id INTEGER NOT NULL AUTO_INCREMENT,
    name VARCHAR(128) NOT NULL ,
    alias VARCHAR(45) NOT NULL ,
    PRIMARY KEY (id) ,
   INDEX alias (alias ASC)
);
CREATE  TABLE posts (
    id INTEGER NOT NULL AUTO_INCREMENT,
    title VARCHAR(255) NOT NULL,
    body TEXT NOT NULL,
    image varchar(255),
    user_id INTEGER NOT NULL,
    PRIMARY KEY (id)
);

You’ll also need an account with SendGrid (the free plan should be more than sufficient for this example application). Once you’ve signed up, navigate to the Developers page using the top navigation and click Parsing Incoming Emails on the right-hand side. Enter suitable values for your hostname and callback.

When an email is received for a given domain, SendGrid will send a POST request to your URL which includes information such as the sender, recipient and body, as well as any file attachments.

And finally, you’ll need to add an MX record for the domain you’re using. The way in which you do this depends on your hosting provider and can vary enormously, so consult the relevant documentation. The record needs to point to mx.sendgrid.net.

Building the Callback

The application should respond to POST requests at the URL you specified, for example:

$app->post('/endpoints/email', function () use ($app, $c) {

If SendGrid tries to “ping” your endpoint and gets a 4xx or 5xx response, it will queue and retry the request, and keep doing so for a period of 3 days. As such, it’s important that a successful ping returns a 200 status.

When SendGrid hits the endpoint, it does so with various pieces of information from the email it received. These are documented in the online API reference, but the ones we’re particularly interested in are:

In an ideal world we could just read the recipient’s email address directly from the “to” field, but the field could match any of the following non-exhaustive list of examples depending on how was sent:

• test-user@example.com
• "Test User" <test-user@example.com>
• "Test User" <test-user@example.com>, another-recipient@example.com
• "Test User" <test-user@example.com>; "Another Recipient" <another-recipient@example.com>

We need a regular expression to parse what could be multiple recipients:

$to = $req->post('to');
preg_match_all('/\b[A-Z0-9._%+-]+@(?:[A-Z0-9-]+\.)+[A-Z]+\b/i', $to, $result, PREG_PATTERN_ORDER);
foreach ($result[0] as $toAddress) {
    // handle the from address
}

For each recipient, we then extract the alias portion of the address and try and find a user that matches:

$alias = substr($toAddress, 0, strpos($toAddress, '@'));
$user = $c['db']->users->where('alias = ?', $alias)->fetch();
if ($user) {
    // great, we've got a valid user
}

For each recipient, creating a post might look like something like this:

$c['db']->posts()->insert(array(
    'title'   =>  $req->post('subject'),
    'body'    =>  ($req->post('html')) ? $req->post('html') : $req->post('text'),
    'user_id' =>  $user['id'],
));

Now we have basic email-to-post functionality! Sure, it’s not very secure, but that’s outside of the scope of this article.

Let’s expand the application a bit and allow users to add an image to a post by sending it as a mail attachment. The POST request from SendGrid includes a parameter called attachments which contains the number of attached files. Attached files are POSTed along with the request and can be handled in the same way you would handle file uploads from a web form.

if ($user) {
    $numAttachments = $req->post('attachments');
    $uploadedFiles = array();
    if ($numAttachments) {
        foreach ($_FILES as $key => $file) {
            $log->info('Saving file to ' . $c['config']['uploads.dir'] . $file['name']);
            move_uploaded_file($file['tmp_name'], $c['config']['uploads.dir'] . $file['name']);
            $uploadedFiles[] = $file['name'];
    }
}

To keep things simple, let’s just save the filename of the first attached file to the database:

$c['db']->posts()->insert(array(
    'title'   => $req->post('subject'),
    'body'    => ($req->post('html')) ? $req->post('html') : $req->post('text'),
    'user_id' => $user['id'],
    'image'   => (count($uploaded_files)) ? $uploadedFiles[0] : ''
));

Summary

In this article we’ve looked at a simple use for incoming email parsing – a basic “email to post” type function for a blogging-style application, allowing users to create posts by sending an email to a special address.

By registering a simple callback, there are all sorts of fun and interesting things you can do to respond to incoming emails. For instance, how about:

• sending alerts when an email has been received
• transferring attachments to Dropbox, Copy.com, SkyDrive, S3, etc.
• allowing people to reply by email to notifications from bulletin boards, messaging systems, etc.
• handling unsubscribe requests

If you think of other interesting uses, do let me know in the comments. Oh, and downloadable code for this article is available on GitHub for you to clone and explore.

Frequently Asked Questions (FAQs) about Handling Incoming Email with SendGrid