Web
Article

How to Connect Your Api.ai Assistant to the IoT

By Patrick Catanzariti

Building an AI assistant with Api.ai

The potential of a personal assistant gets exciting when it has access to personal data and the real world via the Internet of Things. New possibilities arise — from requesting your assistant turn on your lights to asking it how well you slept. We will be connecting our Api.ai assistant to the Jawbone Up API as an example of this.

What You’ll Need

This article builds upon a variety of concepts we’ve already covered in previous articles here at SitePoint. In order to follow along with this tutorial comfortably, you’ll need the following.

  • An Api.ai agent connected to a simple HTML web app – See this article if you’d like to understand this process, otherwise, you can download the code from this guide and use it too.
  • An agent that has been taught the entity of “sleep” – We created this in Empowering Your Api.ai Assistant with Entities. It should understand concepts like “how much sleep did I have last night?” and “how much REM sleep did I get?”. If you’re looking to adapt this to your own IoT device, you’ll need to have created your own custom entity that understands your IoT functionality.
  • A general knowledge of Node.js and running a Node server – Without it, you won’t be able to get the server running!
  • Knowledge of how to use the Jawbone UP API (or another API you intend to use) – We’ve covered the Jawbone Up API previously in Connecting to the Jawbone Up API with Node.js and will be referring to sections from that article throughout.
  • An SSL certificate to run your site on HTTPS – You’ll need this if working with the Jawbone Up API. We cover how to set up a self signed certificate on the Jawbone Up API article.

The Code

All code for this demo is available for you to download and use however you please! You can find it all on GitHub.

How This Works

Our Api.ai assistant is already connected to a simple web app that accepts statements via the HTML5 Speech Recognition API. From here, we need to add a new bit of functionality that listens for a specific action from our Api.ai agent. In our case, this is the action of “sleepHours”.

Whenever our JavaScript detects this action, it triggers a separate call to our Node.js app to ask the Jawbone API for that data. Once the web app receives this data, our web app turns it into a nice sentence and reads it out — giving our assistant a whole new range of intelligence!

Our Project Structure

I’ve adjusted the app from the initial HTML-only structure to one which uses EJS views so that we can switch pages in our web app when logging into the Jawbone Up API via OAuth. In reality, we only really have one page but this method allows us to add more in future if needed for other IoT devices. This single view is at /views/index.ejs. We then have our Node server in the root folder as server.js and certificate files in root too. To keep things relatively simple and contained, all front-end JavaScript and CSS is inline. Feel free to move these into CSS and JS files as you prefer, minify them and make them pretty.

Our file structure

Responding to Api.ai Actions in JavaScript

As you might remember from our previous article, when Api.ai returns a response, it provides a JSON object that looks like so:

{
  "id": "6b42eb42-0ad2-4bab-b7ea-853773b90219",
  "timestamp": "2016-02-12T01:25:09.173Z",
  "result": {
    "source": "agent",
    "resolvedQuery": "how did I sleep last night",
    "speech": "I'll retrieve your sleep stats for you now, one moment!",
    "action": "sleepHours",
    "parameters": {
      "sleep": "sleep"
    },
    "metadata": {
      "intentId": "25d04dfc-c90c-4f55-a7bd-6681e83b45ec",
      "inputContexts": [],
      "outputContexts": [],
      "contexts": [],
      "intentName": "How many hours of @sleep:sleep did I get last night?"
    }
  },
  "status": {
    "code": 200,
    "errorType": "success"
  }
}

Within that JSON object there are two bits of data we need to use — action and parameters.sleep:

"action": "sleepHours",
"parameters": {
  "sleep": "sleep"
},

action is the name we gave to the Api.ai action which the user has triggered. In the case of our sleep example, we named it “sleepHours”. parameters contain the variables in our sentence that can change a few details. In the case of sleep, our parameter tells us what type of sleep – “sleep”, “deep sleep”, “light sleep” or “REM sleep” (or just “REM”).

Initially, in our early article on Api.ai, our prepareResponse() function took the JSON response from Api.ai, put the whole thing into our debug text field on the bottom right and took out Api.ai’s verbal response to display in the web app. We completely relied on what our Api.ai agent said, without adding any of our own functionality:

function prepareResponse(val) {
  var debugJSON = JSON.stringify(val, undefined, 2),
      spokenResponse = val.result.speech;

  respond(spokenResponse);
  debugRespond(debugJSON);
}

This time around, we keep an eye out for the action field and run our own function called requestSleepData() if it the action contains "sleepHours". Within this function, we pass in the sleep parameter so we know what type of sleep is being requested:

function prepareResponse(val) {
  var debugJSON = JSON.stringify(val, undefined, 2),
    spokenResponse = val.result.speech;

  if (val.result.action == "sleepHours") {
    requestSleepData(val.result.parameters.sleep);
  } else {
    respond(spokenResponse);
  }

  debugRespond(debugJSON);
}

Within requestSleepData(), we request all sleep data from our Node.js server and then filter it by looking at the very first value in the returned array of data (data.items[0].details) — this would be last night’s sleep. Within these details, we have data.items[0].details.rem with our REM sleep, data.items[0].details.sound with our deep sleep, data.items[0].details.light with our light sleep and data.items[0].details.duration with the combined amount of sleep recorded:

function requestSleepData(type) {
  $.ajax({
    type: "GET",
    url: "/sleep_data/",
    contentType: "application/json; charset=utf-8",
    dataType: "json",
    success: function(data) {
      console.log("Sleep data!", data);

      if (data.error) {
        respond(data.error);
        window.location.replace("/login/jawbone");
      }

      switch (type) {
        case "REM sleep":
          respond("You had " + toHours(data.items[0].details.rem) + " of REM sleep.");
          break;
        case "deep sleep":
          respond("You had " + toHours(data.items[0].details.sound) + " of deep sleep.");
          break;
        case "light sleep":
          respond("You had " + toHours(data.items[0].details.light) + " of light sleep.");
          break;
        case "sleep":
          respond("You had " + toHours(data.items[0].details.duration) + " of sleep last night. That includes " + toHours(data.items[0].details.rem) + " of REM sleep, " + toHours(data.items[0].details.sound) + " of deep sleep and " + toHours(data.items[0].details.light) + " of light sleep.");
          break;
      }
    },
    error: function() {
      respond(messageInternalError);
    }
  });
}

toHours() is a crude and quick function that formats our times into sentences like “1 hour, 53 minutes and 59 seconds”:

function toHours(secs) {
  hours = Math.floor(secs / 3600),
  minutes = Math.floor((secs - (hours * 3600)) / 60),
  seconds = secs - (hours * 3600) - (minutes * 60);

  hourText = hours + (hours > 1 ? " hours, " : " hour, ");
  minuteText = minutes + (minutes > 1 ? " minutes " : " minute ");
  secondText = seconds + (seconds > 1 ? " seconds" : " second");

  return hourText + minuteText + "and " + secondText;
}

As you’ll see when looking into the requestSleepData() function, the end result is a call to respond() — the same function that previously took Api.ai’s voice response. We reuse our existing functionality to bring speech to our own response, allowing our assistant to tell the user this information once it is ready.

One last part of our front end JavaScript to point out is our error handling. If we have an issue with how Jawbone returns data (usually due to not being logged into the service), our server responds with a JSON value in the format of {"error" : "Our error message"}. The assistant sees this and automatically takes the user to our OAuth login page:

if (data.error) {
  respond(data.error);
  window.location.replace("/login/jawbone");
}

Our Node.js Server

Our Node.js server is based off the one used in Connecting to the Jawbone UP API with Node.js. If any of the code looks confusing, feel free to refer to that earlier article as it explains all about connecting to the Jawbone API via OAuth and setting up a HTTPS server to run it. If you don’t have a Jawbone Up, the same concepts can be used for other IoT devices — you would just need to add your own methods of responding to GET requests with different data (and you may not need to worry about OAuth). The Jawbone Up data here is just an example.

Our Jawbone data has been adjusted from the earlier article to provide a simple JSON response rather than formatting it all into a table template view. The variables of up and options have been moved to global variables so that they can be reused in multiple requests to the API (in that other SitePoint example, we only requested data in one big chunk each time).

To log into the Jawbone API via OAuth, the user can go to /login/jawbone. However, as you saw above, they don’t need to know to do this. Our assistant can also redirect them if it notices they aren’t logged in. You could also add a new intent into your Api.ai agent which understands the phrase “log me into my Jawbone Up data”, if you wanted to make this truly seamless. Our login route in Node.js looks like so:

app.get("/login/jawbone", 
  passport.authorize("jawbone", {
    scope: ["basic_read","sleep_read"],
    failureRedirect: "/"
  })
);

Once we’ve logged into the Jawbone API via passport.use("jawbone", new JawboneStrategy()), we assign this access to our up variable and direct the user to /barry. You could redirect the user to any path of your choice, as long as it is different from your root directory (that seemed to just cause an endless struggle for my server). I chose /barry, as I named my assistant Barry and thought it self explanatory (the page shows the exact same index view and thus isn’t different in any way). You could also use this as a way of providing a different view for users who have logged into their Jawbone device successfully if you so desire. Once logged in, the user can go back to the root https://localhost:5000 page and use Up functionality as well.

Sending Back Our IoT Data

Our retrieval of Jawbone data is done in a very simple way upon receiving a GET request for /sleep_data. We check whether the up variable is defined — if not, our user hasn’t logged in and we tell the web app this so that it can perform the redirection and tell the user they’ll need to log in. We do the same thing if Jawbone returns any errors when we call up.sleeps.get() and if jawboneData.items is not defined:

app.get("/sleep_data", function(req, resp) {
  if (up !== undefined) {
    up.sleeps.get({}, function(err, body) {
      if (err) {
        console.log("Error receiving Jawbone UP data");
        resp.send({"error": "Your sleep tracker isn't talking to me. Let's try logging in again."});
      } else {
        var jawboneData = JSON.parse(body).data;

        if (jawboneData.items) {
          resp.send(jawboneData);
        } else {
          console.log("Error: " + jawboneData);
          resp.send({"error": "Your sleep tracker isn't talking to me. Let's try logging in again."});
        }
      }
    });
  } else {
    console.log("Up is not ready, lets ask to log in.");
    resp.send({"error": "Your sleep tracker isn't talking to me. Let's try logging in again."});
  }
});

The errors here could be caused by other factors too, however to keep it simple, I’m focusing on getting them to try logging in again, in a production level app we’d want to look into various causes and adjust our response.

If all is well and we receive a valid response, we send it as a JSON response back to the web app to read and parse through nicely:

if (jawboneData.items) {
  resp.send(jawboneData);
}

With the web app and our Node.js server working together, we should be able to retrieve sleep data from our Jawbone Up device now. Let’s give it a go.

In Action

We run our server with the usual node server.js. Remember you’ll need to have run npm install for your npm modules and will need to have a certificate on your server to run it via HTTPS.

Visiting https://localhost:5000 in our web browser should bring us to our AI assistant. Let’s ask it how much sleep we’ve had:

Our assistant redirecting us to log in

Turns out we aren’t logged in yet. It directs us to the Jawbone Up OAuth login screen. Log in and agree to provide access to your data, then click “Agree”:

Logging into the Jawbone Up API

If we ask it this time around, we will receive the right answer:

Our Api.ai assistant telling us our sleep patterns

We can also ask it something more specific, such as “How much REM did I get?” to test out the parameters:

Our Api.ai assistant showing how much REM sleep we have gotten

Conclusion

That concludes quite a varied exploration of Api.ai’s capabilities for now! You could expand upon this example to allow it to understand date ranges (e.g. “How much sleep did I get on Tuesday?”) or to format the time slightly better (notice a slight bug in one of the responses?). You might have much nicer and shorter ways of phrasing the responses too. Personalise it — make it your own!

As you can see, by using these methods you can connect up any Node.js compatible or web API compatible service to your Node.js server, hook it up to an intent in your Api.ai agent and teach it all sorts of things! You could connect up plenty of IoT devices via IFTTT, connect up your LIFX smart lights via IFTTT or even connect up your very own Nodebot. Possibilities are limited only by the devices you have at your disposal!

If you have been following along and building your own personal assistant using Api.ai, I’d love to hear how you went! What have you connected it to? Let me know in the comments below, or get in touch with me on Twitter at @thatpatrickguy.

Free Guide:

7 Habits of Successful CTOs

"What makes a great CTO?" Engineering skills? Business savvy? An innate tendency to channel a mythical creature (ahem, unicorn)? All of the above? Discover the top traits of the most successful CTOs in this free guide.

No Reader comments

Recommended
Sponsors
Because We Like You
Free Ebooks!

Grab SitePoint's top 10 web dev and design ebooks, completely free!

Get the latest in Front-end, once a week, for free.