Making HTTP Requests in Node.js

November 17, 2012

In previous articles, I’ve focused on creating a Node.js server to handle HTTP requests. This article looks at the problem in reverse, by showing you how to make HTTP requests from your Node.js applications. You may be asking yourself why you would want to do that. Two applications come to mind right away – web scraping and proxying. Scrapers are pieces of software which download web pages and programatically extract information from them. Proxy servers act as intermediaries, forwarding client requests to other servers and returning the responses.

The Request Module

The simplest way to create HTTP requests in Node.js is by using the request module. Written by Mikeal Rogers, request allows you to make all types of HTTP requests, including GET, POST, PUT, and DELETE. Its flexibility makes the request module ideal for interacting with RESTful APIs. You can install request using the following npm command.

npm install request

Once you have installed request, create a new Node.js file containing the code shown below. The first line of the example is used to import request into the program. Making an HTTP request is as simple as calling the request() function. In this example, we are making a request to download the JSPro homepage. The second argument to request() is, of course, a callback function.

var request = require("request");

request("http://www.sitepoint.com", function(error, response, body) {
  console.log(body);
});

The request() callback function takes three arguments – error, response, and body. As the name implies, error contains information regarding any problems that occurred during the HTTP request. The response argument is an object containing a variety of data returned from the call, including the status code, headers, and much more. The third callback argument, body, is the body of the response. In the previous example, body contained the source code for the JSPro homepage.

The request module can also be combined with the file system module to stream HTTP requests to and from files. The following example again fetches JSPro. However, instead of displaying the source code in the console, it is written to a file named “jspro.htm”.

var request = require("request");
var fs = require("fs");

request("http://www.sitepoint.com").pipe(fs.createWriteStream("jspro.htm"));

Parameterizing Requests

The previous examples have created simple GET requests for URLs. However, we can modify the HTTP request by passing a configuration object as the first argument to request(). The following example uses the more powerful syntax to again download JSPro. Notice the numerous configuration parameters. The uri and method parameters tell request to send a GET request to JSPro.

var request = require("request");

request({
  uri: "http://www.sitepoint.com",
  method: "GET",
  timeout: 10000,
  followRedirect: true,
  maxRedirects: 10
}, function(error, response, body) {
  console.log(body);
});

The timeout parameter tells request to wait 10,000 milliseconds (ten seconds) for a response before giving up. The final two parameters deal with following redirects. Many pages perform redirects behind the scenes without the user’s knowledge. The followRedirect and maxRedirects parameters instruct request to follow up to ten 3xx HTTP redirects. Also, note that request() supports many other parameters which are not shown here.

Submitting Form Data

As previously mentioned, request supports more than just GET requests. One very common operation is the submission of form data using a POST request. The following example uses the form parameter of request() to send a user’s name as part of a POST. The PHP page then responds by greeting the user by name.

var request = require("request");

request({
  uri: "http://www.cjihrig.com/development/php/hello_form.php",
  method: "POST",
  form: {
    name: "Bob"
  }
}, function(error, response, body) {
  console.log(body);
});

Dealing with Cookies

Many sites track various user interactions using cookies. Cookies are small pieces of data that are passed back and forth between the client and server with every HTTP transaction. To facilitate the use of cookies, request provides the jar() and cookie() methods. Individual cookies are created using the cookie() method. On line 3 of the following example, a cookie is created which specifies the user’s name. The cookie is then added to the cookie jar created on line 2. When the HTTP request is made, the jar parameter is used to send the cookie jar to the server.

var request = require("request");
var jar = request.jar();
var cookie = request.cookie("name=John");

jar.add(cookie);
request({
  uri: "http://www.cjihrig.com/development/php/hello_cookies.php",
  method: "GET",
  jar: jar
}, function(error, response, body) {
  console.log(body);
});

When the response is received, the user will be greeted as John, the name stored in the cookie. The cookie jar is also updated with any changes made by the server. If you print out the cookie jar before and after the request, you’ll see that the server changed the value of the cookie from "John" to "Stranger". The same cookie jar can be passed to consecutive HTTP requests in order to mimic a real user’s interactions with a site.

Conclusion

This article has introduced the simple, yet powerful, request module. As this article is not comprehensive, I recommend at least browsing over the documentation to see what else request can do. Stay tuned for an upcoming article which will cover web scraping using request in more depth.

And if you enjoyed reading this post, you’ll love Learnable; the place to learn fresh skills and techniques from the masters. Members get instant access to all of SitePoint’s ebooks and interactive online courses, like Jump Start JavaScript.
Comments on this article are closed. Have a question about JavaScript? Why not ask it on our forums?

JavaScript: Novice to Ninja, 2nd Edition