Making HTTP Requests in Node.js
In previous articles, I’ve focused on creating a Node.js server to handle HTTP requests. This article looks at the problem in reverse, by showing you how to make HTTP requests from your Node.js applications. You may be asking yourself why you would want to do that. Two applications come to mind right away – web scraping and proxying. Scrapers are pieces of software which download web pages and programatically extract information from them. Proxy servers act as intermediaries, forwarding client requests to other servers and returning the responses.
The Request Module
The simplest way to create HTTP requests in Node.js is by using the request
module. Written by Mikeal Rogers, request
allows you to make all types of HTTP requests, including GET
, POST
, PUT
, and DELETE
. Its flexibility makes the request
module ideal for interacting with RESTful APIs. You can install request
using the following npm command.
npm install request
Once you have installed request
, create a new Node.js file containing the code shown below. The first line of the example is used to import request
into the program. Making an HTTP request is as simple as calling the request()
function. In this example, we are making a request to download the JSPro homepage. The second argument to request()
is, of course, a callback function.
var request = require("request");
request("http://www.sitepoint.com", function(error, response, body) {
console.log(body);
});
The request()
callback function takes three arguments – error
, response
, and body
. As the name implies, error
contains information regarding any problems that occurred during the HTTP request. The response
argument is an object containing a variety of data returned from the call, including the status code, headers, and much more. The third callback argument, body
, is the body of the response. In the previous example, body
contained the source code for the JSPro homepage.
The request
module can also be combined with the file system module to stream HTTP requests to and from files. The following example again fetches JSPro. However, instead of displaying the source code in the console, it is written to a file named “jspro.htm”.
var request = require("request");
var fs = require("fs");
request("http://www.sitepoint.com").pipe(fs.createWriteStream("jspro.htm"));
Parameterizing Requests
The previous examples have created simple GET
requests for URLs. However, we can modify the HTTP request by passing a configuration object as the first argument to request()
. The following example uses the more powerful syntax to again download JSPro. Notice the numerous configuration parameters. The uri
and method
parameters tell request
to send a GET
request to JSPro.
var request = require("request");
request({
uri: "http://www.sitepoint.com",
method: "GET",
timeout: 10000,
followRedirect: true,
maxRedirects: 10
}, function(error, response, body) {
console.log(body);
});
The timeout
parameter tells request
to wait 10,000 milliseconds (ten seconds) for a response before giving up. The final two parameters deal with following redirects. Many pages perform redirects behind the scenes without the user’s knowledge. The followRedirect
and maxRedirects
parameters instruct request
to follow up to ten 3xx HTTP redirects. Also, note that request()
supports many other parameters which are not shown here.
Submitting Form Data
As previously mentioned, request
supports more than just GET
requests. One very common operation is the submission of form data using a POST
request. The following example uses the form
parameter of request()
to send a user’s name as part of a POST
. The PHP page then responds by greeting the user by name.
var request = require("request");
request({
uri: "http://www.cjihrig.com/development/php/hello_form.php",
method: "POST",
form: {
name: "Bob"
}
}, function(error, response, body) {
console.log(body);
});
Dealing with Cookies
Many sites track various user interactions using cookies. Cookies are small pieces of data that are passed back and forth between the client and server with every HTTP transaction. To facilitate the use of cookies, request
provides the jar()
and cookie()
methods. Individual cookies are created using the cookie()
method. On line 3 of the following example, a cookie is created which specifies the user’s name. The cookie is then added to the cookie jar created on line 2. When the HTTP request is made, the jar
parameter is used to send the cookie jar to the server.
var request = require("request");
var jar = request.jar();
var cookie = request.cookie("name=John");
jar.add(cookie);
request({
uri: "http://www.cjihrig.com/development/php/hello_cookies.php",
method: "GET",
jar: jar
}, function(error, response, body) {
console.log(body);
});
When the response is received, the user will be greeted as John, the name stored in the cookie. The cookie jar is also updated with any changes made by the server. If you print out the cookie jar before and after the request, you’ll see that the server changed the value of the cookie from "John"
to "Stranger"
. The same cookie jar can be passed to consecutive HTTP requests in order to mimic a real user’s interactions with a site.
Conclusion
This article has introduced the simple, yet powerful, request
module. As this article is not comprehensive, I recommend at least browsing over the documentation to see what else request
can do. Stay tuned for an upcoming article which will cover web scraping using request
in more depth.
And if you enjoyed reading this post, you’ll love Learnable; the place to learn fresh skills and techniques from the masters. Members get instant access to all of SitePoint’s ebooks and interactive online courses, like Jump Start JavaScript.
Comments on this article are closed. Have a question about JavaScript? Why not ask it on our forums?