Headless WebKit and PhantomJS

Share this article

Key Takeaways

  • PhantomJS, a headless browser built on top of WebKit, allows for faster automation and testing of web pages programmatically without the need for a graphical user interface.
  • PhantomJS provides powerful features such as the ability to interact with a page via JavaScript, making it easy to automate tasks like clicking buttons, submitting forms, and even loading and manipulating web pages with standard DOM API or libraries like jQuery.
  • PhantomJS offers an extensive file system API, enabling applications to store source code to the file system, take screenshots of web pages, and even include external script files to a page.
  • Despite its powerful features, PhantomJS does not integrate particularly well with Node.js, with existing projects often using the child process module to spawn instances of PhantomJS and communicate with Node.js via WebSockets.
If you’re reading this article, you most likely know what a browser is. Now take away the GUI, and you have what’s called a headless browser. Headless browsers can do all of the same things that normal browsers do, but faster. They’re great for automating and testing web pages programmatically. There are a number of headless browsers in existence, and PhantomJS is the best. Built on top of WebKit, the engine behind Chrome and Safari, PhantomJS gives you a ton of browser power, without the heavy GUI. Getting started with PhantomJS is easy – just download the executable. Next, create a file named hello.js and add the following lines.
console.log("Hello World!");
phantom.exit();
To execute the script, run the command shown below. Note, the phantomjs executable must either be in your current directory, or somewhere in your environment’s PATH. If everything is configured properly, PhantomJS will print Hello World! to the console, and then terminate when phantom.exit() is called.
phantomjs hello.js

Working With Web Pages

Once PhantomJS is up and running, you can begin automating the Web. The following example loads the Google home page, and then saves a screenshot to a file. Line 1 creates a new instance of a web page. On line 4, the web page loads google.com. Once the page finishes loading, the onLoadFinished() callback function is executed. The callback receives a single argument, status, which indicates whether the page loaded successfully or not. The URL of the loaded page is available in page.url. This property can be particularly useful when pages contain redirects, and you want to know exactly where you landed. A screenshot is taken on line 8 using the page’s render() method. render() can create PNG, GIF, JPEG, and PDF files.
var page = require("webpage").create();
var homePage = "http://www.google.com/";

page.open(homePage);
page.onLoadFinished = function(status) {
  var url = page.url;

  console.log("Status:  " + status);
  console.log("Loaded:  " + url);
  page.render("google.png");
  phantom.exit();
};

Page Settings

Page objects have a number of settings which can be customized based on your application’s needs. For example, if you’re only interested in downloading source code, you can speed up your application by ignoring image files and turning off JavaScript. The previous example is rewritten below to reflect these changes. The changed settings are shown on lines 3 and 4. Note that any settings changes must take place before the call to open(). If you view the screenshot from this example, you will notice that the Google logo image is missing, but the rest of the page is in tact.
var page = require("webpage").create();
var homePage = "http://www.google.com/";

page.settings.javascriptEnabled = false;
page.settings.loadImages = false;
page.open(homePage);
page.onLoadFinished = function(status) {
  var url = page.url;

  console.log("Status:  " + status);
  console.log("Loaded:  " + url);
  page.render("google.png");
  phantom.exit();
};

Accessing the File System

So far, our examples have loaded pages and saved screenshots as image files. While this is undoubtedly cool, many applications would prefer to store the source code to the file system. PhantomJS makes this possible by providing an extensive file system API. The following example uses the FileSystem module to write the google.com source code to a file. First, the FileSystem module is imported on line 2. On line 6, the output file is opened for writing. On line 7, the data is written to file using the write() method. The actual source code is available via the page’s content property. Finally, the file is closed and PhantomJS is terminated.
var page = require("webpage").create();
var fs = require("fs");
var homePage = "http://www.google.com/";

page.open(homePage);
page.onLoadFinished = function(status) {
  var file = fs.open("output.htm", "w");

  file.write(page.content);
  file.close();
  phantom.exit();
};

Executing JavaScript

One of the most powerful features of PhantomJS is the ability to interact with a page via JavaScript. This makes it extremely easy to automate tasks such as clicking buttons and submitting forms. Our next example performs a Web search by loading the Google home page, entering a query, and then submitting the search form. The beginning of the example should look familiar. The new stuff begins, on line 8, where we determine which page has been loaded. If this is the home page, the page’s evaluate() method is called. evaluate() executes the code you provide in the context of the page. This essentially gives you the same power as the page’s original developer. How cool is that?
var page = require("webpage").create();
var homePage = "http://www.google.com/";

page.open(homePage);
page.onLoadFinished = function(status) {
  var url = page.url;

  console.log("Status:  " + status);
  console.log("Loaded:  " + url);

  if (url === homePage) {
    page.evaluate(function() {
      var searchBox = document.querySelector(".lst");
      var searchForm = document.querySelector("form");

      searchBox.value = "JSPro";
      searchForm.submit();
    });
  } else {
    page.render("results.png");
    phantom.exit();
  }
};
Inside of evaluate(), we locate the search box and form. We set the value of the search box to “JSPro”, and then submit the form. This will cause the page’s onLoadFinished() method to be triggered again. However, this time a screen shot is taken of the search results, and PhantomJS exits. PhantomJS also provides two methods, includeJs() and injectJs(), which allow you to add external script files to a page. includeJs()
is used to include any script file that is accessible from the page. For example, you can include jQuery in our previous example using the following code. Notice the call to includeJs() on line 9, as well as the jQuery syntax inside of evaluate().
var page = require("webpage").create();
var homePage = "http://www.google.com/";

page.open(homePage);
page.onLoadFinished = function(status) {
  var url = page.url;

  console.log("Status:  " + status);
  console.log("Loaded:  " + url);

  if (url === homePage) {
    page.includeJs("https://code.jquery.com/jquery-1.8.3.min.js", function() {
      console.log("Loaded jQuery!");
      page.evaluate(function() {
        var searchBox = $(".lst");
        var searchForm = $("form");

        searchBox.val("JSPro");
        searchForm.submit();
      });
    });
  } else {
    page.render("results.png");
    phantom.exit();
  }
};
The injectJs() method is similar to includeJs(). The difference is that the injected script file does not need to be accessible from the page object. This allows you to, for example, inject scripts from your local file system.

PhantomJS and Node.js

Sadly, PhantomJS does not integrate particularly well with Node.js. A few projects have been created which try to control PhantomJS from Node.js, but they are all a bit of a kludge. Existing projects use the child process module to spawn instances of PhantomJS. Next, PhantomJS loads a special web page, which uses WebSockets to communicate with Node.js. It might not be ideal, but it works. Two of the more popular PhantomJS Node modules are node-phantom and phantomjs-node. I recently started working on my own PhantomJS Node module named ghostbuster. Ghostbuster is similar to node-phantom, but attempts to reduce callback nesting by providing more powerful commands. Making fewer calls to PhantomJS also means less time is wasted communicating over WebSockets. Another alternative is zombie.js, a lightweight headless browser built on top of jsdom. Zombie isn’t as powerful as PhantomJS, but it is a true Node.js module.

Conclusion

After reading this article, you should have a basic grasp on PhantomJS. One of the nicest features about PhantomJS is how simple it is to use. If you’re already familiar with JavaScript, the learning curve is minimal. PhantomJS also supports a variety of other features that weren’t covered in this article. As always, I encourage you to check out the documentation. There are also a number of examples which show off PhantomJS in all its glory!

Frequently Asked Questions about Headless WebKit and PhantomJS

What is the main difference between Headless WebKit and PhantomJS?

Headless WebKit and PhantomJS are both tools used for automating web browsers. However, the main difference lies in their functionality. Headless WebKit is a browser without a graphical user interface that can be controlled programmatically for automation, testing, and server rendering. On the other hand, PhantomJS is a scriptable headless browser used for automating web page interaction, providing JavaScript API enabling automated navigation, screenshots, user behavior, and assertions.

Is PhantomJS still maintained?

As of March 2018, PhantomJS is no longer actively maintained. The main reason behind this is the emergence of modern headless browsers like Chrome Headless and Firefox Headless, which offer more features and better support.

What are some alternatives to PhantomJS?

With PhantomJS no longer being maintained, several alternatives have emerged. These include Puppeteer, a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol, and Selenium WebDriver, a collection of open-source APIs used to automate the testing of a web application.

How does PhantomJS work?

PhantomJS works by providing a JavaScript API that enables automated navigation, screenshots, user behavior, and assertions. It’s a headless WebKit scriptable with a JavaScript API. It has fast and native support for various web standards: DOM handling, CSS selector, JSON, Canvas, and SVG.

Can I use PhantomJS for web scraping?

Yes, PhantomJS can be used for web scraping. It allows you to load and manipulate web pages with the standard DOM API, or with usual libraries like jQuery.

How can I install PhantomJS?

PhantomJS can be installed via npm (Node Package Manager). You can use the command “npm install phantomjs” in your terminal or command prompt.

What is the role of Headless WebKit in server-side rendering?

Headless WebKit plays a crucial role in server-side rendering as it allows the server to pre-render the JavaScript-rendered page, convert it into HTML, and then send it to the client. This improves the performance and SEO of your web application.

Can I use Headless WebKit for automated testing?

Yes, Headless WebKit is an excellent tool for automated testing. It allows you to run your tests in a real browser environment without the need for a visible UI.

How can I install Headless WebKit?

The installation process for Headless WebKit varies depending on the specific tool you’re using. For instance, if you’re using Puppeteer, you can install it via npm using the command “npm install puppeteer”.

What are the advantages of using Headless WebKit over traditional browsers for automation?

Headless WebKit offers several advantages over traditional browsers for automation. It’s faster, as it doesn’t need to spend time rendering visuals. It also allows for automated, scriptable browsing which is useful for testing and web scraping.

Colin IhrigColin Ihrig
View Author

Colin Ihrig is a software engineer working primarily with Node.js. Colin is the author of Pro Node.js for Developers, and co-author of Full Stack JavaScript Development with MEAN. Colin is a member of the Node.js Technical Steering Committee, and a hapi core team member. Colin received his Bachelor of Science in Engineering, and Master of Science in Computer Engineering from the University of Pittsburgh in 2005 and 2008, respectively.

Headless BrowserIntermediatePhantomJS
Share this article
Read Next
Get the freshest news and resources for developers, designers and digital creators in your inbox each week