How to Build the Best Browser Test Suite

Contributing Editor

your browser test suiteTesting a website used to be so much easier. You would develop and test your code in Internet Explorer and be totally oblivious to anything but major HTML, CSS or JavaScript errors. Even the most conscientious developers would only test a couple of other minority browsers.

Today, we are faced with 5 mainstream browsers and, to complicate matters further, each of those has multiple versions. Usage statistics vary from site to site, but the following browsers should cater for 99% of users:

  • Microsoft IE6 and IE7 (IE8 coming soon)
  • Mozilla Firefox 2 and 3 (Firefox 3.5 coming soon)
  • Apple Safari 3 (Safari 4 coming soon)
  • Google Chrome 1
  • Opera 9

Need to get closer to 100% user coverage? Throw in IE5, IE5.5, Netscape 6, Mozilla 1.7, mobile Opera mini, a PlayStation 3, a PSP, a few historical versions of Mac Safari, and a couple of screen readers. This is a daunting list: testing your web application could take longer than building it!

Fortunately, it is easy to install multiple versions of Firefox, Safari and Opera on a single PC: just choose a different installation folder. It is early days for Chrome, but I suspect Google will issue automatic updates so most users will have the latest version.

As always, the tough one is Internet Explorer. The browser is tightly integrated into Windows so only one version can be installed. Vista users can not downgrade and must use either IE7 or IE8. However, there are a number of ways to test multiple browsers on a single PC.

Online Browser Simulators

Several online systems can be used to show how your site works across various combinations of OS and browser:

There is nothing to install, however, your site must be live, testing is a slow process, and the better services charge a fee. You only see a screenshot so debugging is tricky and you certainly will not discover JavaScript errors or other subtle problems.

These systems are undoubtedly useful, but it is probably better to use them towards the end of your testing process.

Install Multiple Versions of Internet Explorer

Multiple versions of IE can be installed on a single PC. Standalone versions of IE6 (and any version back to v3.0) can be downloaded from the evolt.org browser archive. XP users can simply extract the files and create a shortcut to iexpore.exe. Unfortunately, they do not work in Vista.

An alternative is the Xenocode Browser Sandbox which provides IE6, IE7, IE8, Firefox 2, Firefox 3, Chrome, Opera and Safari. These are single .exe files that use virtualization technology to ensure the browser runs in its own segregated environment without affecting your core system.

Finally, there is IETester from DebugBar. This is certainly the easiest application to install and use. It offers tabbed browsing with all versions of IE from 5.5 to 8.0 beta.

IETester

Whilst these tools are useful, you should be aware that you are not running ‘real’ installations of IE. The problems include:

  • There are some interface quirks and features such as the browser history, cookies and the cache may not work as you expect.
  • Filters and transitions fail in IE5.5/6.0 so 24-bit PNGs and transparency effects will not appear. There is a workaround for XP standalone browsers, but neither Xenocode or IETester appear to implement a solution.
  • The old browser might use the latest IE7 JavaScript engine.
  • Conditional CSS comments will fail or become confused – which is another good reason to avoid them!

Use a Virtual Machine

The most reliable way to run an old version of IE is to use a dedicated PC with a clean installation of Windows. However, it is not necessary to fill your office with clunky old desktops: you can use virtual machine software to emulate any number of PCs on a single machine. Popular VM software includes:

  • VMware: one of the first and best. Confusingly, VMware Server is free whereas VMware Workstation is a more advanced commercial product offering snapshots and cloning.
  • VirtualBox: an excellent open source product owned by Sun Microsystems. It feels as fast as VMware, although lacks a few minor features such as file drag-and-drop between the host and guest OS.
  • Microsoft Virtual PC: a solid free VM, although it is not as fast or fully-featured as its competitors.

Creating a virtual machine

After installing the software, you create a new PC then boot-up with an Window installation disk or ISO in the virtual CD drive. Any version of Windows from 98 to XP should be adequate for installing old versions of IE.

There are several other advantages to using virtual machines:

  • Virtual disks can be copied and backed up. You can trash the OS and revert to a previous image in seconds.
  • You can install older browsers or try out new software without polluting your main PC.
  • You can install other operating systems such as Linux and test a range of other browsers (Lynx is great for assessing accessibility).

Building your browser suite takes time and none of the solutions are perfect. However, the newly released Microsoft SuperPreview could be the answer to many browser testing problems…

Have I missed your preferred method of testing multiple browsers?

Free book: Jump Start HTML5 Basics

Grab a free copy of one our latest ebooks! Packed with hints and tips on HTML5's most powerful new features.

  • SoreGums

    I’m not bothering with IE6 anymore – use IE7 or a newer browser.
    I’ve started spreading this message on my sites.
    Be good if M$ had killbits in its code and ofrced an upgrade, well where pratical. Can’t force a win98 person to IE7…

  • loganathan

    always, I use IE + FF .. other browsers i don’t test it… will this make problem to my sites?

  • http://www.optimalworks.net/ Craig Buckler

    will this make problem to my sites

    It really depends on your target audience. However, I’d recommend you test as many browsers as possible.

    In my experience, a site that works in IE6 and Firefox is likely to work in every other browser, although quirks can still occur. Even if you’ve given up on IE6, it’s still worth a cursory glance to ensure your site remains usable (or unusable if you’re trying to eradicate it!)

    Safari, Chrome, and Opera have a good 10% market share between them so it’s worth spending a few minutes on each.

  • Anonymous

    Many many thanks,most developer is not aware with this.Nice post..

  • biswa

    Many many Thanks,many developer as well designer is not aware of this.Thanks nice post and very useful.

  • Anonymous

    I notice that this is a Windows solution (with a nod to Safari which is native to Macs IIRC).

    Any solutions for other operating systems such as Ubuntu/other Debian-based distros, or even Mac?

    I, myself, do most of my web developing on the road on my laptop which only has linux since even XP refuses to work on it (and I don’t plan on switching back ;)

  • http://www.lowter.com charmedlover

    It’s “Opera Mini”, not “Opera mini”.

  • krues8dr

    Doesn’t the evolt standalone version of IE6 still have problems with cookies and conditional comments? There’s also Multiple IE from Tredosoft, but that seems to use parts of the IE7 engine in IE 6. At least for the current moment, the only real option is to use separate virtual machines.

  • http://www.optimalworks.net/ Craig Buckler

    Doesn’t the evolt standalone version of IE6 still have problems with cookies and conditional comments?

    Yes – the article mentions some of the drawbacks. VMs or real PCs are your safest bet for IE6 testing.

  • pjries

    IE Tester from DebugBar works very well for me. It handles the conditional style sheets I’ve used to get around IE 6 bugs and it allows me to refresh the page, clearing the cache along the way. Easy to download, install and use without having to switch to a Virtual Machine. Worth taking a closer look at, I think.

  • http://www.tyssendesign.com.au Tyssen

    I’ve been using Xenocode for IE6 a bit lately; conditional comments work fine, but javascript implementation can be a bit flaky.

  • JD

    One thing with the VM route, doing the IE tests is dead simple for Windows users using Virtual PC. Microsoft provides free pre-installed virtual machine drives containing XP or Vista with IE 6, 7 and 8 in various combinations. I have IE 7 on my main Vista development machine, with 2 virtual machines, onr running XP with IE6 and one running XP with IE8 as my test platforms. The only downside, is that the installs expire every couple of months to discourage people from using them for anything but their intended purpose of testing websites. Still a good deal though.

  • Gabi Moore

    I have used http://tredosoft.com/Multiple_IE for years now. The only problem is that once in a while, for no reason, all the earlier versions of IE think they are IE7, so they ignore any conditional styles you might have. But what I like about it is that you actually have the real IE5.5, IE6 and IE7 on the same machine, and not just test versions of them.

  • http://www.optimalworks.net/ Craig Buckler

    Any solutions for other operating systems such as Ubuntu/other Debian-based distros, or even Mac?

    There are several virtual machine solutions for both Mac and Linux. There is also IEs4Linux; it’s obviously not a perfect representation of IE, but good enough for simple testing.

  • Craig Bovis

    May be worth an update to include SuperPreview for Internet Explorer in this article.

  • http://www.optimalworks.net/ Craig Buckler

    May be worth an update to include SuperPreview for Internet Explorer in this article.

    There’s a link at the bottom of the article.

  • pjries

    Has anyone been able to get SuperPreview to work? I’m still running Windows XP Pro and it crashes every time I start it up. Had to install .NET 3.5 first. Mmaybe it only works on Vista?

  • http://www.sitepoint.com/ mmj

    People often suggest using virtual machines for running Windows instances, but how do licensing issues work? Do you need additional Windows licenses for the virtual machines? What serial codes do you use when installing in the virtual machine and do you do the product activation on each instance, or do people just pirate it (and is that frowned upon)?

    It’s something I’ve always wondered but been afraid to ask.

  • http://www.optimalworks.net/ Craig Buckler

    Do you need additional Windows licenses

    Yes. However, if you have an old unused PC, then I’m sure you can transfer the licence. XP is preferable, but anything back to Win98 will do (you can still buy old legal licences but they can be expensive).

    I’m not sure what MS think about VM cloning. In VMware Workstation you can take an OS snapshot and make a ‘new’ VM instance. However, you’re still effectively using one OS so I suspect it’s OK.

    I think MS released a time-restricted Virtual PC with XP and IE8 beta. It would be great if they could do the same for IE6 and IE7 too.

  • MauiMan2

    Throw in IE5, IE5.5, Netscape 6, Mozilla 1.7, mobile Opera mini, a PlayStation 3, a PSP,

    I’m actually reading this article right now with Netscape 9 and it works just fine.

  • MGoddard

    I always wondered about the licensing issue with Win as well. I just converted over to Mac and using Parallels for my Win VM however I would like to create multiple VM’s so I can have working environment of IE6, IE7 and IE8.

    Is there a clause in the license that allows you to install Win on multiple computers as long as they are only run one at a time and not simultaneously??

  • frozenmyst

    Filters and transitions fail in IE5.5/6.0 so 24-bit PNGs and transparency effects will not appear

    Wrong :

    On Windows XP, IETester does not have any problem with transparent PNGs via CSS filters in IE6…

  • frozenmyst

    IE8 has a feature of rendering with the IE7 engine: try F12 and look at the options.

  • http://www.optimalworks.net/ Craig Buckler

    IE8 has a feature of rendering with the IE7 engine

    Unfortunately, there are reports that it’s not identical to a real IE7. It’s probably good enough for most testing, but MS aren’t really helping!

  • http://www.optimalworks.net/ Craig Buckler

    On Windows XP, IETester does not have any problem with transparent PNGs via CSS filters in IE6…

    It certainly doesn’t work on Vista with filters applied in JavaScript.

  • ‘Paul McKeown

    Doesn’t the evolt standalone version of IE6 still have problems with cookies and conditional comments? There’s also Multiple IE from Tredosoft, but that seems to use parts of the IE7 engine in IE 6. At least for the current moment, the only real option is to use separate virtual machines.

    Yes – the article mentions some of the drawbacks. VMs or real PCs are your safest bet for IE6 testing.

    Use the Tredosoft Multiple IE packager. I have used it for a couple of years and find it indispensible. Conditional comments and javascript (don’t know about cookies, go look see) are handled correctly by each standalone – read the website. Just sorry that the clever fellow that put it all together has decided not to support it any longer. IE 8 is not supported. Pity.

    I also use the IE Tester tool, as it supports IE 8. Not so sure about it’s reliability, but I suspect that that is simply a measure of the shorter period that I have used it. It is actively under development and I must say that its creator provides excellent support – I reported one fault on its forum and the response was instant. I know that others have had the same experience. See the website.

    One tool that hasn’t been mentioned is KDE 4 for Windows (!!), which I use to give easy one workstation access to the leading Linux browsers. Big fan, particularly of Konqueror 4 under Windows! Note to Steve Jobs: stop being a prissy mare, and give us OS/X (or OS/XI ??) under Windows and Linux.

    Ultimately of course one must look at it using the real browser. The virtual OS solutions are useful for this, although the images eat up disc space like it’s for breakfast and are heavy on a PC’s runtime resources.

    The various online services are interesting, but they all inevitably seem to miss some useful or interesting browser or platform. Anyone know one which provides IE Mac 5.23 and IE Mac 5.17? With access to a virtual session for behaviour as well as just layout?

    My ultimate test is always a select audience of 100 or so “crash test dummies” who use a widely varied range of browsing tools.

    One thing that your article doesn’t really deal with is testing for non desktop media. I restrict this at the moment to Lynx to linearise the output, Jaws 9 for an idea of how screen readers deal with it. I always print preview each page also.

    Mobiles are problem – any ideas? I use the mobiforge .mobi emulators from time to time, but am never sure how good the feedback is that I’m getting…

  • ‘Paul McKeown

    Also has anyone managed to install multiple versions of Safari under Windows? Currently I use the awfully clumsy method of having installers for 3.2.2 and 4.0 beta on my desktop and install 40 beta/test/uninstall 40 beta/reinstall 322 whenever I need to test with 4.0 beta Win.

    It’s a bit aggravating… Mozilla don’t make me got through this, why should Apple?

    Have to say Safari 4.0 beta is bliksem fast! Love it. I suspect that a lot of people might end up using it under Windows, especially as it no longer looks so strangely err un-PC.