HTML5 Gaming: Benchmarking Sprite Animations

When I meet with game studios, I often have the same question put to me over and over: if I’m writing/porting my game in HTML5, will it run well on the various targeted devices? Will it be playable or will the gameplay suffer too much? To answer that question, I often use my own experience based on what I know and what worked well during my own tests. But I also had the feeling it wasn’t enough to just provide some good advice. In the meantime, there were some obvious facts. For instance, we all know that mobile devices can’t animate as many sprites as a desktop PC and preserve 60 FPS (frames per second). We know also that combining SVG and Canvas is a good idea to write games that scale across devices but this could also impact the performance. Moreover, even if GPU and hardware acceleration is available on mobile, their hardware architectures differs a lot from the PC and this also impacts a lot on performance. There are dozens of scenarios like that to address and to be aware of while writing HTML5 games for mobiles. But in which proportions? With my friend David Catuhe, we decided to measure these various scenarios and build a benchmark framework to have a better idea on what to pay attention to. It’s called the HTML5 Potatoes Gaming Bench framework. The concept is then to help you benchmarking your targeted platforms and to obtain indicators for your future games: number of simultaneous sprites supported, SVG & Canvas composition performance, usage of videos, etc. It’s a tool we’d like to provide you to help you benchmarking your own scenarios for your games. David has published here an article explaining how we’ve built our benchmark framework: Benchmarking a HTML5 game: HTML5 Potatoes Gaming Bench. It took us time to validate it against tools like the ones that ship with the Windows Performance Toolkit, for instance. We won’t dig into that part here, but this is an important article to read as we needed to rely on the stability of the framework for all the future benchmarks we will build on top of it. This article will illustrate how to use this tool and will concentrate on the number of sprites you can use in your game to maintain a good frame rate on all platforms. We will also discover interesting details on how GPUs (Graphic Processing Units) are actually being used during these scenarios.

Questions I was Asking Myself Before Benchmarking

During my various discussions with game developers, we were often pondering a list of recurring questions:

• What’s the FPS of the benchmarked machine/browser with 200, 1000, 5000 sprites?
• What is the optimum number of sprites the benchmarked device can display at 30 FPS and 60 FPS?
• What is the impact of the resolution of the canvas used to draw the sprites? Can I use a canvas of 800×480 only or can I scale up to 1920×1080? If so, what’s the performance cost?
• What is the performance cost of using hardware scaling (i.e. setting a style.width & style.height higher than the canvas size itself)?

I was wondering also if the performance was due to GPU or CPU limited scenarios. The idea is to know if the CPU is waiting for the GPU to do its job or the opposite. To answer all these questions, I’ve built some specific benchmarks to measure the scenario.

The Core Used by All Benchmarks in This Article

To animate my sprites, I’ve re-used some of my existing assets like these two articles:

• HTML5 Gaming: animating sprites in Canvas with EaselJS
• HTML5 Gaming: building the core objects & handling collisions with EaselJS

We will then use the famous EaselJS framework from the CreateJS suite as I know it runs everywhere and has already been optimized for performance. Please read the main logic exposed in this embedded demo available on JSFiddle here: https://jsfiddle.net/ppwFT/light/. We’re going to re-use the very same logic inside our benchmark framework. The various benchmarks are described like the simplified code below to be able to be integrated in the HTML5 Potatoes Gaming Bench framework:

(function () { 
    "use strict"; 
 
    var canvas, context, stage, screen_width, screen_height, Monsters, contentManager; 
 
    var spritesNumber, displayBench, currentWorkBench, dynamicSpritesNumberLabel; 
    var shadowsEnabled = false; 
    var useRAF = false; 
 
    var init = function (workbench, spritesNumParam, shadowsEnabledParam, canvasSizeXParam,  
                         canvasSizeYParam, then) { 
        // Initialization logic, creating the canvas, the EaselJS Stage, etc. 
    }; 
 
    var buildFixNumberOfSprites = function () { 
        // Our Monsters collection 
        Monsters = new Array(); 
 
        for (var xMonsters = 0; xMonsters < spritesNumber; xMonsters++) { 
            var newRandomMonster = new Monster(contentManager, shadowsEnabled, screen_width, screen_height); 
            Monsters.push(newRandomMonster); 
            stage.addChild(newRandomMonster); 
        } 
 
        createjs.Ticker.addEventListener("tick", tick); 
        // Best Framerate targeted (60 FPS) 
        createjs.Ticker.useRAF = useRAF; 
        createjs.Ticker.setFPS(60); 
 
        // Waiting 1s before doing taking first FPS due to some warm up needed on most machines 
        setTimeout(displayBench, 1000); 
    }; 
 
    // tick function called-back by EaselJS 
    var tick = function () { 
        // looping inside the Monsters collection 
        for (var monster in Monsters) { 
            var m = Monsters[monster]; 
            // Calling explicitly each tick method  
            // to launch the update logic of each monster 
            m.tick(); 
        } 
 
        // update the stage: 
        stage.update(); 
    } 
 
    var stopEaselJSBench = function () { 
        // cleaning things  
    } 
 
    var easelJSSpritesBench_200_NoShadow_640_480 = new POTATOES.GamingBench.Bench("Drawing 200 sprites  
        (640x480)", "http://blogs.msdn.com/davrous", 
        function (workbench) { // Init 
            init(workbench, 200, false, 640, 480, this.onInitCompleted); 
        }, function () { }, function (workbench) { // End 
            stopEaselJSBench(); 
        }); 
 
    POTATOES.GamingBench.registerBench(easelJSSpritesBench_200_NoShadow_640_480); 
})();

You will need to build equivalent code to build your own benchmarks that will be monitored by the framework. The key point is to add your benchmark definition into the monitored collection via the registerBench function. To validate my layout (but not the performance) on the various available browsers and devices, I’ve used Browser Stack. Thanks to modern.IE, you can obtain a three month trial. It lets you test and validate your website layouts on IE from 6 to 10, Firefox, Chrome, Opera, Android & iOS devices. A very useful tool you should have a look at. Most of the below shared results were done on an Asus Ultrabook UX31A (Zenbook Touch) running Windows 8 Pro on a Core i5 integrating an HD4000 GPU from Intel. But I’ve also done some tests on the Nokia Lumia 920, on an iPad 2 under iOS 6.0, Surface RT and Xbox 360. I’m letting you test the shared benchmarks on your own devices.

Average FPS with 200, 1000 & 5000 Sprites

Let’s start by testing some arbitrary numbers of sprites to check how your device/browser will handle that. I inserted a series of benchmarks that will run some sprints of 20 seconds to display in a 640×480 canvas: 200 sprites without shadow, 200 sprites with shadows enabled, 1000 sprites & 5000 sprites without shadow. You can run this benchmark in a separate window via this link: prefixed sprites numbers series benchmark. Just press the “launch prefixed sprites numbers series” button and it will run all of them. At the end, you will have a summary of all benchmarks with a score. The score is simply the number of frames that your device was able to render in 20 seconds. The best logical score is then 1200. As you can render 60 frames per seconds * 20 seconds = 1200 frames. We’re using requestAnimationFrame when available. So it could sometimes occur where the tick is calling us back just a bit above 16ms, that’s why you could have some cases where you will have 1201, like in the following screenshot: If you’re clicking on the average FPS computed, it will display the graph of all instant measured FPS during the complete duration of the benchmark. We’re using the famous d3.js framework for that which generate some SVG. Press the “Back” button to go back to the summary screen or simply touch the graph also.

Some Results and Analysis

I’ve first run this benchmark on my Windows 8 machine with IE10, Chrome 26, Firefox 20 all with hardware acceleration enabled. The good news is that all the modern desktop browsers have some really good hardware acceleration layers in place. IE10 seems to have the best overall score (3030 on my machine) but can’t handle the shadows as well as Chrome (average 27 FPS with 200 sprites vs 12 for IE10). Chrome has an overall score of 2870. It can’t maintain 60 FPS with 1000 sprites on my screen whereas IE10 and Firefox don’t have any problem to achieve that. Firefox 20 has an overall score of 2913. The most difficult test for it is to handle the 200 sprites with shadows (average of 6 FPS vs 12 and 27 for IE10 and Chrome). Except for that, it has the exact same scores as IE10. So, what should we conclude at this stage? The first thing is that you absolutely shouldn’t say to your users: “please use this browser instead” as we’re talking here about building games for the web and all platforms. The users shouldn’t adapt their usages to your code, your code should adapt itself to the usages of your users on the web. This is a different story if you’re building a game only targeting the Windows Store Apps. In this case, you just have to benchmark IE10 to determine your assets limits. But if we think about the web in general, we see that all browsers on can easily handle between 500 and 1000 sprites at 60 FPS on my desktop machine. We absolutely shouldn’t use shadows if we want to keep a good frame rate. For an HTML5 desktop game targeting integrated Intel GPU like mine, you can use a bit less than 1000 animated sprites on screen without any issue. Please bear in mind that this test doesn’t take into account some collisions tests and/or the impact of a physics engine. But it already gives some interesting data. On the mobile side, I’ve run the same series on my Nokia Lumia 920 (Windows Phone 8 is embedding IE10). It can maintain an average of 36 FPS for 200 sprites without shadow. Activating shadows is a no-go as we’re falling at 1 FPS. So, we already have some interesting information to digest. The same rendering and JavaScript engine (IE10) is able to display 1000 sprites at 60 FPS on a desktop PC using an integrated GPU but 200 sprites is already too much to handle for low-end hardware like the current ARM architectures. There is a 5X to 10X performance difference between a mobile and a desktop device. So the next step now is to find what is the optimum number of sprites to maintain 60 FPS (for a good desktop experience) and also 30 FPS (for a relatively good mobile experience).

Optimum number of sprites at 30 and 60 FPS

To help you finding the best optimal lower number of sprites that will keep a 30 or 60 FPS target, I’ve built the following benchmark series: launch 30 and 60 FPS target series.

Some Results and Analysis

This time the score provided at the end is the number of optimal sprites to maintain 30 or 60 FPS. On IE10 on my Windows 8 Asus machine, the HD4000 is able to display 3750+ sprites at 30 FPS and 1600 sprites at 60 FPS. IE10 on my Nokia Lumia 920 running Windows Phone 8, I’m able to display 286 sprites at 30 FPS and 75 at 60 FPS. On Surface RT, IE10 is able to display 370+ sprites at 30 FPS and 100+ sprites at 60 FPS. We’re then falling from 1600 sprites on desktop to 75 on mobile! I’ve even gone further: I’ve benchmarked the version of IE available on the Xbox 360. Indeed, we can imagine building a HTML5 game running inside the Xbox 360 browser. I’ve done it with my game for instance. You can test it here: http://aka.ms/platformer. You can play to the game with the gamepad and we’ve got 60+ FPS! You can also play to the game on mobile/table touch devices thanks to Hand.JS (more details here: Creating an universal virtual touch joystick working for all Touch models thanks to Hand.JS). Let’s benchmark it with the 30/60 FPS target series: The Xbox 360 can display 171 sprites to maintain 30 FPS and 43 to maintain 60 FPS. So, using Internet Explorer as a base of comparison, Asus Zenbook > Surface RT > Nokia Lumia 920 > Xbox 360 to maintain 60 FPS in HTML5 games with sprites animations. I was also curious about changing another parameter to confirm some of my thoughts. I was convinced that the most important piece of hardware to boost the performance in my HTML5 games was the GPU. Indeed, all modern browsers are now heavily using them to offload the job needed to be done for the layout engine. I needed then to have the same machine with 2 GPUs available. Everything else should remain the same (CPU, memory, hard drive, screen resolution, etc). For that, I’ve been using a Sony Vaio Z13 that has an Intel Core i7 Sandy Bridge processor with an integrated HD3000 GPU plus a discrete nVidia GT330m GPU. Using Intel HD3000, the Vaio is able to display 2900 sprites @30 FPS and 1100 sprites @60 FPS. Using the nVidia GT330m, the very same Vaio is able to display 5400 sprites @30 FPS and 2500 @60 FPS. We then have approximately a X2 performance boost by switching from the Intel GPU to the nVidia GPU. GPU really matters for HTML5 games even for 2D canvas games.

Device tested	Max sprites @30 FPS	Max sprites @60 FPS
Asus Zenbook with HD4000	3750	1600
Sony Vaio Z13 with HD3000	2900	1100
Sony Vaio Z13 with GT330m	5400	2500
Surface RT	370	100
Nokia Lumia 920	286	75
Xbox 360	171	43

Conclusion

You have three options here to build a cross-platform game running fine everywhere.  You can take the lowest value (43 sprites here) and never put more that this number of sprites on the screen to be sure to have 60 FPS everywhere on Xbox 360, Windows Phone 8, Surface RT and Windows 8 machine running HD4000 GPUs. Of course, you need to run the same benchmark on your other targeted platforms: Android tablets & phone, iOS tablets & phone, and so on to find the optimal magic number. Taking this first option is the easiest one but it’s a pity that desktop machines are under-exploited. A second option is then to build two versions of the game (like two versions of your websites): one for mobile and one for phone. You can just adjust the graphical complexity based on the performance of each platform. Or you can also take the optimal number on mobile to target 30 FPS and you will be sure to run @60 FPS on desktop. The idea is then to target 30 FPS on mobile/tablets and 60 on desktop. Lastly, the third and last option is to build a game engine that will itself dynamically adjust the complexity of the graphical engine based on the performance detected. This is something that some game studios are frequently doing on PC for instance. It requires more work, for sure, and you need also to decide the kind of assets that will be displayed or not in your game without impacting the global gameplay. But today, doing this kind of benchmark is really critical if you’re targeting the mobile markets. Use a “mobile first” approach, otherwise you will get into trouble optimizing the performance for these lower GPUs.

Impact of the Canvas Resolution

This time we’re going to always display the same number of sprites (500) but we’re going to vary the resolution of the canvas: 320×200, 640×480, 1024×768 and 1920×1080. I was convinced on my side that increasing the resolution would lower the average FPS for sure, even just to display some animated sprites. Well, let’s check that via this link: launch various canvas resolutions series.

Some Results and Analysis

Well, look at the result in IE10 on my machine: On my machine, the various resolutions had zero impact on the average frame rate! This is also confirmed on my mobile WP8 devices and on Xbox 360. Only Chrome has a lower average FPS only in 320×200, for an unknown reason. This doesn’t seem logical. But it’s probably because of a specific optimization I’m not aware of. Firefox and all other rendering/JavaScript engines I’ve been testing on several devices provide the same result: resolution has no impact on the global performance in my scenario. Please note also that this doesn’t mean you shouldn’t take care of the resolution of your canvas for your global performance. This just means that for sprite animations, and with this specific benchmark, the resolution has no impact. The GPU seems to take the load without any problem even on mobile platforms. I was very surprised by these first results so I’ve done further tests and benchmarks. To double check that GPUs weren’t saturated, I had the following idea. I’ve got a laptop machine embedding two GPUs, a Sony Vaio Z13. It has a nVidia GT330m and an Intel HD3000 integrated GPU. The Intel HD3000 contains up to 12 scalar 128-bit execution units where the nVidia GT330m contains 48 128-bit execution units. There are also tools that help to check the load of the GPU. For instance, GPU-Z can provide you the GPU Load in realtime. I’ve then run the last benchmark series first on the nVidia GPU with GPU-Z opened on the “Sensors” tab to verify the GPU load: 15% at 320×200, 30% at 640×480, 25% at 1024×768, 55% at 1920×1080 (don’t ask me why it’s lower in 1024 than in 640, I really don’t know). Then, I’ve re-run the same benchmark on the HD3000: 30% at 320×200, 40% at 640×480, 62% at 1024×768 & 100% at 1920×1080. Logically, this time, the average FPS drop from 60 FPS in 320×200 –> 1024×768 to 55 FPS in 1080p. Being below 60 FPS is because we’re falling into a GPU limited scenario. You can’t do anything else than lowering the rendering resolution and/or display less sprites, etc. By the way, I’ve done some testing and using “hardware scaling”, as described in this article: Unleash the power of HTML 5 Canvas for gaming, has almost no impact on the global average FPS as this operation seems to be done with no effort by today’s GPU even on mobile. This is then a really good option to keep in mind. If the frame rate is too low due to a too high GPU load, try to render your canvas in a lower resolution and stretch it full screen using this hardware scaling method. There could be cases where the lost of frames will be due to CPU limited scenarios. To monitor that, you need profiler tools like the one embedded in most recent browsers. You will then see which parts of your code you should try to work on to regain some FPS. You need to avoid that the GPU is waiting too long for the CPU to do its job. Some cases could be optimized using HTML5 Web Workers, for instance. But I’ve found also some very specific cases where dropping under 60 FPS was due to more complex reasons. Here is one of them: GPU-Z shows a GPU load around 50% but the FPS is below 60 FPS. Using the F12 tool included in IE10 to profile the code shows the following result:  It will show that you’re only limited by the drawImage function. drawImage is a native function handled by the browser. It’s probably mixing CPU usage and GPU usage. You can’t optimize this part at this level in JavaScript. So, you just need to deal with it!

Going Even Further

You can have a look at the set of tools available in the Windows Performance Toolkit and read the following methodology: Measuring Browser Performance with the Windows Performance Tools. You’re also probably wondering why we have such slight FPS drops even with a relatively constant 60 average FPS: With David Catuhe, we had a precise idea why. But again it’s better to be able to verify it. I’ve then used the following methodology:

• Create a blank Visual Studio 2012 Windows Store App project. Indeed, as Windows 8 is using IE10 to run Windows Store App in HTML5, I had just to copy/paste the code of one of the benchmarks and I had a Windows Store app ready to be analyzed by Visual Studio. :) You can download this test project here if you’d like: HTML5PotatoesModernApp.zip
• I’ve setup a unique bench displaying 1000 sprites and gone into “Analyze” –> “JavaScript Analysis” –> “UI Responsiveness” –> “Launch Startup Project”:

After some processing, you will obtain such results: We can see that this is the GC which is responsible for the drop of some frames. I remember reading interesting details about GC in this article: Are We Fast Yet? associated with this other benchmark: HTML5-Benchmark.

Conclusion

You know what I really like about this whole story? Even with a high level layer like HTML5, understanding the targeted architectures will always help you to optimize your code for your HTML5 games. This is really what you should keep in mind to build your games that should scale across all HTML5 compatible devices. It’s now possible to have a unique code base to run everywhere. But layout & JavaScript compatibilities is just a small part of the story. You need to know which devices you’re going to target, understand their limitations & GPU characteristics and finally benchmark all of them to take the appropriate design decisions for your games. I really hope that this article and our benchmark framework will help you in your future HTML5 games. We will soon work on similar articles focused on other topics important for HTML5 games.  This article is part of the HTML5 tech series from the Internet Explorer team. Try-out the concepts in this article with free virtual machines @ http://modern.IE.

Frequently Asked Questions about HTML5 Gaming Benchmarking and Sprite Animations