HTML5 Benchmark Scores: NME RunnerMark

If you are on Twitter, you may have seen links to RunnerMark, a benchmark that tests rendering performance of Adobe AIR and NME.

We have been improving the performance in the (beta) HTML5 target for NME, so I am happy to share some of the results. These improvements will be released with the next version of NME.

All of the desktop benchmarks were done on a Sony Vaio T with Windows 8. It is an entry level model with an i3-3217U (1.80 GHz) CPU and 4 GB of RAM.

As a baseline, the test run using the NME C++ target for the Windows desktop, then it was run three times in each browser and averaged.

RunnerMark

Desktop

  • Reference (C++) — 6335
  • Firefox 16 — 1637
  • Internet Explorer 10 — 1717
  • Chrome 23 — 1288

Mobile

  • Dev Alpha B (BlackBerry 10) — 332
  • Galaxy S (Android 2.3) — 140
  • iPod Touch (4th Gen, iOS 5) — 260

RunnerMark (30 FPS)

Desktop

  • Reference (C++) — 14,348
  • Firefox 16 — 3596
  • Internet Explorer 10 — 3509
  • Chrome 23 — 2292

Mobile

  • Dev Alpha B (BlackBerry 10) — 377
  • Galaxy S (Android 2.3) — 140
  • iPod Touch (4th Gen, iOS 5) — 290

Although the browser cannot achieve the same performance as native C++, I am impressed by how canvas acceleration has advanced. For example, Safari on Windows only gets a score of 150, which is similar to how most browsers used to perform.

Even on mobile, the performance of HTML5 has advanced enough that creating a more complex browser game could be possible. We hope that NME will be a useful tool, and that we will be able to continue to advance HTML5 support as Flash Player support in the browser, even on the desktop, is no longer ubiquitous.

NME can publish this benchmark for iOS, Android and BlackBerry, in addition to Flash Player, Windows, Mac, Linux and HTML5.

  • Franco Ponticelli

    This is really interesting but it would be even more if we could see the results compiling using the current version of NME vs the upcoming version.

  • That is true.

    Actually, I just remembered that Philippe posted a version with HTML5, before the current improvements: http://philippe.elsass.me/lab/RunnerMark/

    Testing again, I scored 210 on BlackBerry 10 Dev Alpha, Android had an error and the iPod Touch scored 100, so up to 2.5 times better performance on mobile, and possibly much more on the desktop.

    Also, the margin is so close that this makes a big difference in the feasibility of HTML5… whether you can create something like a platformer or if you have to do something minimal.

  • Nicolas Cannasse

    I’m a bit curious about Chrome score, which I supposed should have been higher. Any insights ?

  • Philippe

    Chrome is a bit disappointing – it’s features-packed but overal performance isn’t impressive.
    Can someone test Safari Mac? it should be even worse.

  • Danny Marcowitz

    How does the performance compare to CreateJS? specifically for mobile devices.

  • Yep, that would be interesting to know!

    I found a CreateJS benchmark, and just ported it for NME so we can compare:

    CreateJS — http://www.joshuagranick.com/examples/penguin/createjs/
    NME — http://www.joshuagranick.com/examples/penguin/nme/

  • Daniel Constantinescu

    Hello. Don’t know why my first post disappeared; will post again:

    First of all, nice benchmark!

    I ran some tests on RunnerMark 30 FPS (don’t know what the first one should do but the results seem to be limited to 300):

    Tablet:
    • Ipad3 – iOS 6.0.1 – Safari 741 / Chrome 588
    • Ipad2 – iOS 6.0 – Safari 781 / Chrome 602
    • Ipad1 – iOS 5.0.1 – Safari 325
    • Samsung – Android 4.0.3 – Chrome 210
    • Surface – Windows 8 – IE10 654

    PC – Window 7:
    • Firefox 16 – 3445
    • Chrome 23 – 2821
    • IE9 – 2534
    • Opera 12.10- 431
    • Safari 5.1.7 – 475

    Mobile:
    • LG Optimus 2X – Android 2.3.4 – Dolphin – 110 / Firefox – 70 / Opera 130

    Secondly, the first time i tested NME targeting HTML5 i ran into 2 problems:

    – the MouseUp and MouseMove events don’t fire outside the stage;

    – there is a bug on scaleX & scaleY.

    Do you have any idea if these were fixed in the new version, or should I report them?

    I am researching the possibility to switch from CreateJS to NME for multi-platform content development, but the current release its kind of disappointing.

    I have an example online: http://www.daniel-constantines
    It is a drag&drop with change scale on press/release + URLLoader test.

  • Hi all! On the BlackBerry Playbook OS 2.1with the default browser the RunnerMark scores 120 and 447 for the 30FPS. Odd result for the former but good for the later.
    Cheers.

  • RunnerMark is set up to award 10 points for each FPS then to give an extra point for every extra enemy.

    So this would mean PlayBook only got 12 FPS total when timed for 60 FPS, but for 30 FPS it had a stable frame rate and 147 additional enemy instances.

    I just got a score of 884 running native. All in all, pretty good, I think.

  • Danny Marcowitz

    Thanks! 🙂