Microsoft Does Its Own Browser Benchmarking

By  |  Thursday, March 12, 2009 at 11:16 am

Internet Explorer 8In recent months, the hottest topic in the world of Web browsing has been speed. Apple says its beta version of Safari 4 is the world’s fastest browser. The first thing Google tells you about Chrome is that it’s “faster.” Better performance is a key feature in Mozilla’s upcoming Firefox 3.5. Opera says that its alpha of Opera 10 is “30% faster.”

And Microsoft? Well, mostly it’s had to contend with coverage like this story that reports that Safari is forty-two times faster than Internet Explorer 7 and six times faster than IE 8.

Today, the company is fighting back. It’s done its own speed benchmarks and has created a video about them and published a white paper about browser benchmarking. Here’s a stunner: It’s not concluding that IE is a horribly slow browser. In fact, it says that Internet Explorer 8 is not only competitive, but loads many of the world’s most popular Web sites faster than Firefox 3.0 or Chrome 1.0. I met with IE general manager Dean Hachamovitch last week, and he made the same claim.

Why the disparity? On a high level, it boils down to one thing: Nearly all discussion of browser speed centers on performance of JavaScript code, most often as measured by the SunSpider benchmark. And much of the work that browser companies have done to ramp up speed involves optimizing their JavaScript engines. IE 8 performs better in such benchmarks than IE 7 (as witness its better results in the Safari 4 story I link to above) but still lags other browsers.

Microsoft says–correctly–that browser speed is about more than JavaScript. Its own benchmarking involved timing how long it takes IE 8, Firefox 3.0, and Chrome 1.0 to load the 25 largest Web sites, a process that tests not only JavaScript but also how long it takes a browser to render the page and other factors. Judged this way, IE looks competitive. Here’s an image from the video:

Internet Explorer Speed Tests

As with benchmarks of any sort, it’s possible to spend all day poking holes in Microsoft’s tests if you feel like it. It used the shipping version of Firefox, not the speedier beta, and didn’t include Safari or Chrome at all. And while simply loading a major site’s home page is certainly one valid way to test a major aspect of Web performance, it too isn’t definitive. JavaScript becomes more important as you do tasks in sophisticated Web-based apps–for instance, I’d love to see a well-done benchmark involving how long it takes to perform major e-mail tasks in Gmail in major browsers. And I suspect that some knowledgeable folks will argue that Microsoft is downplaying JavaScript’s importance simply because IE’s JavaScript is still slow.

Then there’s the fact that while most chatter about Safari’s speed involves JavaScript, Apple has also published results for the i-Bench HTML benchmark. They too make Safari look zippy and both IE 7 and IE 8 look sluggish. I don’t know enough about i-Bench (I believe Apple uses a custom version based on a discontinued Veritest benchmark) to have an opinion about how well-rounded it is.

The Microsoft video ends with a line about staying tuned for more info. I’m guessing Microsoft will contend that there’s more to fast browsing than can be measured by any benchmark–it’s also about how quickly you can perform typical tasks, how easy it is to wrangle tabs and find stuff online, how well the browser recovers from problems, and the like. All valid points.

If nothing else, I think Microsoft is performing a service–a self-serving one in this case, but a service nonetheless–by reminding us that all benchmarks are inherently limited. Even the best ones only test certain scenarios, and no matter how good SunSpider is, it doesn’t claim to be a well-rounded browser benchmark–just a JavaScript one. Rather than declaring one browser to be faster or slower than another, period, it’s simply more accurate to specify that it was faster or slower in a specific test.

Once IE 8, Safari 4, Firefox 3.5, Opera 10, and whatever version of Chrome comes next are all final, I’d love to see someone do a really comprehensive suite of speed tests on all of them. Such a project would incorporate both JavaScript tests like SunSpider and page-load tests like the ones Microsoft did–and probably some additional ones, too. And if it were done right, it would tell us more about how the browsers stack up than anyone’s tests to date…

 
11 Comments


Read more: , ,

9 Comments For This Post

  1. tom b Says:

    It’s totally pointless to even RUN these benchmarks; IE and Chrome are only available on Windows so, already, they limit their audience. Web designers should check first on Firefox, the most widely available quality browser; then on iPhone Safari, the best mobile browser. Then, and only then, should they be thinking about the legacy (IE) or the “not even version 1.0″ browsers, like Chrome.

  2. Marc Says:

    Of course it will be skewed to Microsoft’s benefit, just like the Firefox one is skewed to theirs, and the same for Google and Apple. Microsoft could have included a few extensions for Firefox, just to emulate what the average user is running.

    Of course speed doesn’t matter. Perceived speed does. IE8 shows a half painted tab that doesn’t respond to any mouse tabs, just for a split second while loading a new tab. Chrome hides the entire tab creation process, and shows the new tab in a smooth manner once all the background work is done, and so appears quicker. Which one actually is I don’t know, but if Chrome, which its limited feature set, isn’t significantly quicker, then I’d be disappointed.

    Even if IE8 is quicker, will users want to sacrifice their favourite extensions for a few milliseconds? Browser extensions may slow Firefox down, but they also make certain things quicker.

  3. Marc Says:

    tom b: You mean IE and Chrome limited their audience to 95% of PC users? Er… yeah…

  4. jdg Says:

    huh? Limit their audience? You’re funny. FF, IE and Chrome run on 94% of the computer audience. Mac’s only account for 6% (or less) of computers so it’s not worth developing for it. To bad for you fanboy.

  5. larryv Says:

    jdg: Last I checked, Firefox runs on Mac OS X and Linux. That makes more or less 100%.

    Also, if your website doesn’t run on Safari or Camino, which are standards-compliant browsers, you’ve probably messed up something pretty badly.

  6. larryv Says:

    tom b: Chrome is at 1.0.154.48, by the way. While a newcomer it may be, beta it isn’t.

  7. Jeff Says:

    Let Microsoft spin this all they want. The fact is that based on my experience, it’s the slowest browser I’ve tried. It’s amazing to me how much faster Chrome is than IE.

  8. Stilgar Says:

    As someone who’s a developer on a Web service, I just want to chime in and say that everyone on my team hates IE. It’s slow (at javascript) and horribly compliant with standards compared to the other browsers. There are some who believe that Microsoft deliberately keeps IE crappy because MS has yet to develop a strong Web service on their own (they bought Hotmail).

    They also do other things that make it a PITA to develop Web apps, like having different versions of IIS for their desktop OSes than their server OSes. The IIS connection limits imposed by XP and Vista are also real fun when you’re running a copy of your code on your dev machine and you hit the connection limit after testing in two or three browsers. I could go on, but I don’t want to sound too much like John Dvorak. :)

  9. Victor Ellis Says:

    As an aged user, I would like to say that having installed IE 8, I find it very slow to start up (Firefox starts up much faster). Any ideas how I can get rid of 8 and revert to 7 ??

    V Ellis

2 Trackbacks For This Post

  1. Chrome: Faster Than Safari? | Technologizer Says:

    [...] slower). The test only tells you so much about browser performance. ¬†And maybe Google, like Microsoft, is saying that “fast” is about more than traditional speed [...]

  2. Microsoft Previews the Revamped Internet Explorer 9 Platform Says:

    [...] been busy souping up their JavaScript engines to run Web apps as fast as possible, Microsoft has contended that JavaScript performance isn’t that important. That position may stem in part from the fact that IE8 runs JavaScript at a crawl compared to the [...]