In recent months, the hottest topic in the world of Web browsing has been speed. Apple says its beta version of Safari 4 is the world’s fastest browser. The first thing Google tells you about Chrome is that it’s “faster.” Better performance is a key feature in Mozilla’s upcoming Firefox 3.5. Opera says that its alpha of Opera 10 is “30% faster.”

And Microsoft? Well, mostly it’s had to contend with coverage like this story that reports that Safari is forty-two times faster than Internet Explorer 7 and six times faster than IE 8.

Today, the company is fighting back. It’s done its own speed benchmarks and has created a video about them and published a white paper about browser benchmarking. Here’s a stunner: It’s not concluding that IE is a horribly slow browser. In fact, it says that Internet Explorer 8 is not only competitive, but loads many of the world’s most popular Web sites faster than Firefox 3.0 or Chrome 1.0. I met with IE general manager Dean Hachamovitch last week, and he made the same claim.

Why the disparity? On a high level, it boils down to one thing: Nearly all discussion of browser speed centers on performance of JavaScript code, most often as measured by the SunSpider benchmark. And much of the work that browser companies have done to ramp up speed involves optimizing their JavaScript engines. IE 8 performs better in such benchmarks than IE 7 (as witness its better results in the Safari 4 story I link to above) but still lags other browsers.

Microsoft says–correctly–that browser speed is about more than JavaScript. Its own benchmarking involved timing how long it takes IE 8, Firefox 3.0, and Chrome 1.0 to load the 25 largest Web sites, a process that tests not only JavaScript but also how long it takes a browser to render the page and other factors. Judged this way, IE looks competitive. Here’s an image from the video:

As with benchmarks of any sort, it’s possible to spend all day poking holes in Microsoft’s tests if you feel like it. It used the shipping version of Firefox, not the speedier beta, and didn’t include Safari or Chrome at all. And while simply loading a major site’s home page is certainly one valid way to test a major aspect of Web performance, it too isn’t definitive. JavaScript becomes more important as you do tasks in sophisticated Web-based apps–for instance, I’d love to see a well-done benchmark involving how long it takes to perform major e-mail tasks in Gmail in major browsers. And I suspect that some knowledgeable folks will argue that Microsoft is downplaying JavaScript’s importance simply because IE’s JavaScript is still slow.

Then there’s the fact that while most chatter about Safari’s speed involves JavaScript, Apple has also published results for the i-Bench HTML benchmark. They too make Safari look zippy and both IE 7 and IE 8 look sluggish. I don’t know enough about i-Bench (I believe Apple uses a custom version based on a discontinued Veritest benchmark) to have an opinion about how well-rounded it is.

The Microsoft video ends with a line about staying tuned for more info. I’m guessing Microsoft will contend that there’s more to fast browsing than can be measured by any benchmark–it’s also about how quickly you can perform typical tasks, how easy it is to wrangle tabs and find stuff online, how well the browser recovers from problems, and the like. All valid points.

If nothing else, I think Microsoft is performing a service–a self-serving one in this case, but a service nonetheless–by reminding us that all benchmarks are inherently limited. Even the best ones only test certain scenarios, and no matter how good SunSpider is, it doesn’t claim to be a well-rounded browser benchmark–just a JavaScript one. Rather than declaring one browser to be faster or slower than another, period, it’s simply more accurate to specify that it was faster or slower in a specific test.

Once IE 8, Safari 4, Firefox 3.5, Opera 10, and whatever version of Chrome comes next are all final, I’d love to see someone do a really comprehensive suite of speed tests on all of them. Such a project would incorporate both JavaScript tests like SunSpider and page-load tests like the ones Microsoft did–and probably some additional ones, too. And if it were done right, it would tell us more about how the browsers stack up than anyone’s tests to date…

 
11 Comments


Read more: Browsers, Internet Explorer, Microsoft