Missing from the roster of browsers tested were Safari and Opera (both of which claim to be the "fastest"), among others.
Microsoft used 25 popular Web sites, based on a comScore rankings, for its testing. IE 8 was the fastest browser at 12 sites, followed by Chrome (nine sites) and Firefox (four sites), according to the paper, "Measuring Browser Performance: Understanding Issues in Benchmarking and Performance Analysis" (downloadable here).
The study results aren't new; Microsoft executives have been citing them of late, including Mike Nash, Microsoft's corporate vice president for Windows product management, in an interview conducted last week. What's new is that Microsoft describes its benchmarking methodology in simple terms for IT pros who want to replicate the tests.
Microsoft takes the view that measuring browser component speeds -- such as measuring JavaScript rendering via SunSpider, Celtic Kane, V8 and other such tools -- is not very helpful in assessing user experience. Instead, the Microsoft team focused on measuring overall page loading speeds.
Microsoft's IE lab set up media player software to record the page loads and times. The team did not rely on the browser itself to report page load completion. Instead, the team used recorded visual cues. A video produced by Microsoft dramatizes this testing, which used media player software that's capable of "approximately 1/30th of a second" recording accuracy.
The use of visual cues seems a bit odd. In addition, Microsoft in some cases relied on additional information to determine the actual page load completion times. "For sites on which specific visual cues aren't sufficient, we use a combination of visual cues and the ability to interact with content on a page to determine if a page is 'done,'" the paper explains.
Web sites that used AJAX technology to report page load completion introduced complexity, according to the paper. In those cases, Microsoft relied on the visual cues.
IT pros might have difficulty trying to replicate Microsoft's benchmark tests, especially when relying on visual cues to determine page load times. Moreover, the testing team used similar test hardware, loaded the sites beforehand to cache content and took other measures requiring lots of work to replicate.
Microsoft also tested the browsers with at most three add-ons, including Adobe Flash, Microsoft Silverlight and Windows Media Player. While that's a typical use scenario, performance test results using those add-ons are not reported in the white paper.
Those obsessed with browser speed differences of milliseconds doubtless may criticize the sample number of Web sites used in the study. Testing 25 Web sites may not be enough to truly proclaim one browser faster than another, statistically speaking.
Kristen Stewart And Dakota Fanning: New ‘Twilight’ BFFs?
‘The Uninvited’: I’m Not There, By Kurt Loder
Windows 7 Is ‘Rescuing’ Apps, Microsoft Says
