Tests: Firefox 3.1 JavaScript outpaced by Safari 4, Google Chrome

If the race to be the biggest alternative Windows Web browser to IE8 comes down to JavaScript interpreter speed, the latest Firefox will have some catching up to do.

Internet enthusiasts today are getting their first glimpse of the TraceMonkey JavaScript interpreter that Mozilla claims will be one of the principal reasons to own and use Firefox 3.5 -- what the latest Firefox will inevitably be called once the numerology gets sorted out. A fresh round of comprehensive Betanews tests Thursday afternoon indicate that Firefox 3.1 Beta 3 will demonstrate close to eight times the general JavaScript calculation and rendering performance of Microsoft Internet Explorer 7 -- a clear performance gain.

But as we were able to verify Thursday, it may not be nearly enough to earn the latest Mozilla beta the performance lead. In a battery of 20 calculation and rendering tests assembled from four different developers' benchmarks and benchmark suites, Apple's Safari 4 beta for Windows perceptibly and appreciably outperformed every other test browser on our virtual machine.

There is no widely accepted industry standard benchmark suite for Web browser performance just yet, but there are quite a few potential nominees. Over the last few weeks, we've been discovering some that were created by independent developers and shared with the rest of the world through Web sites.

In an effort to produce a fair and unbiased rendering of JavaScript engines' relative performance -- something closer to what you'd expect if we were comparing six different grades of CPU to one another -- we gathered a group of respectable benchmarks that appear to cover the gamut for different types of workloads: cryptography, data handling, DOM handling, AJAX declarations, parsing regular string expressions, string manipulation, mathematics, rendering, compliance with standards. And we re-cleaned and re-assembled our Windows Vista SP1-based virtual machine -- again, not the best candidate for a "super-speedway," but a plausible venue for an unencumbered test of how the latest experimental beta, alpha, and release candidate versions of major Web browsers perform in comparison to one another. (Though Google lifted the "beta" designation from its Chrome browser, its development is continually tested and its clients are automatically updated to reflect results from those tests, so we included the latest Chrome edition with the other brands.)

Firefox 3.1 Beta 3 was released to the general public yesterday. We spent a good part of yesterday comparing that release to the first Apple Safari 4 beta for Windows -- which performed astonishingly well in our initial tests -- as well as Google Chrome, Microsoft Internet Explorer 8 Release Candidate 1, and in response to many of our readers' requests, the latest alpha edition of Opera 10.

Knowing that our test venue would be an arguably slow virtual system, our goal here was to test relative performance -- to illuminate the scale of performance differences that all users would likely see on their own, probably faster systems. The way we chose to represent relative performance is by creating a base index -- something we already know to be slow and that we expect will be outpaced.

So we ran our series of tests on the most recently updated version of Internet Explorer 7, prior to upgrading our test VM once again to run IE8 RC1. For each test, we gave IE7 a score of 1.0, to serve as our baseline. We then used four benchmark "suites" for which every final result is rendered relative to that 1.0; so for example, a score of 2.3 on a speed test would mean the browser was 230% the speed of IE7. And for the Acid3 benchmark, which tests compliance, an index score of 8.0 for a browser would mean its rendering score would be eight times better than for IE7.

Our test suites include the three benchmarks we used last week to demonstrate Safari 4, including HowToCreate's CSS rendering test (since the rendering times can vary, we averaged the times for five renders for each browser), developer Sean Patrick Kane's octathlon which renders a cumulative speed after a battery of eight tests, and the Web Standards Project's Acid3 compliance test, which renders its score as a percentage.

To these, we added one more suite: the SunSpider JavaScript Benchmark, a pentathlon of real-world simulations that include elements like AES, MD5, and SHA1 cryptography puzzles; ray-tracing formulas; and handling of JavaScript Object Notation (JSON) expressions. With this and the Celtic Kane battery, we'll be able to find out where any browser excels and where it fails. Each of our four suites produced a cumulative index score, which we then counted toward 25% of the final score.

The final index scores for our tests indicate one thing conclusively: The performance differences between all the major Windows-based Web browsers, compared to one another, are clearly perceptible. No test browser is close to any other; in everyday use, the differences in JavaScript rendering speed are enough to be seen with the naked eye.

And there is a clear leader in that department: Apple Safari 4.

Microsoft can say that its IE8 candidate has more than double the performance of its predecessor, and it can probably smile at that. IE8 RC1's index score is 2.19. However, the latest Safari 4 beta's score under the same conditions is 14.39, blowing away anything Microsoft produces. We expected more from the Opera 10 alpha, given what that browser's faithful followers told us. But with an index score of 5.38, Opera's performance right now is bested by Firefox 3.1 Beta 3, which finished with a 7.85. But Mozilla will have to settle for the bronze for now. Google Chrome is the one vying for the lead, with a second-place score of 11.62.

http://pic.leech.it/i/b2c41/c779e78f2948.jpg

What were the browsers' strong points? Safari 4 scored well in the HowToCreate box rendering experiment. In the Celtic Kane battery, Safari shined with regular expressions, handling the Document Object Model, and at parsing RegEx expressions -- on that one part alone, Safari scored a 13.94 versus IE7. Its AJAX parser also proved strong. But the SunSpider benchmark was able to highlight some even brighter points for Safari, including just the way it manages control of scripts. In that heat, it scored a 90.94 versus IE7. And scoring 100% on the Acid3 certainly helps.

Chrome has clear strengths and weaknesses. It was not the fastest at the HowToCreate rendering experiment, scoring only 4.73 against IE7. Its 79% score on the Acid3 set it back. But it managed to outperform even Safari 4 in the SunSpider suite, with a 33.36 overall versus 32.81 for Safari. The control flow heat alone produced a 148.37 for Chrome versus IE7, which is testament to how well that browser's unique system architecture pays off.

The browser that will be Firefox 3.5 has some ground to make up. It actually performed rather poorly with the HowToCreate rendering test, actually loading the test slower than IE7 (0.93) and producing a rendering score of only 1.83. Its 93% score on the Acid3 is the best we've ever seen from a Mozilla browser. However, if this were a math quiz, Firefox 3.1 Beta 3 would win hands down, and here is where we're seeing that "orders of magnitude" difference we were promised for TraceMonkey: In the math heat of SunSpider, Firefox pulled off a 17.24, compared to 8.97 for Chrome and 6.93 for Safari.

Since last December, Opera has waved its flag high atop the mountain that is the 100% Acid3 score, and we saw it again yesterday. Opera did win the day on the HowToCreate rendering test, scoring a 5.03 -- the highest of the test browsers, although that score was offset by its slower loading speed. But Opera couldn't keep up in the octathlon and pentathlon, scoring only 3.05 and 6.67, respectively -- the lowest scores for any non-Microsoft browser.

Internet Explorer 8 was never expected to be a breakthrough performer. What we did not expect was a rendering score that is actually slower than IE7 (0.83). IE8 put in an appreciable 5.05 on the SunSpider suite and a less-than-appreciable 1.67 on the Celtic Kane suite. But scoring merely 20% on the Acid3, these days, may not be forgivable.

During the era of the so-called "browser wars," the only real matchup was between Netscape Navigator and Microsoft Internet Explorer. And that wasn't even really a battle on equal terms, but rather one between performance and functionality versus convenience. IE won that battle. So the fact that we are actually comparing Web browsers' performance on an equal scale today is symbolic of the fact that this market, for the very first time in its history, may be becoming truly competitive.

http://pic.leech.it/i/f7bb2/5b19d3e02947.jpg

http://www.betanews.com/article/Test...e/1236959567/2