An analysis of Internet browsers by Top Ten Reviews places Google Chrome at the head of the list. The analysis shows, however, that it's a real horse race between the top five: Chrome, Firefox, Internet Explorer, Opera and Safari. Here are some of my observations based on this analysis and personal tests and experience:
Change after new releases: All of the browser developers are working hard to improve their products. With each new browser version release, the competitive landscape can change. For example, Internet Explorer had recently lagged in speed due to the lack of graphics hardware acceleration. Based on my tests, they're now the fastest running HTML5 Canvas animations.
Graphics hardware acceleration: GPU (Graphics Processing Unit) hardware acceleration is an important factor in browser speed. A browser displaying video or animation with GPU acceleration will outperform a browser without it by a factor of 3-4. That's right, 300-400%. A computer using just the CPU to manipulate graphics displays simply can't keep up with one using the combination of CPU and GPU. Hardware acceleration is a fairly complex topic. Basically, the browser offloads calculations from the CPU to the GPU. If you're interesting in some of the details, take a look at this discussion of hardware acceleration from the Chromium Projects.
Mobile lags desktop: Especially in speed, the mobile browsers lag desktop versions. Mobile devices simply don't have the computing power to match desktop devices.