Has Intel Really Beaten ARM? EE Times Community Spotlights Benchmarks' Vagaries

SAN FRANCISCO, July 12, 2013 /PRNewswire/ -- The electronics industry has been abuzz around new AnTuTu benchmark results and a recent ABI Research report claiming, "Intel apps processor outperforms Nvidia, Qualcomm, and Samsung."  Has Intel surpassed the entire ARM ecosystem in mobile processors for the all-important high-end smartphone segment?

(Logo: http://photos.prnewswire.com/prnh/20130712/SF46383LOGO)

In response to the report, EE Times member Jim McGregor investigated further and compiled a variety of benchmark information from tech reviewers, benchmarking organizations and other industry resources. In particular, he looked at processors from Samsung, Intel and Qualcomm and, in effect, debunked the entire report, showing that ARM-based processors came out on top. At the same time, he pointed to the nuances and traps of processor benchmarking in general.

"Evaluating current mobile processors is challenging because these processors, known as systems-on-chips (SoCs), are complex systems of heterogeneous processing elements combined with memory, I/O, high-speed networks, communications modems and a host of other dedicated system functions," he wrote.

"Integration of the processors into mobile devices further complicates any evaluation because the overall performance and efficiency of these processors is impacted by the other system components. As a result, the industry turns to benchmarks to compare processors and devices. Unfortunately, mobile benchmarks are plagued by many issues and also fall short of providing an accurate evaluation."

Despite what seemed a fairly comprehensive conclusion, the EE Times community took McGregor's analysis, and benchmarking in general, to task, with an emphasis on power consumption:

eewiz: "...this analysis kinda sidestep[s] the issue of power consumption. It was not the processor's computational speed that was in question. It was that Intel CPU had more or less the same performance at HALF the current drain/power."

jaybus0: "Well, long calls affecting battery life is much more a function of the RF chipset efficiency and software control of transmit levels, etc. I don't see how it would fit into a comparison of digital SoCs."

rob12340: "Also, what OS was running on each platform to carry out these tests, since a highly optimized OS can make these benchmark tests show amazing performance on a slow processor vs. poor results on a badly ported OS running on a considerably faster processor."   

David.Robinson: "The ABI article is clearly more about the current draw than about raw performance.  So, while I agree that they could have done a better job by averaging multiple benchmarks, I think the point of the article is that Intel seems to have finally conquered what analysts have considered its 'Achilles' heel': power consumption."   

Questioner: "The RAM scores seem highly unusual.  Is there some kind of "cheating" going on with AnTuTu?"  

On the topic of compilers:

Performance guy: "What's wrong with Intel getting ahead using better compiler technology?"

To which Jeff Bier responded: "Nothing, if we're talking about making real applications run faster. But that's not what we're talking about here.

What we're talking about here is the compiler removing portions of the benchmark, contrary to the intent of the benchmark.  As a result, the benchmark results become meaningless."   

The discussion continues to heat up on a very contentious issue. Clearly, all benchmarks should be questioned and none used exclusively; and recent headlines were more sensational than truthful.

To see the latest comments and join in the discussion, go to: Has Intel really beaten ARM?           

About UBM Tech

UBM Tech is a global media business that brings together the world's technology industry through live events and online properties.  Its community-focused media and events provide expertly curated content along with user-generated content and peer-to-peer engagement opportunities through its proprietary, award-winning DeusM community platform. UBM Tech's brands include EE Times, Interop, Black Hat, InformationWeek, Game Developer Conference, CRN, and DesignCon. The company's products include research, education, training, and data services that accelerate decision making for technology buyers.  UBM Tech also offers a full range of marketing services based on its content and technology market expertise, including custom events, content marketing solutions, community development and demand generation programs. UBM Tech is a part of UBM (UBM.L), a global provider of media and information services with a market capitalization of more than $2.5 billion.

Available Topic Expert(s): For information on the listed expert(s), click appropriate link.
Patrick Mannion
https://profnet.prnewswire.com/Subscriber/ExpertProfile.aspx?ei=110107

For more information on UBM Tech please contact:
Felicia Hamerman, VP, Marketing, Electronics
T: 516.562.5652, E: felicia.hamerman@ubm.com

SOURCE UBM Tech