In an industry that gets off on throwing obscure benchmarks at buyers ( “pixel fill rate,” anyone?), laptop battery life is one of the easiest to understand.
It’s also long been one of the least useful, critics charge, due to the industry’s deceptive use of the dominant standard, the MobileMark benchmark created by the Business Applications Performance Corp. (BAPCo), an industry consortium whose members include Intel Corp., Dell Inc., Hewlett-Packard Co., Advanced Micro Devices Inc. and others.
“Everyone in the industry knows this benchmark is wildly optimistic and that the actual battery life you’ll get is often less than half what MobileMark suggests,” wrote analyst Rob Enderle last month. “This is because MobileMark measures battery life much like you might measure gas mileage if you started the car, put it in neutral, and coasted down a long hill.”
The latest MobileMark 2007 report measures laptop battery life under three scenarios: reading a document, watching a DVD movie, and doing a “representative” mix of productivity tasks, such as reading and composing documents, editing photos and encoding Flash videos (see page 13 of white paper).
But rather than using an average time based on all three measures, BAPCo designates its third scenario as the way most people use their laptop.
There are several problems with this, according to critics. First, doubters such as AMD say that the productivity test assumes that the notebook is idle 90% or more of the time.
Second, MobileMark’s “productivity” scenario assumes that users, when active, are using only software such as Adobe Photoshop or Microsoft Office. They don’t test usage of music or video applications such as iTunes or Windows Media Player, games or Web browsing. The test also assumes that Wi-Fi is turned off.
That seems particularly unjustified today, since netbooks are touted as on-the-go, Web-centric devices, or high-definition video-capable machines.
Finally, MobileMark 2007 allows PC vendors to set their laptop screen brightness at the lowest possible setting, provided it is no lower than 60 nits (a nit is a measure of brightness. The problem, again pointed out by AMD, is that 60 nits is quite dim, being only about one-fifth of most notebook PCs’ maximum screen brightness.
According to an informal reader poll at Neowin.net, a Windows community site, fewer than 15% of respondents run their notebooks that dim.
BAPCo defended its MobileMark benchmarks. “The content of BAPCo benchmarks are vigorously debated and cooperatively developed by BAPCo members according to a long and rigorous process,” the company said in an e-mailed statement. “As is the case with all BAPCo benchmarks, MobileMark 2007 was approved by BAPCo according to a democratic voting process similar to ones used by most industry work groups.”
Despite the criticism, many vendors are willing to tout the battery life from the Productivity test as their overall MobileMark score. See these offers from Hewlett-Packard Co., Lenovo Group Ltd., and Dell Inc.
Only Acer Inc. (download PDF here) identified its MobileMark time as a productivity score. Asus Inc., Apple Inc. and Toshiba Corp. didn’t mention MobileMark on their Web sites.
In late June, a class-action lawsuit was filed in U.S. District Court in San Jose, targeting Intel Corp. for, according to the San Jose Mercury News, “essentially rigging those tests to inflate the battery life of laptops powered by its chips.”
Intel denied the claims, and noted that the same law firm, Girard Gibbs of San Francisco, unsuccessfully filed a separate class-action lawsuit against Intel several years earlier, according to the Mercury News.
Intel also disputes the claim that the public is being misled. “Anyone who criticizes consumers’ intelligence when shopping for laptops is underestimating the consumers,” an Intel spokesman told the Mercury News.
Carol Hess-Nickels, director of marketing for business notebooks at HP, took the same line. “I’d say we are pretty pleased with the benchmarks used today,” she said in an interview last month several days before the lawsuit was filed. “I’ve not personally gotten complaints.”
HP claims business netbooks such as the Mini 2140 and the Mini 5101 can run up to eight hours, or a full business day, on an optional, extended battery.
Lenovo, which has claimed as long as 7.5 hours of battery life for its laptops on extended batteries, acknowledges there is a problem, however.
“We don’t really like the fact that something is supposed to get four hours and users routinely say, ‘We divide that number by two and that’s what we get,’” said Lenovo segment marketing manager David Critchley in an interview, also several days before the lawsuit’s filing.
Dell appears to agree with Lenovo. “Customers expect the advertised battery life to reflect the way they really use the product,” Ketan Pandya, head of AMD-based products at Dell, told Newsweek last month.
As a counterbalance, some magazine reviewers go overboard to turn off all of a laptop’s power-saving features, Critchley said, which is equally inaccurate. “We put a lot of time and effort into our power manager,” he said. “You’ll see some significant gains from the way we handle sub-components.”
AMD, which complained that MobileMark essentially discriminates against its chipsets because they are more graphically powerful than Intel’s, is all for reforming MobileMark.
In a blog entitled “There has to be a better way,” AMD suggests turning the widely used performance benchmark, 3DMark06, into “an active battery life test” that it argues would be a more accurate measure of average battery life.
Lenovo, although game, is more cautious. “If Dell, HP and Apple all said, ‘Yea, verily,’ Lenovo would be right there with them,” Critchley said.
