MediaTek defends itself after benchmark cheating accusations

After reports emerged suggesting MediaTek has been cheating the benchmarking system, the chipset manufacturer has vehemently defended its position.

It has been alleged in AnandTech that MediaTek has been cheating the mobile enthusiasts with some clever code. In the firmware files, references were found tying benchmark apps to a so-called ‘sports mode’. When triggered (if a benchmark app has been initiated), features on the phone were ramped up to give the impression of better performance.

AnandTech claims the cheating was brought to light thanks to testing two different OPPO Reno 3 devices. The Reno 3 Pro (the European version) beat the Reno 3 (the Chinese version) in the PCMark benchmark utility, despite its Helio P95’s Cortex-A75 CPU cores being two generations older than the Dimensity 1000L’s Cortex-A77 CPU cores. And not only did the Reno 3 Pro has older MediaSet chipsets than the Reno 3 devices, it had half as many.

The difference in the test results were slightly unusual, though when a ‘stealth’ benchmark apps were used, the lower results were confirmed.

Why those in the industry feel it is necessary to cheat benchmarking tests is anybody’s guess. The negatives of being caught far outweigh the gains of impressing a few hyper-geeks, and the cheaters eventually get caught. It is embarrassing and some might ask whether they are a reliable partner. The chipsets in questions have been used in OPPO, Vivo, Xiaomi and Sony devices.

Following the original statement, which you can see at the foot of the article, an expanded blog post was offered to the industry.

“We do find it interesting that AnandTech has called into question the benchmarking optimizations on MediaTek powered devices, when these types of configurations are widely practiced across the industry,” MediaTek said. “If they were to review other devices, they would see, as we have, that our key competitor has chipsets that operate in the exact same way – what AnandTech has deemed cheating on device benchmarking tests.”

Although this is a very reasonable explanation, it is still a bit fishy. It is perfectly understandable for performance to be ramped up for some applications, but the fact the ‘sports mode’ has been linked to the initiation of a benchmarking app as well as other functions (gaming for instance) suggests the aim is to fool the tests. Most reasonable individuals would assume these tests are performed in ‘normal’ mode.

Whether this is an adequate explanation, we’ll let the court of public opinion decide, but it is somewhat of a flimsy excuse.

Original MediaTek statement:

MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets. We work closely with global device makers when it comes to testing and benchmarking devices powered by our chipsets, but ultimately brands have the flexibility to configure their own devices as they see fit. Many companies design devices to run on the highest possible performance levels when benchmarking tests are running in order to show the full capabilities of the chipset. This reveals what the upper end of performance capabilities are on any given chipset.

Of course, in real world scenarios there are a multitude of factors that will determine how chipsets perform. MediaTek’s chipsets are designed to optimize power and performance to provide the best user experience possible while maximizing battery life. If someone is running a compute-intensive program like a demanding game, the chipset will intelligently adapt to computing patterns to deliver sustained performance. This means that a user will see different levels of performance from different apps as the chipset dynamically manages the CPU, GPU and memory resources according to the power and performance that is required for a great user experience. Additionally, some brands have different types of modes turned on in different regions so device performance can vary based on regional market requirements.

We believe that showcasing the full capabilities of a chipset in benchmarking tests is in line with the practices of other companies and gives consumers an accurate picture of device performance.