Huawei caught being naughty with benchmark scores, but it doesn't seem to care

Four Huawei smartphone models have been delisted from independent benchmark service UL after they were found to be producing artificially high and misleading benchmark scores.

After testing the devices in the UL lab and confirming claims breached rules, the P20, Pro Huawei Nova 3 and Honor Play models have been delisted from UL’s performance rankings. The Huawei P20 has also been removed after AnandTech’s testing and reporting, the outlet which initially discovered Huawei’s suspect behaviour, was taken into account.

“Over the last five years, more than a few smartphone manufacturers have been caught and shamed for trying to manipulate benchmark scores,” the team said in a blog post. “That’s why UL has clear rules for manufacturers that govern how a platform can interact with its benchmarking software.

“After testing the devices in our own lab and confirming that they breach our rules, we have decided to delist the affected models and remove them from our performance rankings. Delisted devices appear unranked, and without scores, at the bottom of our popular list of the best smartphones. 3DMark scores from delisted devices should not be used to compare models.”

Each model was tested with the public version of UL’s benchmarking tool 3DMark, available from Google Play, and a private version of 3DMark that is not available to the public or manufacturers. As you can see from the performance results below, it does appear Huawei has been caught out gamifying the public version of the tool, while the private version unveils the real story.

When using the public version of the testing tool, there seems to be a hidden ‘performance mode’ that overrides the devices’ usual power profile, cheating the test. Scores using the public version of the tool were up to 47% higher than want UL’s private tool reported. Smartphones are allowed to adjust their performance depending on workloads, but they cannot be taught to maximise their behaviour when specific benchmark apps are running.

Huawei hasn’t necessary created a device which performs better than rivals on the market, it’s just figured out how to trick the tests. UL doesn’t unveil the mechanics of the private version, so Huawei devices would not be able to recognise this version of the app, thus giving a more genuine reading of typical performance.

This is of course not the first time manufacturers have been caught out trying to trick the performance rankings. Back in 2013, Samsung was caught out for the same naughtiness with its S4 model, which kicked off broader scepticism in the industry. Several more investigations were launched and several more manufacturers found out for mischief.

Perhaps the most interesting aspect of this story is Huawei’s response to the original accusation of dodginess. It didn’t offer any denial of wrong-doing, simply said that everyone else was it as well. That’s alright then.

Ultimately you have to wonder why Huawei bothered trying to trick its way up the leadership board. Such rankings are designed to get a very small segment of society fidgety and over-excited, it is a very niche group of nerds. Most ‘normal’ people, Huawei’s target audience as a mainstream brand, probably couldn’t care less. Fancy branding and clever advertising would more likely convince this audience to purchase a Huawei device as opposed to hyper-geeky performance stats.

The gain was seemingly very small, but now Huawei looks like a bit of a d*ck. Not exactly the more risk-sensible decision.