Futuremark Delists HTC One(M8) From 3DMark for Manipulating Scores 1


htc_one_m8_delist

Since last year we’ve been hearing of different OEMs optimizing their handsets to look better in the benchmarking apps. Well, continuing the same story, the latest one do it is the HTC’s flagship phone One (M8). This comes after AnandTech’s review of the device which as usual deals with the intrinsic details of the various devices.  AnandTech, in it’s review observed that the latest HTC flagship responds differently to the Play Store version of the 3DMark benchmarking app and the renamed APK of the same app. So as a result, HTC One M8 now appears at the bottom of the ranks without any score.

htc_one_m8_scores_cheating

Image credit: AnandTech

The chart shows how the HTC One (M8) behaves when running the public version of 3DMark in the Play Store compared with running a renamed, but otherwise identical version of 3DMark not available to the public or manufacturers.
As you can see, the average CPU frequency is about 15% higher when running the public version of 3DMark, despite the two tests having identical workloads. The blue line shows the HTC One’s real performance without the benchmark detection.

Surprisingly, HTC is very cool about it. The company has openly admitted the fact that its flagship treats benchmarking softwares differently. HTC was quoted saying,

…our engineers optimize in certain scenarios to produce the best possible performance. If someone would like to get around this benchmarking optimization there are ways to do so, but we think most often this will not be the case.

FutureMark has strict rules that define the good behavior of the handsets,

The platform may not detect the launch of the benchmark executable. The platform must not…modify the usual functioning of the platform based on the detection of the benchmark.

Further, HTC defends its optimizations saying,

Benchmarking tests look to determine maximum performance of the CPU and GPU.

In the statement issued directly by the Futuremark president, company has made clear that the purpose of its benchmarking softwares is to determine the “real world” performance of the devices and not the maximum performance of the device.

It’s very wrong on OEMs’ part to resort to such techniques but it seems this saga is not going to end anytime soon. All we can do is hope that benchmarking apps like Futuremark to continue taking the action any such malpractices. And it would be a very different world when this ends.


About Swapneel Bandiwadekar

I'm into tech since 2006.You can find me on forums as @swap002. I also used to write reviews for Hindustan Times and been a member of pilot programs. Connect with me on Twitter or Google Plus.