Date:
Monday , May 24, 2004
Author:
Kyle Bennett
Editor:
Sean Quinn
Google +1

Cheating the Cheaters

Our thoughts and feelings on how we evaluate video cards and the reasoning behind our new philosophy.

continued...

The event that got my thinking moving back in the right direction was when the horrific NVIDIA benchmark cheating was the NVIDIA/3DMark scandal uncovered by the team at ExtremeTech. We are not strangers to cheating in benchmarks, but NVIDIA crossed the line. Still, however, this event did not get me back on track immediately, as I was still looking for game content developers to come to our rescue.

One thing I will still stand firm on is that the “NVIDIA cheating in 3DMark03” situation very much put the final nail in the 3DMark03 coffin as to it being a useful benchmark when it comes to showing an end user what kind of gaming experience he is going to have. We already thought that 3DMark03 did not accomplish that, but when the metric the benchmark delivers was repeatedly compromised, it really slammed the door on the issue in my mind. We touch more on NVIDIA and their actions here, and ran another editorial in November of last year on Synthetic benchmarking. This subject alone is probably worth a few thousand words, but we will save that for another day.

The point in bringing all of this up is to illustrate how we saw our primary video card benchmarking tool set becoming highly devalued at HardOCP. It was getting increasingly more difficult to look our readers in their eyes while relying on 3DMark03 to help shape my views about video hardware. Sure, using 3DMark03 is the easy way out, as anyone can run the benchmark and get a score that is accepted by many, but using 3DMark03 as a tool to help our readers make buying decisions seemed to lack integrity and value. We phased out 3DMark2003 early on.

The other portion of the tool set that was lacking was the ability to run time demos in the most recent games that were in retail. Some of them had the features, but many of the important games that enthusiasts were playing did not. That being said, we have the assurances of id Software and Valve that the benchmarking tool set will be in their upcoming games DOOM3 and Half Life 2. We will probably not use the time demo features, but there are certainly some other tools included that make evaluation easier for us.

So HardOCP was in a place where we were faced with some decisions we had to make. If we had just kept going the way we were, it just would not have been ethical in our eyes. If you don’t believe the work you are doing is hitting the target you are aiming at, you can’t just keep churning the stuff out without making some changes, and that is exactly what we did last October 2003.