Today's Hard|Forum Post
Today's Hard|Forum Post

Benchmarking the Benchmarks

It is time to put your money where your mouth is, or maybe where your keyboard is. HardOCP sets out to prove that real world video card testing is where it's at. Beware, we may make you feel dirty every time your run a benchmark from here out!

Introduction

HardOCP has been thoroughly mired in "real world" testing scenarios for years now, and there is no doubt that we have taken a lot of heat from various sources. Most of it is generated through ignorance and the human quality of being resistant to change. Some of it is generated by websites and forums that will proclaim that there is no need for tools beyond a timedemo or cutscene in order to understand graphics cards' performance.

Derek Wilson, Anandtech - 01/28/08: "I'll still stand by the fact that it is not necessary to look at gameplay situations in order to build an accurate picture of the relative performance of a graphics card."

There is no doubt that there are also some highly intelligent, easy going folks, that simply do not find value in our real world testing as they think it to be too subjective. All of this is good with us as it has spurred a lot of conversation and site growth over the last few years and in our opinion has certainly made HardOCP a more valuable resource to the enthusiast hardware and gaming communities, or at least parts of them. We value differing opinions and have posted links to opinions other than ours on our news page for almost a decade now. We understand the world of computer hardware is hardly ever as easy as "black and white." We have never been a website that has tried to be all things to all people and never will be.

I have no problem stating that we think the real world computer hardware testing methodology is vastly superior to most of the synthetic and canned benchmarks we see today. That is not to say that synthetic and canned benchmarks do not have their places in testing, we just don't usually find those metrics to be indicative of what the end user has in terms of actual experiences. Some websites want to tell you the "relative performance of a graphics card" based on a timedemo that in no way represents playing the game. That is not what we want to focus on here at HardOCP.

A Moment of Clarity

About five years ago or so, I was heading out to San Francisco to do a live episode of the Screen Savers on TechTV. I was invited out to talk about the latest and greatest video card of the day on live TV, which was the NVIDIA GeForce FX 5800. Obviously, live TV is something you want to feel as though you are prepared for. We had tested the FX 5800 for days and days before I was to leave.

A couple of days before I was to be in San Francisco someone asked me, "How did you like gaming on the card?" I had no idea. I had never actually used the video card to play games, I had only "benchmarked" it using traditional methods. Here I am supposedly an "expert" on video cards telling our readers which cards are best for them and I had never used the card for the primary application that it is sold for; playing 3D games. It was a "moment of clarity."

I truly knew at that moment HardOCP's direction in video card reviews had to change. I was not sure what needed to change at that moment, but this clarity pointed me in the right direction. This is not the first time I have discussed this turn of events publicly, but I think it is worth telling again because there are a lot of folks that do not understand why we do what we do when it comes to evaluating video cards. Hopefully this anecdote highlights HardOCP's mission. You can go back and read, Cheating the Cheaters from 2004 as it covers a lot of reasons behind our motivation for change as well.

[H]ardOCP's Video Card Focus

The point is this. HardOCP stopped doing video card "reviews" quite a while ago. We started doing video card "evaluations." Now, what the hell a "video card review" is, I am not quite sure of and that is one reason why we stopped referring to our coverage of video cards as "video card reviews." HardOCP does video card "evaluations." You may see this as splitting hairs, but we hope our label rings true with most of you and serves to differentiate HardOCP from the cookie cutter review sites out there.

We think our job is to explain the level of gaming experience you should expect when you purchase the video card we have evaluated. If we are suggesting to you that you purchase a $400 video card, I think we had damn sure better serve up a reason to you other than a timedemo, canned benchmark, or synthetic benchmark metric. Some people think video card value can be communicated by running simple timedemos; we contend it's not that easy. It was at one time, but those days have passed. At least for HardOCP.

If you refuse our philosophy, I do appreciate that. I do ask that you give us a couple pages of your time though. If after you finish here you still think we are nuts, then so be it. We will have to agree to disagree, and I think we can all live with that.

On the following pages we are going to do our best to prove to our readers why we think our way of evaluating the experiences that video cards provide is ultimately better than a simple "review."