Date:
Wednesday, February 12, 2003
Author:
Kyle Bennett
Google +1

Benchmarking Right

An editorial about the millions and millions of dollars that are invested in hardware to play computer games, yet when it comes to tools in order to evaluate that hardware, it is slim pickens at best.

continued...

Benchmark Facts

Synthetic gaming benchmarks wield an amazing amount of power over the hardware world that includes everyone from the chip producer to the person buying the card. Ask a GPU/VPU maker and they will tell you that they have to optimize for synthetic benchmarks. That is how their products get rated in print and web publications. Would it surprise you if I told you that large companies use synthetic benchmarks to measure video card performance and those results are used as a guideline for purchasing tens of thousands of cards that will be sold in complete systems? Believe it.

Evaluating video cards is not really something that is easy to do unless you are really in tune with hardware. This is why we see “point and click” benchmarks that give you a one number result have so much impact. Sometimes simplicity seems to outweigh data value in benchmarking.

Overall the ultimate responsibility of giving benchmarks traction falls into the laps of every computer hardware reviewer and editor in the world. We are the ones that give benchmarks credence and value in the community. If we use bad tools, so will everyone else. If editors and other decision makers are using these tools and basing conclusions on their results, companies like ATI and NVIDIA have no choice but to allocate huge resources to make sure they score well. To do otherwise would be to suffer a large economic impact and that is certainly something that no company currently needs in the tech industry.

The Solution

I wish working out a solution was as easy as pointing out the problem, but we all know that is seldom the case. This is also not an issue that will be fixed based on the actions of one person. It is going to take a lot of us working together to change the industry around us for the better.

So who is part of this solution? First and foremost are the game developers. We must convince them that native benchmarking tools are a huge benefit to not only the industry but to themselves. It is possible that by working together we can put incentives in place for game developers that make it more worth their while for taking on the native benchmarking tool task. We have to look to the print and web editors that use these tools as well and hope they put a bit of thought into qualifying the tools that they use to measure video card and system performance in a gaming environment. Companies such as AMD, ATI, Intel, and NVIDIA are the keys to the solution when it comes down to actually making this happen. These are the companies that already have developer relations teams in place that work with the game developers. If we can get those companies to utilize their contacts and relationships with the developers and reassure them that native benchmarking tools will deliver a benefit to all involved we have a chance at making this solution a reality. Bringing companies like HP/Compaq and Dell on board are important as well in order to be involved with utilizing these benchmarks in their evaluations of products and supplying feedback about their use. Of course performance box builders like Falcon Northwest and Voodoo are a must as well. In fact, there are a lot of “must haves” on the list.

Maybe separate benchmarks to pick from that are based on games that are shipping or are currently in development is what is needed for a proper evaluation process. Maybe part of the solution lies in having companies such as FutureMark, working with a new organization of all the folks mentioned above, in order to make sure that the 3DMarks of the world utilize the proper technology and give a score that we can all sign off on. Maybe sharing a bit of technology with the right folks could give great benefit to the gamers and the industry. Maybe we need an organization with a logo that can be included on game boxes so you know the game you are buying is part of the solution in getting better products to your hands. One thing is for sure and that is that many companies and individuals need to come together on a united front to build a better mousetrap.

We are currently talking to many of the right people in order to move this initiative in the proper direction when it comes to getting better benchmarking utilities for all of us to use. In doing this, we will hopefully benefit the person that is the only one that really matters when it comes right down to it; and that is you, the guys with the cold hard cash in your pockets that wants a great product for your money.