Today's Hard|Forum Post
Today's Hard|Forum Post

Benchmarking Future

We all know where we have been lately when it comes to benchmarking. But where are we going?

Introduction

If you are new to this storyline, we wrote an editorial about benchmarking back in February of this year that looked at how synthetic benchmarking was impacting the marketplace entitled Benchmarking Right. This article discussed some of the very real negative situations we identified with synthetic benchmarking. We recently followed this up with Benchmarking Wrong which is a look at the underbelly of benchmarking and the hundreds of thousands of dollars changing hands that many are not aware of.

"Optimizing"

The entire hardware community has been watching the world of benchmarking changing around us recently. What we have seen initially, that we find important, are situations where synthetic benchmarks have been آ“optimizedآ”. Optimizing for these benchmarks allows the company doing it to enjoy the advantages of having a higher score or however the data is represented. Some argue these scores are unfairly inflated, while others will argue that the company doing the optimizing is simply doing what they should be doing. The morals and ethics surrounding this have been hotly debated for a while now. I personally have seen this all as a waste of time and energy as no amount of argument and blame is going to close this Pandoraآ’s Box that has been opened.

For the last month or so, there seems to have been a new benchmark cheating scandal every day. There has been some time now for all of this to soak in and we would like to share some thoughts on what has transpired and what needs to be done to move forward beyond all of this. Don't take this editorial for more than it is, as it outlines some of the changes taking place around us and what we are going to have to do inside the hardware community to adjust for them. This is certainly not the end all be all on the subject and our thoughts will certainly evolve more as the industry does.

Quite simply, NVIDIA has changed the rules of synthetic benchmarking. Synthetics were understood to be free standing utilities that were not optimized for. We think that is how the hardware community saw these tools. But letآ’s look at the facts and legalities. As in the case of 3DMark03, we don't think there was anywhere in their EULA that specified you could not آ“optimizeآ” for the benchmark. We donآ’t think آ“cheatingآ” was defined. Like many other synthetic benchmarks, everyone understood the rules of the game, but no one defined them, as they possibly should have been. That is how we think NVIDIA skated on their 3DMark 03 آ“optimizationsآ” with a pack of lawyers clearing the way. NVIDIA is optimizing for benchmarks quite clearly and that is something we will have to deal with. It seems certain that they are not alone in doing that, but from what information we have been privy to in the last month it certainly seems to us that NVIDIA is doing a bit more optimizing than their competition.

The BIG Questions

These situations beg the question, آ“Do we stop using synthetic benchmarks as they are becoming of less and less value for evaluating future in-game or performance?آ” It could be argued that they have never done that anyway. But letآ’s stay focused on looking at the future of benchmarking video cards.

Apples to apples benchmarking is becoming more difficult. We have brought this up before and DOOM3 is a good example of this. We have separate code paths for different hardware. Many will argue that we need to benchmark on the same code path to give a true hardware comparison. The fact of the matter is that benchmarking the same code path on competitive video cards will not give us an accurate representation of gameplay on one of the two major cards out there. The reason we evaluate video cards here at HardOCP is to let people know how they are going to experience the games that they play when using the showcased 3D accelerator. That is something we are staying focused on. While we know there are many technical discussions to be had about video cards, it is the gameplay that is of the utmost importance and the overall experience derived from the hardware.

What we mention above with DOOM3 is only going to become more widespread and not necessarily in such a cut and dried fashion. We are going to see D3D games that behave differently within the same API. As the video card companies move forward with developing tighter bonds with game developers and publishing houses, there is no doubt we are going to see more hardware specific effects in games. Depending on your hardware, your game experience may differ. To what extent is yet to be seen, but we think that this alone will show that the days of apples to apples is coming to a close as the proprietary hardware technologies diverge. While this is just speculation, we think that NVIDIA moving some of their high-end business to IBM will facilitate this happening more rapidly as each faction of the video wars becomes a bit more shielded from each otherآ’s technology.

When 3DMark03 recently fell from grace, we knew there were sure to be others. This is far from over in my opinion. It is our duty as hardware evaluators to make sure we are representing video cards for the gameplay they deliver and little else.