Date:
Monday , November 04, 2002
Author:
Brent Justice
Editor:
Kyle Bennett
Google +1

UT2K3 Benchmarking

The full version of UT2K3 is here and we have put together a utility that will allow for apples to apples benchmarks and this time it has a GUI and is super-simple to use.

Article Image

Introduction

The full version of Unreal Tournament 2003 is finally here, and it was probably one of the most anticipated games this year. Back before the summer most people were getting the vibe that Unreal 2, the successor to Unreal, would be released first. But apparently they changed gears and decided to get Unreal Tournament 2003, the original Unreal Tournament's follow-up, out the door first. Unreal 2 is rumored to be gearing up for a Christmas launch, but as was with UT2K3, we won't actually know until it's a few weeks away from being on the shelves. Both games are based around the same game engine, dubbed the Unreal Engine. This engine is primarily DX7 based using TCL (Transformation, Clipping and Lighting) available in fixed hardware T&L video cards, though there are actually minimal amounts of DirectX 8 pixel and vertex shaders being used if available on the video card. If you have a DX8 card, pixel shaders will be used to take multiple terrain passes and combine them into one. There is, however, only a minimal performance difference and absolutely no image quality differences between DX7 and DX8 hardware. For all intents and purposes, it's a DX7 based game. UT2K3 does use cubemaps extensively as well as many times more polygons and texture sizes than its predecessor. The physics engine is known as the Karma Engine, which basically uses ‘building blocks’ between objects that allow for systems that will behave according to the laws of physics. This is the ragdoll effect you see with the characters. This physics engine can be used for other objects as well, to simulate real world physical movement. Sometimes though it seems not to be set quite correctly, showing characters moving or sliding a bit too much. Some people have referred to it as a 'jello' effect.

As we discussed in our Benchmarking with the UT2K3 demo article, when testing video cards and other hardware it's best to use real world applications. And nothing is better than using the actual game you play as a benchmark. Epic has been very smart to include with UT2K3 all of the benchmarking tools you need right out of the box to build a standardized benchmarking system for the game. We've learned a great deal since our dabbling with the benchmarking in the demo. We have taken what was already in there and spiced it up to give to you a much easier way to benchmark that will be fair and comparative for everyone.

Included Benchmarking Tools

Let's first talk about what was already in there, and then we'll go into what we've done. I'm sure most of you have noticed the benchmark.exe file located under the (C:\\UT2003\\System) directory of the game. This is a simple EXE file that when launched gives you the option of which resolution you want to run the benchmarks in and then completes two flybys and two bot matches and produces an average result for each.

Article Image Article Image Article Image

There are a few problems with this. For one, the bot match tests themselves cannot be used to accurately compare systems. The bot match tests utilize "bots" that are players controlled by AI. CPU power is very important here. Derivations in what the CPU is doing while running the bot match test can actually change the actions of the bots in such a match. So much in fact, that bot match benchmarks run back to back on the same machine can yield a different result. The other problem is the fact that the resultant score is based on the average of both flyby benchmarks. You can manually go into the (C:\\UT2003\\Benchmark) folder and open up the benchmark.log file or the lowframerate.log and see the results of each individual test. This, however, can become very tedious if you want to do a lot of benchmarking. This method also doesn’t give you any options other than resolution that you can change. I have to give props to the benchmark script though just for being there. Most games include no such benchmarking tools at all. The benchmark.exe does also use the same set of quality settings for each run, so that the flyby tests are comparative across different systems. Inside the (C:\\UT2003\\Benchmark\\Stuff) folder you'll find MaxDetail.ini and MinDetail.ini as well as MaxDetailUser.ini and MinDetailUser.ini. These are the configuration files the benchmark.exe uses for each test, so they do remain consistant on each run. Inside they have the MinDesiredFrameRate set at 35FPS, but as has been explained to me the benchmark ignores MinDesiredFrameRate when in -benchmark switch mode, so it's not a factor at all during benchmarking tests. This little app is good if you want to do some quick benchmarks and compare them with others - using the flyby tests only, of course - but it doesn’t give you the kind of control that most people want when benchmarking.

For that we have to go back to good ol' batch file scripting. If you look under the (C:\\UT2003\\Benchmark) folder, you'll see pre-configured batch files already in there for you. There are seven total bot match tests and seven batch files for the flyby tests. As we stated before, we are not going to use the bot match tests; we will only concentrate on the flyby tests. The flyby maps are Antalus, Asbestos, Citadel, Face3, Inferno, Phobos2, and SunTemple. For reference, the two flyby maps used in the benchmark.exe are Antalus and Asbestos. You can clearly see the advantages of using the batch scripts to compare performance on seven different maps versus just two. Let’s break down what's inside one of these batch files. Opening up flyby-antalus.bat, we see this:

..\\System\\ut2003.exe dm-antalus?game=engine.gameinfo exec=..\\Benchmark\\Stuff\\flybyexec.txt -benchmark -seconds=77 -nosound %1

The two dots at the beginning of the path indicate this to be a relative path, i.e. we have to have this batch file in the appropriate folder to work in its current state. The ut2003.exe from the (UT2003\\System) folder is being called upon and it's being told to execute the dm-antalus map using the engine.gameinfo information. It then executes a .txt file with a couple of commands in it. The –benchmark switch engages it into benchmark mode and the –seconds=77 switch is signifies how long the flyby runs for. The –nosound switch is used to disable sound while the flyby is in progress. By default it's going to use the ut2003.ini file located in your (UT2003\\System) folder. Our benchmark utility for the Demo overwrote that INI file for three different quality settings. This new benchmark for the full version game does things a little bit differently, which I'm sure you're going to like.

Once you run one of these batch files the resultant scores are written to a benchmark.log and lowframerate.log file in the same directory (UT2003\\Benchmark). Both of these files show the low/average/max and total average frames per second. The lowframerate.log file represents the data a little more organized by showing the map name above each test. Our benchmark utility for the Demo used this file as our main results file. There is also a complete detailed breakdown of each benchmark that's been done in the (UT2003\\Benchmark\\CSVs) folder. These .CSV files can be opened using Excel. There's a lot of information contained within these files, giving you a frame by frame count of what happened during the flyby test. For our purposes, however, they're a bit much to go through. We need to be able to see the Minimum and Maximum and Average FPS much easier, so we use the lowframerate.log file to view this information after each test. With the batch files, many combinations can be made to make some very good benchmark scripts. This is what we have done, plus more.