FutureMark 3DMark06 Technical Review

Introduction:
Who in the gaming and enthusiast worlds (increasingly becoming the same community) has not heard of, and used, FutureMark’s 3DMark? Used as a benchmark for system evaluations and for bragging rights amongst the growing number of gamers and PC enthusiasts, 3DMark has become the standard for measuring the performance level of both systems as a whole, and more specifically the performance of the video adapters that we spend so much of our hard earned money on.

However, the days of using a static benchmark to test performance are fast declining. Time demos, once used by many to see how well a game and hardware would run, are no longer a true measure of performance. New games are using massive AI routines, Physics engines, particle effects and HDR rendering. All of these have an even bigger impact on testing a system and video card than the current system tests available. To add to this, GPUs are becoming more complex and powerful and able to render many effects in hardware, that until recently did not exist.
FutureMark has not sat idly by while these changes have been going on; instead they have shifted the philosophy behind 3DMark to match the new advances in gaming technology. We here at Planetx64.com have taken a close look at FutureMark’s new system test; 3DMark06, which promises to be a new standard in measuring gaming performance. Lets take it for a few laps around the track, shall we?

New Benchmark? I don’t see any new benchmark.
When you first start 3DMark06 you notice that the splash screen is not very different. This does not give much hope for new and advanced tests when the application finishes loading and you are faced with the same configuration screen that was in 3DMark05. It is somewhat of a let down. Fear not, for the most significant changes are not on the surface. Those lie under the hood, not only in the way that 3dMark06 runs, but the way your final score is calculated. To understand the differences in design and thought, it is helpful to take a look back at the history of 3DMark and see what each version was developed to evaluate.

3DMark99 was developed to test fixed function vertex transformation and lighting, along with multitexturing.

3DMark2000 added support for hardware based transformation and lighting; it also increased the complexity of the individual game tests.

3DMark2001: once again the complexity of the game test was increased by raising the number of polygons rendered per frame into the tens of thousands. This version also saw the introduction of the shader technology. While the scenes used fixed function vertex and pixel processing the shader was used for the special effects. These special effects used vertex shader 1.1. Game test 4 included the first material using the 1.1 pixel shader.

3DMark03: in this version the focus moved to ShaderModel 1.x and 2.0. FutureMark left one test for legacy systems that still tested fixed function multitexturing; the rest used the Shaders for all material. The last test was the first to introduce ShaderModel 2.0. This last test also raised the number of polygons rendered into the hundreds of thousands.

3DMark05: With 3DMark05 the level of performance was again raised. ShaderModel 2 and 3 were used for all vertex and pixel processing, the number of rendered polygons per frame rises over one million on average. DirectX9 allows for the use of ShaderModel 2a, 2b, and, 3. This was enabled throughout the test suite to simulate what shader heavy games would offer.

3DMark06
With this latest release FutureMark once again raises the bar on the technology included in their testing suite. In 3DMark06 we see extensive use of ShaderModel 3 (SM 2 is still used in two of the tests) the introduction of HDR rendering, the use of five Uniform shadow maps for rendering dynamic shadows, sixteen and four sample smooth shadows, an increase from three GPU tests to four (two for SM2 and two for SM3/HDR) and a new SM3.0 feature test.
Added to all of this, FutureMark has increased the total number of tests to nineteen (from sixteen), the level of detail and texture size has increased, the level of the shader complexity has risen from a maximum of 96 instructions to 512. There are now three sub scores (SM2.0, HDR/SM3, and the CPU Score) the amount of VRAM used rises from 128 to 256. Default resolution has been raised from 1024×768 to 1280×1024. 3DMark06 also includes support and optimizations for multi-CPU/Core systems whether they are physical cores or logical.

I like a ShaderModel but I like them in the sun too.
Now that we know what each version of 3DMark was geared towards it will probably help to understand the different ShaderModels and their key features.
ShaderModels were introduced in DirectX8, and offered a new approach to the way graphics were rendered. Instead of using the CPU to perform character skinning, now, through the use of the ShaderModel, it could be done by the GPU. In addition to this, now instead of the limited per pixel light and environment bump mapping, it became possible to illuminate all of the surfaces in an entire scene.

SM1.0 was limited in the operations that it could perform, although this was extended slightly with SM1.4. Shaders also had to be written in a low level language similar to assembly code. SM1.0 was mostly used for special effects on specific surfaces.

SM2.0 brought about longer and more complex shaders, it also allowed more options. With SM 2.0, shaders could now be written in a higher language similar to C instead of the lower level assembly code.

SM3.0 provides even greater flexibility and is the ShaderModel used with HDR.

Thanks for the history lesson, but what does it really do that’s new?
3DMark06 is a graphics system test, composed of four graphics tests and two CPU tests. Also included is a set of feature tests and batch size tests. The feature and batch size tests evaluate specific 3D functionality; however their results are not included in the scoring. The feature tests are there to test certain key 3D features (imagine that). The batch size tests execute rendering batches of different sizes. 3DMark06 also includes a set of image quality tools to help illustrate the way different image quality settings affect the graphics driver and hardware setup.
Some of the new key features of 3DMark06 are:
Real-Time Rendering:
All graphics tests in 3DMark06 are rendered in real-time; these are not merely static renderings but are designed to perform in the same manner as a true 3D game.
DirectX 9.0c:
All tests are reliant on DX9.0c (December 2005) and require DX9 hardware with support for Pixel Shader 2.0 or higher.
Updated 3D engine:
In early versions of 3DMark, the 3D engine did not perform in the same manner as a 3D game would. This changed with 3dMark05; now in 3DMark06 FutureMark has updated the 3D engine, adding support for ShaderModel 3.0, 16 bit floating point textures and blending as well as adding support for High Dynamic Range rendering.
HDR:
High Dynamic Range (HDR) rendering allows for extremely realistic 3D rendering. 3DMark06 included two HDR/SM 3.0 graphics tests. These tests require 16 bit floating point texture and blending, they also extensively use 16 bit floating point filtering but it is not required to run the tests. Also included with the HDR/3.0 SM tests are a more advanced post-processing bloom effect, and a ghost/lens reflection. The whole image uses Tone-Mapping to ensure the correct lighting values.
Full SM 3.0 Support:
3DMark06 uses all of the main features of ShaderModel 3.0 with the exception of vFace Register.
Dynamic Shadow Rendering:
Although this is not a new feature, the method used in 3DMark06 is. 3DMark06 uses Cascaded Shadow Maps; this is a new approach and allows for angle independent quality shadows.
Texture Compression:
3DMark06 uses three forms of texture compression; DXT1 is used for color maps, DXT3 for alpha maps, and DXT5 for all normal maps.

Canyon Flight:
Another returning scene from 3DMark05 is Canyon Flight; this test has been updated again with improved artwork, SM 3.0, HDR rendering and the CSM Shadow technique, as well as a brand new shadow filtering technique. This test is an example of HDR rendering of a large-scale outdoor scene with very large water areas with many reflections. Also added are light scattering algorithms (to provide a perception of distance), HDR reflections and refractions, a depth fog to make the sea monster appear to be deep under the surface, a complex fog effect to give the appearance of humidity, and cloud blending. In all, the Canyon Flight test is an impressive display of HDR and SM3.0 at work.

Deep Freeze:
Deep Freeze is the second of the two HDR/SM3.0 tests. It is a brand new test designed to show HDR and SM3.0 performance in a vast landscape with very detailed HDR and soft shadow effects throughout the scene. Included in this scene are more fog effects and heavy particle effects to simulate a snowstorm effect.

Higher path finding task complexity

CPU TESTS:
3DMark06 features two CPU tests; both of these are performed using the same game model built into 3DMark06, the game called Red Valley features 87 bots. These bots are a combination of “speeders” and “tanks”. The speeders are attempting to navigate around the course to a goal, while the tanks are attempting to destroy them. The game allows for the measurement of three types of CPU performance: game logic, physics, and path finding AI. Game logic (including graphics engine) runs as a single main thread, the physics simulation also runs as a single thread and is synced to the main thread at each physics step. The AI runs as multiple worker threads that is able to scale with available CPU resources, these worker threads are then synced at set intervals with the main thread. To enhance the CPU load, the tests are run at 640×480 with a fixed frame rate of 2fps.
Each test measures a different aspect of CPU performance. The differences are listed below:

Test 1:

Tighter AI Synchronization
Duration 40 frames
Fixed Frame Rate (2FPS)
Shader Profile 2.0
640×480 Resolution

Test 2:
Lower path finding task complexity
Laxer AI Synchronization intervals
Duration 60 frames
Fixed Frame Rate (2FPS)
Shader Profile 2.0
Resolution 640×480
The CPU tests are also heavily geared towards dual core or dual CPU systems.

3Dmark06 also provides a large number of feature and batch size tests. These tests have no bearing on the overall score but can provide useful information to show the strengths and weaknesses of individual graphics cards. The number of feature tests prohibits a detailed description of each; only a listing of the tests provided will be included here.
Fill Rate:
Pixel Shader
Vertex Shader
Shader Particles Test (SM3.0)
Perlin Noise (SM3.0)

Batch Size Tests:
These tests render a very simple scene without optimizations; this exposes a “weak” spot in most graphics systems as game developers generally want to render as large a batch as possible. Added in between each batch is a color change state to prevent the individual tests from being rendered as a single batch.

The testing methods:
To test how much of an impact 3DMark06 has on a system, I wanted to use multiple GPUs from different IHVs. Unfortunately, no ATi hardware was available at the time of testing. For this round of testing I used Leadtek’s PX6600XT TDH Extreme (both in single and in SLI) as well as Leadtek’s PX7800GTX TDH (256MB) once again in both single and SLI. Added to this, I ran these tests on three different CPUs: an Intel Extreme Edition 840 (with Hyper Threading on and off), an AMD Opteron 146 and an Opteron 170. This gave me a very good idea of how well the 3DMark06 testing suite scales with CPU speed and the addition of multiple cores.  All tests were run from a clean install with the latest patches and updates; in addition the following software was installed to simulate normal usage on a normal system:
Avast Anti Virus
Microsoft Office 2003
Half Life2
Quake 4
Fraps 2.7
FEAR
Driver version for the graphics cards was 81.98.

The test systems:
Test system 1
Intel Extreme Edition 840
Asus P5WD2 Premium
OCZ DDR2 PC2-6400 Platinum Enhanced Bandwidth Dual Channel 1GB (2x512MB)
3x Hitachi T7K250 160GB 7200RPM SATA 3.0Gb HDD (RAID5)
Antec Titan Server case with TruePower 2 550 Watt PSU
Generic 16x DVD-ROM
Leadtek PH7800GTX TDH 256MB (single mode only)
Leadtek PH6600GT TDH Extreme (single mode only)

Test system 2
ECS KN1 SLI Extreme
AMD Opteron 146
AMD Opteron 170
Geil ONE PC3200 DDR 1GB (2x512MB)
2x Hitachi T7K250 160GB 7200RPM SATA 3.0Gb HDD (RAID 0)
Anted True Power 2.0 550 Watt PSU
Generic 16x DVD-ROM
Leadtek PH7800GTX TDH 256MB (single and SLI mode)
Leadtek PH6600GT TDH Extreme (single and SLI mode)
Microsoft Windows XP Professional and Windows XP x64 Edition was used on both test systems