|
Author: Sean Kalinich Introduction: |
{mosimage} |
However, the days of using a static benchmark to test performance are fast declining. Time demos, once used by many to see how well a game and hardware would run, are no longer a true measure of performance. New games are using massive AI routines, Physics engines, particle effects and HDR rendering. All of these have an even bigger impact on testing a system and video card than the current system tests available. To add to this, GPUs are becoming more complex and powerful and able to render many effects in hardware, that until recently did not exist.
FutureMark has not sat idly by while these changes have been going on; instead they have shifted the philosophy behind 3DMark to match the new advances in gaming technology. We here at Planetx64.com have taken a close look at FutureMark’s new system test; 3DMark06, which promises to be a new standard in measuring gaming performance. Lets take it for a few laps around the track, shall we?
{mospagebreak title=A New 3DMark?}
New Benchmark? I don’t see any new benchmark.
When you first start 3DMark06 you notice that the splash screen is not very different. This does not give much hope for new and advanced tests when the application finishes loading and you are faced with the same configuration screen that was in 3DMark05. It is somewhat of a let down. Fear not, for the most significant changes are not on the surface. Those lie under the hood, not only in the way that 3dMark06 runs, but the way your final score is calculated. To understand the differences in design and thought, it is helpful to take a look back at the history of 3DMark and see what each version was developed to evaluate.
| |
|
{mospagebreak title=3DMark99, 2000 and 2001}
3DMark99 was developed to test fixed function vertex transformation and lighting, along with multitexturing.
3DMark2000 added support for hardware based transformation and lighting; it also increased the complexity of the individual game tests.
3DMark2001: once again the complexity of the game test was increased by raising the number of polygons rendered per frame into the tens of thousands. This version also saw the introduction of the shader technology. While the scenes used fixed function vertex and pixel processing the shader was used for the special effects. These special effects used vertex shader 1.1. Game test 4 included the first material using the 1.1 pixel shader.
{mospagebreak title=3DMark03 and 05}
3DMark03: in this version the focus moved to ShaderModel 1.x and 2.0. FutureMark left one test for legacy systems that still tested fixed function multitexturing; the rest used the Shaders for all material. The last test was the first to introduce ShaderModel 2.0. This last test also raised the number of polygons rendered into the hundreds of thousands.
3DMark05: With 3DMark05 the level of performance was again raised. ShaderModel 2 and 3 were used for all vertex and pixel processing, the number of rendered polygons per frame rises over one million on average. DirectX9 allows for the use of ShaderModel 2a, 2b, and, 3. This was enabled throughout the test suite to simulate what shader heavy games would offer.
{mospagebreak title=3DMark06}
3DMark06
With this latest release FutureMark once again raises the bar on the technology included in their testing suite. In 3DMark06 we see extensive use of ShaderModel 3 (SM 2 is still used in two of the tests) the introduction of HDR rendering, the use of five Uniform shadow maps for rendering dynamic shadows, sixteen and four sample smooth shadows, an increase from three GPU tests to four (two for SM2 and two for SM3/HDR) and a new SM3.0 feature test.
Added to all of this, FutureMark has increased the total number of tests to nineteen (from sixteen), the level of detail and texture size has increased, the level of the shader complexity has risen from a maximum of 96 instructions to 512. There are now three sub scores (SM2.0, HDR/SM3, and the CPU Score) the amount of VRAM used rises from 128 to 256. Default resolution has been raised from 1024×768 to 1280×1024. 3DMark06 also includes support and optimizations for multi-CPU/Core systems whether they are physical cores or logical.
{mospagebreak title=The ShaderModels}
I like a ShaderModel but I like them in the sun too.
Now that we know what each version of 3DMark was geared towards it will probably help to understand the different ShaderModels and their key features.
ShaderModels were introduced in DirectX8, and offered a new approach to the way graphics were rendered. Instead of using the CPU to perform character skinning, now, through the use of the ShaderModel, it could be done by the GPU. In addition to this, now instead of the limited per pixel light and environment bump mapping, it became possible to illuminate all of the surfaces in an entire scene.
SM1.0 was limited in the operations that it could perform, although this was extended slightly with SM1.4. Shaders also had to be written in a low level language similar to assembly code. SM1.0 was mostly used for special effects on specific surfaces.
SM2.0 brought about longer and more complex shaders, it also allowed more options. With SM 2.0, shaders could now be written in a higher language similar to C instead of the lower level assembly code.
SM3.0 provides even greater flexibility and is the ShaderModel used with HDR.
{mospagebreak title=3DMark06 Features and Tests}
Thanks for the history lesson, but what does it really do that’s new?
3DMark06 is a graphics system test, composed of four graphics tests and two CPU tests. Also included is a set of feature tests and batch size tests. The feature and batch size tests evaluate specific 3D functionality; however their results are not included in the scoring. The feature tests are there to test certain key 3D features (imagine that). The batch size tests execute rendering batches of different sizes. 3DMark06 also includes a set of image quality tools to help illustrate the way different image quality settings affect the graphics driver and hardware setup.
Some of the new key features of 3DMark06 are:
Real-Time Rendering:
All graphics tests in 3DMark06 are rendered in real-time; these are not merely static renderings but are designed to perform in the same manner as a true 3D game.
DirectX 9.0c:
All tests are reliant on DX9.0c (December 2005) and require DX9 hardware with support for Pixel Shader 2.0 or higher.
Updated 3D engine:
In early versions of 3DMark, the 3D engine did not perform in the same manner as a 3D game would. This changed with 3dMark05; now in 3DMark06 FutureMark has updated the 3D engine, adding support for ShaderModel 3.0, 16 bit floating point textures and blending as well as adding support for High Dynamic Range rendering.
HDR:
High Dynamic Range (HDR) rendering allows for extremely realistic 3D rendering. 3DMark06 included two HDR/SM 3.0 graphics tests. These tests require 16 bit floating point texture and blending, they also extensively use 16 bit floating point filtering but it is not required to run the tests. Also included with the HDR/3.0 SM tests are a more advanced post-processing bloom effect, and a ghost/lens reflection. The whole image uses Tone-Mapping to ensure the correct lighting values.
Full SM 3.0 Support:
3DMark06 uses all of the main features of ShaderModel 3.0 with the exception of vFace Register.
Dynamic Shadow Rendering:
Although this is not a new feature, the method used in 3DMark06 is. 3DMark06 uses Cascaded Shadow Maps; this is a new approach and allows for angle independent quality shadows.
Texture Compression:
3DMark06 uses three forms of texture compression; DXT1 is used for color maps, DXT3 for alpha maps, and DXT5 for all normal maps.
{mospagebreak title=The Graphics Tests SM2.0}
| |
Return to Proxycon is included in 3DMark06, just as it was in 3DMark05. This test is one of two that measures SM 2.0 performance and is also a good measure of how well your system will handle first person and multiplayer shooter games. Improvements in this level include more lighting, improved artwork, more shadow casting objects and the use of the new CSM (Cascaded Shadow Maps) shadow casting technique.
| |
Firefly Forest:
This test is the second test in the SM 2.0 suite and is a good measure of how well a graphics system can render an outdoors scene. Improvements in this level are once again improved artwork, the use of the CSM shadow technique, and the addition of a second firefly to increase the graphics load. One of the main points of interest in this test is actually the vegetation on the ground, it is dynamically distributed and the level of detail varies with the distance to the camera. Firefly Forest is an outstanding test for showing off the benefits of the new CSM technique.
{mospagebreak title=The Graphics Tests HDR/SM3.0}
Canyon Flight:
Another returning scene from 3DMark05 is Canyon Flight; this test has been updated again with improved artwork, SM 3.0, HDR rendering and the CSM Shadow technique, as well as a brand new shadow filtering technique. This test is an example of HDR rendering of a large-scale outdoor scene with very large water areas with many reflections. Also added are light scattering algorithms (to provide a perception of distance), HDR reflections and refractions, a depth fog to make the sea monster appear to be deep under the surface, a complex fog effect to give the appearance of humidity, and cloud blending. In all, the Canyon Flight test is an impressive display of HDR and SM3.0 at work.
| |
 |
Deep Freeze:
Deep Freeze is the second of the two HDR/SM3.0 tests. It is a brand new test designed to show HDR and SM3.0 performance in a vast landscape with very detailed HDR and soft shadow effects throughout the scene. Included in this scene are more fog effects and heavy particle effects to simulate a snowstorm effect.
{mospagebreak title=The CPU Tests}
CPU TESTS:
3DMark06 features two CPU tests; both of these are performed using the same game model built into 3DMark06, the game called Red Valley features 87 bots. These bots are a combination of “speeders” and “tanks”. The speeders are attempting to navigate around the course to a goal, while the tanks are attempting to destroy them. The game allows for the measurement of three types of CPU performance: game logic, physics, and path finding AI. Game logic (including graphics engine) runs as a single main thread, the physics simulation also runs as a single thread and is synced to the main thread at each physics step. The AI runs as multiple worker threads that is able to scale with available CPU resources, these worker threads are then synced at set intervals with the main thread. To enhance the CPU load, the tests are run at 640×480 with a fixed frame rate of 2fps.
Each test measures a different aspect of CPU performance. The differences are listed below:
Test 1:
Higher path finding task complexity
Tighter AI Synchronization
Duration 40 frames
Fixed Frame Rate (2FPS)
Shader Profile 2.0
640×480 Resolution
 |
Test 2:
Lower path finding task complexity
Laxer AI Synchronization intervals
Duration 60 frames
Fixed Frame Rate (2FPS)
Shader Profile 2.0
Resolution 640×480
The CPU tests are also heavily geared towards dual core or dual CPU systems.
 |
3Dmark06 also provides a large number of feature and batch size tests. These tests have no bearing on the overall score but can provide useful information to show the strengths and weaknesses of individual graphics cards. The number of feature tests prohibits a detailed description of each; only a listing of the tests provided will be included here.
Fill Rate:
Pixel Shader
Vertex Shader
Shader Particles Test (SM3.0)
Perlin Noise (SM3.0)
Batch Size Tests:
These tests render a very simple scene without optimizations; this exposes a “weak” spot in most graphics systems as game developers generally want to render as large a batch as possible. Added in between each batch is a color change state to prevent the individual tests from being rendered as a single batch.
{mospagebreak title=Performance Testing Methods and Tests Systems}
The testing methods:
To test how much of an impact 3DMark06 has on a system, I wanted to use multiple GPUs from different IHVs. Unfortunately, no ATi hardware was available at the time of testing. For this round of testing I used Leadtek’s PX6600XT TDH Extreme (both in single and in SLI) as well as Leadtek’s PX7800GTX TDH (256MB) once again in both single and SLI. Added to this, I ran these tests on three different CPUs: an Intel Extreme Edition 840 (with Hyper Threading on and off), an AMD Opteron 146 and an Opteron 170. This gave me a very good idea of how well the 3DMark06 testing suite scales with CPU speed and the addition of multiple cores. All tests were run from a clean install with the latest patches and updates; in addition the following software was installed to simulate normal usage on a normal system:
Avast Anti Virus
Microsoft Office 2003
Half Life2
Quake 4
Fraps 2.7
FEAR
Driver version for the graphics cards was 81.98.
The test systems:
Test system 1
Intel Extreme Edition 840
Asus P5WD2 Premium
OCZ DDR2 PC2-6400 Platinum Enhanced Bandwidth Dual Channel 1GB (2x512MB)
3x Hitachi T7K250 160GB 7200RPM SATA 3.0Gb HDD (RAID5)
Antec Titan Server case with TruePower 2 550 Watt PSU
Generic 16x DVD-ROM
Leadtek PH7800GTX TDH 256MB (single mode only)
Leadtek PH6600GT TDH Extreme (single mode only)
Test system 2
ECS KN1 SLI Extreme
AMD Opteron 146
AMD Opteron 170
Geil ONE PC3200 DDR 1GB (2x512MB)
2x Hitachi T7K250 160GB 7200RPM SATA 3.0Gb HDD (RAID 0)
Anted True Power 2.0 550 Watt PSU
Generic 16x DVD-ROM
Leadtek PH7800GTX TDH 256MB (single and SLI mode)
Leadtek PH6600GT TDH Extreme (single and SLI mode)
Microsoft Windows XP Professional and Windows XP x64 Edition was used on both test systems
{mospagebreak title=Performance The Tests}
In order to understand the scores listed below you need to understand the way 3DMark06 calculates – not only the overall score, but also each subsystem score.
SM2.0 Score = 120×0.5x(SM2 GT1 Ave FPS + SM2 GT2 Ave FPS)
HDR/SM3.0 Score = 100x.05x(SM3 GT1 Ave FPS + SM3 GT2 Ave FPS)
CPU Score = 2500xSprt(CPU1 Ave FPS x CPU2 Ave FPS)
3DMark06 Score = 2.5×1.0/((1.7/GS +0.3/CPU Score)/2)
GS score will be different depending upon the type of hardware used.
GS for SM3.0 hardware = 0.5x(SM2 Score + HDRSM3 Score)
GS for SM2.0 hardware = 0.75xSM2 Score
 |
Test System 1 Hyperthreading On
 |
|
Test System 2 Opteron 146
| |
Test System 2 Opteron 170
| |
As you can see, 3DMark06 does take advantage of multiple CPU cores and even receives a small benefit from the two additional logical cores on the XE840 CPU. This is a good thing to see, as the extra overhead from hyperthreading actually slowed 3DMark05 down. It is also interesting to note that while SLI shows an almost 100% improvement on the graphics tests, in no way does it effect the CPU score, just as a very powerful CPU does not effect the graphics scores. This will allow users to determine what areas in their systems may be weak in terms of gaming performance. Even with a 10% overclock we do not see a 10% overall score increase.
Finally below are screen shots taken during testing:
| |
|
||
| |
|
|
{mospagebreak title=After Thoughts}
3DMark has become both a staple for measuring graphics performance by the technical press and as a method for establishing bragging rights about one’s system. With 3DMark05 the tests could be swayed significantly by using either a powerful (or heavily overclocked) CPU or GPU. In 3DMark06, this is not the case. As noted above in the testing section, even OCing the CPU or GPU did not have a huge impact on the overall score.
FutureMark has also embraced many new gaming trends and technologies in their testing suite, making 3DMark06 the most in-depth single testing suite for graphical system performance as it relates to 3D gaming available today. While no static benchmark will ever be a true measure of in-game performance, 3DMark06 comes closest to real world gaming of any testing software produced.
You can view the entire gallery of images here
Now go and grab it at Futuremark
Discuss the review here