FutureMark’s 3DMark Vantage DX10 Gaming Test

Introduction:
From its humble beginnings as MadOnion, gamers and enthusiasts alike have enjoyed using the tests that the mad scientists at FutureMark come up with to establish bragging rights. With each new version of 3DMark, FutureMark has given us something new to “ooh and aah” about and given the latest hardware a good solid work out. But it has been some time since we have seen a new version of this benchmark from them, and with changes in hardware and software, we have all been waiting for something to come along from them to wow us again. But all that has changed as FutureMark as dropped its latest testing bomb on us with 3DMark Vantage, the first gaming test that takes advantage of all of the goodies that Vista and DX10 gaming have to offer.

Product: 3DMark Vantage
Author: Sean Kalinich
Price: See the FutureMark site
Manufacturer: FutureMark
Spelling and Grammatical editor: Planetx64 Staff

{mospagebreak title=What’s New?}
What’s New?
Well to be perfectly honest about the only thing that is not new is the Icon you use to start 3DMark. FutureMark built 3DMark Vantage around a completely new render engine, one that is designed to take advantage of the features available in DX10. They have slimmed down the testing somewhat and now feature only four main tests (there are an additional six feature test that I will discuss later). Two are graphics tests that test specific graphical features of DX10 while the other two are CPU based tests.

The Two CPU tests are also completely new and feature better Physics and AI tests.  Vantage now supports Physics hardware acceleration (finally something to use that Ageia card with). They have even planned in a limited future proofing for Vantage, at least as far as Physics workload is concerned.
Another new feature of Vantage is the Feature Preset, before when you adjusted resolution or AA/AF you would not get an official score. Now 3DMark Vantage offers four presets; these are Entry, Performance, High, and Extreme. This offers you a greater flexibility when testing and gives you even greater bragging rights when you have the top “extreme” score.

One more new item and one that might be a negative is that 3DMark Vantage is OS limited, this is the first time that 3DMark has been limited to a specific OS that I can remember. It actually requires Vista with Service Pack one and will not install and run on an OS without it.

{mospagebreak title=System Requirements}

Speaking of requirements here are the new ones:
Minimum System Requirements to run 3DMark Vantage:

Processor:
Requirement: SSE2 support
Recommendation: A dual-core processor with performance equivalent to Intel Core 2 Duo E6600, AMD Athlon 64 X2 6000+, or higher
Graphics Card:
Requirement: Fully DirectX10-compliant graphics hardware with the latest available display drivers

System Memory:
Requirement: Windows Vista minimum requirement
Recommendation: 2 GB or more
Hard Disk:
1 GB of free hard disk space
Operating System:
Windows Vista with Service Pack 1

If you are uncertain about you systems ability to run 3DMark Vantage you can check it here http://www.yougamers.com/gameometer/10171

Pretty steep, and no wiggle room on the OS or GPU requirements. There is no DX9 capability in this new version.

{mospagebreak title=What’s Under the Hood?}
The Engine
“yeah I bored her out, dropped in a Holley double barrel with phase four lifters and a beverage rack”
As mentioned above FutureMark gave 3DMark Vantage a whole new render engine.  This new engine is designed to work with the new features available in Vista and DX10. It is much more complex than any other render engine that FutureMark has worked with and is responsible for doing quite a lot per frame.

1. GPU simulation update loop
a. Rendering of simulation inputs (depth views)
2. Shadow map generation
3. Pre-depth for appropriate materials (speed optimization)
4. Opaque illumination
a. Possible hierarchical rendering steps (reflection, refraction)
5. Translucent illumination
6. Post-processing
a. Possible custom scene rendering passes

Some of the work is actually performed before the main render sequence begins (Shadow Maps). With the rest following in line from step 3 to 5.

The lighting model of 3DMark Vantage uses HDR (High Dynamic Range) lighting using 16-bit-per-component floating-point render targets. HDR then passes this information on to post-processing before it is tone-mapped and displayed.
The render engine is made up of three shaders Material, light and transformation. Material shaders deal with reflectivity and the way an object emits light. Light shaders deal with how the light reaches the material. Transformation shaders perform vertex transformation. All shaders are pre-generated and cached for real-time access during the loading phase of the test. Shadows are implemented using Variance Shadow Maps and PCF-filtered cascaded shadow maps (say that five times fast) PCF shadows use penumbra size estimation without penumbra dithering.
The Variance Shadow Maps use light bleeding reduction.
Post processing is its own separate step in the render process which begins after the main scene render. Post processing includes the following per-camera animated effects:
Bloom
Streaks
Anamorphic flare
Lens flare (ghosting)
Lenticular halo
Depth-of-field
Motion blur
Depth fog
Film grain noise
Volumetric fog
Tone-mapping w/ gamma correction and vignette

I am not going to go into detail on each of these but FutureMark has made improvements to how 3DMark Vantage handles each during post processing.
Physics is handled with either Texture simulations or Vertex simulation and are done per pixel using pixel shaders. Particles are rendered using soft clipping.

Render Options:
You can adjust the render engine in to following ways:
Resolution:  All D3D resolutions are supported
Multisample AA: All MSAA counts are supported for the chosen render target
Texture Filtering and Maximum anisotropy
Texture Quality
Shadow Shader Quality
Shadow Resolution Quality
Shader Quality
Post Processing Scale
Post Processing disable per effect: allows you to disable each of the post processing effect independently with the exception of tone-mapping.

{mospagebreak title=The Graphics Tests}
The Tests:
As I mentioned above 3DMark Vantage has four main tests and six feature tests.
The Main Tests are broken into two categories Graphics and CPU:
The Graphics tests both use the same render engine and have many common features between them (such as post processing,  each has a fluid surface simulation and GPU based particle effects) but each one also tests  a few specific graphical features as well.

Graphics Test 1 – Jane Nash

This test is a simulation of an indoor game with multiple dynamic lights, complex surface models, complex characters and character rigs and contains water reflection and refraction physics simulation collision map rendering and a host of other features. It tests the effect of rendering a large number of static objects with complex dynamic skins, cascaded shadows, cloth simulation, anisotropic materials, it contains very few instanced objects and no volumetric effects.

What you will see is an Austin Powers-ish scene with a chest heavy, mini-skirted female spy running from the usual inept bad guys in the standard sea cave “bad-guy” lair. After the bad guys corner her on a hanging bridge she jumps onto an obliging crane with metal boxes. She tries to ride that to safety when one of the bad guys shoots the cable (accidentally as they are horrible shots) and it plunges into the water. Fortunately it lands right behind and dock with an experimental water craft covered by a camouflaged cloth. Our heroine is able to sneak (without getting a drop of water on her clothes apparently) into the craft and drive it out of the cave where it turns into a plane and she flies away.

The orange camo jump suits the bad guys wear in this scene is a plus as is the brief “up the skirt” from behind shot of Jane while she is lying on the crates. An additional physics related part of this test is the heaving chest of our heroine when she runs right into the camera, something for all the lonely geeks out there.  Perhaps the programmers at FutureMark should get out more, or think about a simulated sports bra for the next release.

Graphics Test -2 New Calico

This is a test of a huge space scene with a large amount of moving objects. The majority of these objects are rigid. There is also a planet and asteroid belt to render.
Almost all of the objects in this scene are moving, there are no skinned objects, there are lots of instanced objects, as well as local and global ray tracing effects.

To you this will look like a dramatic space scene with large battle cruiser approaching a planet that looks like it is right out of Star Wars. As the cruiser approaches it passes into the outer ring of an asteroid belt and then fires off a few planet busting missiles. Once these hit and begin to burn away the surface the battle cruiser launches its fighters that streak toward the surface to kill whatever the missiles did not.

{mospagebreak title=The CPU Tests}
CPU tests:
In prior versions of 3DMark the CPU tests were limited items dealing with basic AI and physics a locked frame rate and low resolution. Now FutureMark has changed the way they approach these areas of gaming performance. Gone is the locked frame rated and low resolution and in its place is a realistic representation of actual gaming in terms of resolution and effects. Both of the CPU tests are run using the same render engine that the graphics tests although the GPU work load is reduced by removing post processing(with the exception  of tone-mapping), complex shaders and shadows. The geometry used by the models is simplified and there is no world modeled outside the view of the camera.

CPU Test-1  AI

The AI test is represented by a crowded air race where bi-planes have to race though inflatable gates.  All of the planes are trying to find the most advantageous route through the gates.
The major AI load comes from 3D path finding. This algorithm follows three main steps.
1 The planes will follow the gates in order
2 Simulation of fixed wing flight
3 collision avoidance

The “search” feature behind all of this looks at the entire path for each plane. It will generate random candidates for the planes according to the physical flight model for each path. Once these are generated another function selects the best candidate based on items like ground proximity and the proximity to other planes.
All of this is run one thread per core with work dispatched to the first available thread.

The test is then measured in operations per second; you will see this as OPS on the information bar at the bottom of the screen.  The number of OPS is equal to the number of flight paths calculated. This number is divided by the run time to give you your score.

CPU Test-2 Physics
Physics is becoming a part of our gaming worlds. As game developers strive to give us more realistic and immersive worlds to play in, physics and particle effects become more important. FutureMark has included physics in its testing suite since 3DMark06, but it was very basic and did not do a good job at test or displaying these features. In 3DMark Vantage, FutureMark once again is using the AGEIA PhsyX library for its game physics simulation. However unlike 3DMark06, Vantage now has support for an add-in AGEIA PPU (physics processing unit). This support can be turned off in the settings to allow you to show the performance differences without the need to remove the PPU. (turning this support off will not invalidate your results). The physics test does not use GPU based physics as these are covered in the graphics tests.
The actual test consists of several air collisions as multiple planes try to fly through the same gates. This allows for the breakage of planes into a maximum of 12 pieces that are attached at 11 joints. Lift and propulsion is simulated by the PhysX engine also, while colored smoke is simulated by a fluid simulation also available in the PhysX library, this smoke dissipates realistically.
The planes are all fighting to get through multiple gates, these gates are of two types, one is a donut shape and the other consists of two tapered columns. The donut gate is simulated tough, pressurized cloth and floats above the ground. The Pole gates are a simulated elastic, foamy material.

One new feature of the CPU Physics test is its scaling. Without a PPU the test will simulate one pair of gates per CPU core. With a PPU it will add an additional four gates that are simulated only on the PPU and remove one pair from the CPU load. (for example a quad core CPU without a PPU will have four pairs of gates while the same quad core with a PPU will have seven pairs three for the CPU and four for the PPU)
Each gate pair rendered is done it its own world that does not interact with the others. Each of these worlds contains its own group of crashing planes and are assigned o individual CPU cores or to the PPU. The only exception is that the Rigid Bodies for the PPU worlds are run on a single CPU core (hence the reason to remove one pair of gates with the PPU installed) the worlds are simulated as fast as possible and are not synchronized. The Physics test is again measured in OPS where each physical step in a world represents a single OP. The test has a fixed time length at the end of the test the number of completed OPS is divided by the run time to give you the result.

{mospagebreak title=The Feature Tests}
3DMark also has six Feature tests that are available in the Advanced and Professional editions of the testing suite. These test various “features” (go figure) of your GPU as they relate to DX10.

Texture Fill Rate:
This test draws frames by filling the screen with different values using multiple texture coordinates that are read from a tiny texture. These are then scaled and rotated between each frame.
Color Fill Rate:
This test fills the screen multiple times with the color and alpha of each corner being animated. The pixel shader used in this test is pass-through. Color is interpolated and written directly to the target using alpha blending.
POM Shader:
POM or Parallax Occlusion Mapping is tested by drawing a rectangle that consists of two triangles. The camera position is animated to form the view. The pixel shaders in this test use Parallax Occlusion Mapping to simulate complex geometry. Heavy ray-tracing against the depth-map determine the intersection of the view ray with the geometry. Further ray-tracing determines visibility from multiple animated light sources. The surface is also shaded using the Strauss shading model. This tests your systems ability to render a very complex, heavy pixel shader that contains large amounts of texture reads. All geometry on screen is rendered in two triangles and is done entirely in the pixel shader.
GPU Cloth:
This one is pretty easy to figure out and tests the ability of your system to render cloth on the CPU. It is a vertex simulation with a combination of vertex shader and geometry shader stages. Several simulation passes are used for each step. Stream out is used to cycle cloth vertices from one pass to the next.
GPU Particles:
This tests the ability of your GPU to render particles it features hundreds of thousands of particles that collide with an invisible shape (a height Map) Stream out is used to cycle particle vertices from one pass to the next.
Perlin Noise:
This Math Heavy shader test is performed by featuring multiple octaves of Perlin noise that is evaluated in the pixel shader. It tests the arithmetic computing power of your GPU.

{mospagebreak title=Advanced Options}
Advanced Options:
As with all of FutureMark’s testing suites 3DMark Vantage give you control over what you test as well as how the tests run. In the options tabs there are controls that allow you to enable or disable features, change resolution, AA, AF, and other graphical settings.
3DMark Vantage allows you to create your own customized preset (in addition to the four that are provided). This allows you to test specific functions or to alter individual settings (hi resolution but no AA or AF) and tests for more specific testing. Using the custom setting does not generate a 3DMark score but you will get a score for the individual test you select and can be compared to other scores including the reset ones.
3DMark Vantage Professional Edition also offers test automation using test-based configuration files and a command line interface. This allows for easy testing on multiple systems. Output is in comma-separated text files.
3DMark Vantage can also read and provide a detailed display of your system info.

3DMak Vantage allows you, for the first time, to run the test at different resolution and actually generate a valid 3DMark Score.  Unfortunately you do not have complete control over this and have to use one of four provided presets. These and their settings are shown below.

{mospagebreak title=Scoring}
Scoring:

The scoring algorithm used in 3DMark is pretty complex and takes many items into account. Full details are reproduced from the 3DMark Vantage white paper below. (because I am lazy and did not want to type them all out).

Each graphics test will produce a raw performance result in frames per second (fps). We then take a weighted sum of these raw results, to reach a Graphics Score (𝑆𝐺𝑟𝑎𝑝ℎ𝑖𝑐𝑠), as follows:

𝑆𝐺𝑟𝑎𝑝ℎ𝑖𝑐𝑠=𝐶𝐺𝑇1𝐹𝐺𝑇1+𝐶𝐺𝑇2𝐹𝐺𝑇2

Where 𝐶𝐺𝑇1 is the scaling constant and 𝐹𝐺𝑇1 is the fps result for Graphics Test 1, and 𝐶𝐺𝑇1 is the scaling constant and 𝐹𝐺𝑇2 is the fps result for Graphics Test 2. The constant scaling is done to bring the score in line with traditional 3DMark score levels at the time of launch. The Graphics score will be calculated using identical weights for each preset. Thus, it will remain comparable across all presets, and even when using custom settings.

CPU Scoring
Each CPU test will produce a raw performance result in operations per second (ops). We then take a weighted sum of these raw results, to reach a CPU Score (𝑆𝐶𝑃𝑈), as follows:

𝑆𝐶𝑃𝑈=𝐶𝐶𝑃𝑈1𝑂𝐶𝑃𝑈1+𝐶𝐶𝑃𝑈2𝑂𝐶𝑃𝑈2

Where 𝐶𝐶𝑃𝑈1 is the scaling constant and 𝑂𝐶𝑃𝑈1 is the ops result for CPU Test 1, and 𝐶𝐶𝑃𝑈1 is the scaling constant and 𝑂𝐶𝑃𝑈2 is the ops result for CPU Test 2. As with the graphics tests, the scaling factor 𝐶𝐶𝑃𝑈 is used to bring the overall score into a traditional range. The CPU score will be calculated using identical weights for each preset. Thus, it will remain comparable across all presets, and even when using custom settings.

3DMark score

The 3DMark Score (𝑆3𝐷𝑀𝑎𝑟𝑘) for each preset in 3DMark Vantage is formed from the GPU score and the CPU score using a weighted harmonic mean, as follows: 𝑆3𝐷𝑀𝑎𝑟𝑘=𝑊𝐺𝑟𝑎𝑝ℎ𝑖𝑐𝑠+𝑊𝐶𝑃𝑈𝑊𝐺𝑟𝑎𝑝ℎ𝑖𝑐𝑠𝑆𝐺𝑟𝑎𝑝ℎ𝑖𝑐𝑠+𝑊𝐶𝑃𝑈𝑆𝐶𝑃𝑈

Where 𝑊𝐺𝑟𝑎𝑝ℎ𝑖𝑐𝑠 is the Graphics score weight for the preset, and 𝑊𝐶𝑃𝑈 is the CPU score weight for the preset. The constants and weights in the score formulas will be set as follows for each preset:

{mospagebreak title=The Victims}
Test systems:
Desktop GPUs used:
ATi 780G UMA GPU (with 512MB shared memory)
ATi HD2900XT
Foxconn 8800GTX 768MB
Foxconn 9800GTX

Desktop 1
GIGABYTE GA-MA78GM-S2H mainboard
AMD Phenom X3 8750 2.4GHz
2GB Kingston KHX7200D2K2/2G (512MB shared to Video for UMA testing)
Western Digital 74GB 10k RPM Raptor HDD
Mushkin 1.1KW PSU
Sony 16x DVD-ROM
Microsoft Windows Vista Ultimate x64

Desktop 2
Intel QX9770
Intel X48BT2
4GB Kingston KHX13000D3LLK2/2GX (4×1GB)
Western Digital 74GB Raptor
16x Sony DVD-ROM
Generic 1.44MB Floppy
1.1KW Mushkin PSU
Microsoft Windows Vista Ultimate x64

Performance Results:
For the performance testing a ran all of the presets with each of the Desktop cards (of course the x48 was not able to use the integrated 780G GPU) A screen shot of each result was taken for you to see below. I also added in an Asus PhysX P1 PPU and ran the tests again with each of the listed GPUs.
One other item you will see in the performance results is the use both existing nVidia drivers 169.25 for the 8800GTX and 174.74 for the 9800GTX) as well as the optimized drivers released by nVidia just for 3DMark Vantage. (175.12) For the ATi cards I used Catalyst 8.4.

Desktop1
This was an interesting run. I knew the 780G would not do well but I did not expect for it to fail all other tests except for the Entry preset. This was very surprising. I would have thought it would have at least been able to finish the Performance preset as that is the default run. I also could not get any of the feature tests to run. They would fail almost immediately.
For the rest of the system test the X3 performed fairly well maintaining a score in the high 3000 range.
As mentioned above you can see the effect the new 175.12 drivers have on performance.

(nVidia 169.25 Drivers)

(nVidia 175.12 Drivers)

As you can see even the best GPU gets a workout with the new Graphics tests and your CPU is going to take a pounding too.
To Save Space in the text of this article I only posted scores for the “Extreme” preset.
All other results images can be found here

Desktop 2
Here is where the fun began, With the QX9770 I was able to see just how much of an impact the CPU tests would have on Intel’s finest. The 4GB of DDR3 was also going to add some extra punch to the system. Desktop 2 also featured testing with and without a PPU. The reason I waited was due to a lack of room on the smaller Gigabyte board I used for the Phenom X3. Unfortunately my results are incomplete, during testing my 9800GTX died on me. This is the reason for the lack of performance numbers with the PPU and with the 175 drivers.
(nVidia 169.25 Drivers)

(nVidia 175.12 Drivers)

(nVidia 169.25 Drivers)

(nVidia 175.12 Drivers)

As with Desktop1 the rest of the results images can be found hereI also took the opportunity to run the entire suite including the feature tests to see what I would get. This run was done with the PPU and with the 8800GTX since the 9800GTX had died by this time.
Even with the faster CPU, RAM and the 9800GTX 3DMark Vantage still punishes the system. True the CPU score here is simply amazing but that still does not have a huge impact in overall score you can also gauge the CPUs impact on gaming by this lopsided result.

3DMark® Vantage Editions, Pricing and Availability{mospagebreak title=Pricing}
The following information is reproduced from the FutureMark website:

There are three editions of 3DMark® Vantage available: The Basic, Advanced and Professional Editions.

Due to the increased amount of work necessary to create a benchmark for Windows Vista OS which has required new software engines, tooling and artwork as well as an increased level of web services and analysis tools, offered to the benchmarking consumers, we have had to change our pricing strategy for 3DMark® Vantage. A no cost option is still available in the form of a one-time use trial of the Basic Edition. The trial version will deliver a 3DMark Vantage score and allow for submission of a result into the Online Results Browser, but, will require upgrading to the Basic Version for benchmarkers who want to use it for an unlimited number of runs. This can be done by simply entering in the correct registration code for the basic version without requiring an additional download. The differences between 3DMark Vantage Basic, Advanced and Professional editions are as follows:
The 3DMark(R) Vantage Basic Edition ($6.95) (Download Price)
Unlimited number of test runs using a single Preset setting
Network connection required to view results
Licensed for non-commercial and Personal Use only
The 3DMark(R) Vantage Advanced Edition ($19.95) (Download Price)
Unlimited testing, access to all Presets and custom settings
Network connection required to view results
Licensed for non-commercial and Personal Use only
3DMark Vantage Professional (this is the Edition for Press)  $495.00/seat (download or CD-ROM)
All Advanced version features
Technical support
View results without network connection
Benchmark automation with command line scripting
Licensed for Full Commercial Use

Now all of this is true, with respects to the extra time and manpower involved in writing these increasingly demanding tests. I do not think that $6.95 is too far off the mark in pricing. I do feel that the free version should have more than one use though. Perhaps as many as 10 before shutting down and not longer working.

Conclusion:
I like FutureMark and think they do a good job at providing an accurate simulation of modern gaming.However, I think that the graphics in this release could have been much better. To me they seems flat and bland the colors did not stand out nor did the textures, in 3DMark06 the colors shadows and textures were vivid and crisp, here everything seems blurry and faded. The models of people are also not as well done as in previous releases with skin tones that look like they were done with crayon instead of lifelike colors. Now, that does not mean this is not a great testing suite, just that the visual representation you watch during the running is not up to the usually very high standards of FutureMark.The tests themselves are very well done, with the noted exception of the visual quality, the items incorporated in each of the graphical tests does push the limit of even the most powerful GPUs out at the time. The CPU tests have been written in such a way as to accurately test multiple CPU cores and Multiple CPUs.  This is something that not even the gaming companies have gotten right. 3DMark Vantage also (finally) shows what can be done with an add -in PPU, again something the gaming manufacturers have missed. So while I am disappointed in the poor graphics of the tests I still feel that again FutureMark has give us the closest thing to real world testing that you can find in a synthetic testing suite.