|
|
Introduction:
If you call yourself an enthusiast then you are familiar with FutureMark and their synthetic testing suites 3DMark and PCMark. Well, FutureMark has been back in the lab, working on updating their rapidly aging PCMark05 benchmark. Now after 2 years we are finally going to see a new version. Let’s take a look at the new test and see how it ticks.
|
|
Product: PCMark Vantage
Author: Sean Kalinich
Reviewed on: 25th 2007
Product cost: $6.95 – $495 (See Below)
Manufacturer: FutureMark
Spelling and Grammatical editor: Planetx64.com Staff
{mospagebreak title=What’s New?}
What’s new?
The more things change, the more things change.
PCMark has traditionally been a productivity test with a few image manipulation and graphics tests thrown in. But now PCMark is more of a complete system test. FutureMark has added in some of the same gaming tests from 3DMark as well as some new tests that throw some interesting AI calculations at the CPU and some streaming data tests at the HDD. Below are some of the major changes I will not go into over lavish detail nor cover all of them just the outstanding ones that go to make this a completely new product.
|
|
|
|
|
For starters the look of the suite has been cleaned up, now it is much simpler to see what you have going on in terms of the tests you want to perform. You are able to select the hard drive you want to use for your HDD testing.
|
|
The in-test screens have been improved and actually provide small explanations about why any given test is important in testing overall system performance.
But even more than the cosmetic changes are impressive changes into the way the suite tests your system.
FutureMark has revamped the actual tests and the suites to provide a more realistic testing scenario.
They have used real-world situations and created synthetic representations of these for PCMark Vantage. While it can be argued that PCMark05 did the same thing, it was nowhere near the scale that Vantage does.
The testing suites have been named according to what they test making it easier to select the appropriate test for the platform you are using. Each suite contains certain synthetic tests to measure a given set of hardware’s ability to handle certain tasks, while the PCMark testing suite contains a little of all of the suites for an overall view of your system performance.
Another new item is not a technical item but is one that always tends to cause controversy; that is pricing. Due to the extra coding and re-writing for Vista FutureMark has changed the pricing scale on PCMark Vantage. Although there is still a free version it is only a one time use product. To get the basic application you will have to come up with $6.95 not a terribly high amount but one that I am sure will have some people ranting and raving.
The complete details of the new pricing structure are below:
PCMark Vantage Editions, Pricing and Availability
PCMark Vantage is available in Basic, Advanced and Professional editions. All editions are available at http://futuremark.com/products/pcmark_vantage/ Differences between the Basic, Advanced and Professional editions are as follows:
PCMark Vantage Basic
$6.95 (download only – limited free trial edition also available)
-PCMark Test Suite
-Network connection required to view results
-Licensed for non-commercial and Personal Use only
PCMark Vantage Advanced
-$19.95 (download only)
-All Test Suites enabled
-Advanced online services
-Licensed for non-commercial and Personal Use only
PCMark Vantage Professional
-$495.00/seat (download version)
-$495.00/seat (CD-ROM)
-All Advanced edition features
-Full benchmark settings
-Technical support
-View results without network connection
-Benchmark automation with command line options
-Export results to Microsoft Excel
-Licensed for Full Commercial Use
{mospagebreak title=The Suites}
The actual tests involved have been changed to reflect more realistic usage.
The Suites are broken up as follows:
PCMark Suite
This is the standard benchmarking suite and contains the following tests:
Suite Tests –
Memories 1 & 2
TV and Movies 1 & 2
Gaming 1 & 2
Music 1 & 2
Communications 1 & 2
Productivity 1 & 2
CPU Tests -
Data Encryption, Data Decryption, Data Compression, Data Decompression
CPU Image Manipulation(compression/decompression/resize/rotate)
Audio Transcoding
Video Transcoding
Text Editing
Web Page Rendering
Windows Mail
Windows Contacts
CPU Game Test
Graphics Test -
Video Playback
CPU Game test
HDD Test -
6 HDD Tests Total
HDD Tests use new 4k block size (With a fresh install of Windows Vista the 4K alignment will be correct upgrades and 3rd party partitioning tools may cause alignment issues)
All HDD Tests use RankDisk using Intel’s IPEAK Storage Performance Toolkit
WinTrace is used to record the HDD traces
AnalyzeTrace is used to analyze what the traces are doing
RankDisk is a FutureMark modified non-destructive version that runs in a temporary file.
Now all of this is just the PCMark suite. As you can see there are tests in the PCMark suite that are parts of other suites. FutureMark made the choice to allow you test your system for specific performance. Each of the additional suites covers a different aspect of PC performance. These suites and their tests are listed below:
Memories Suite:
This suite is a combination of tests designed to test the most common memory usage on a system
if you run this suite on its own you will receive a score based solely on memory performance.
The Memories Suite contains the following tests:
Memories 1
Two simultaneous threads
CPU image manipulation
HDD picture import
Memories 2
Two simultaneous threads
GPU image manipulation
HDD video encoding
Memories 3
Video Transcoding (simulating transcoding from DV to a portable device)
Memories 4
Video Transcoding (simulating transcoding from a media server archive to portable device)
{mospagebreak title=The Suites II}
TV and Movies Suite:
This suite, as its name implies, is for testing your system’s ability to play SD and HD movies and TV under Windows Vista.
|
|
This suite contains the following tests:
TV and Movies 1 -
Two simultaneous threads
Video Transcoding – HD DVD to media server file
Video playback – HD DVD (additional lower BR HD content from HDD - downloaded from internet)
TV and Movies 2 -
Two simultaneous threads
Video Transcoding – HD DVD to media server file
Video playback – HD MPEG-2 (19.39 Mbps over the air HDTV playback)
TV and Movies 3 -
HDD Media Center
TV and Movies 4 -
Video Transcoding – Media server file to portable device
Video playback – HD MPEG-2 (48 Mbps Blu-ray playback)
Gaming Suite:
Another of the self-explanatory titles the gaming suite is a series of tests designed to mimic the “normal” gaming environment and give an indication of how well your system should perform during gaming tests. All of the gaming tests are based on DX9, DX10 gaming tests should arrive in the next version of 3DMark.
|
|
Gaming 1 -
GPU Game Test
Gaming 2 -
HDD: Game HDD (This test is simulated streaming data)
Gaming 3 -
Two Simultaneous Threads (this is a combination of the tests listed below)
CPU game test
Data Compression: Level loading simulation.
Gaming 4 -
Three Simultaneous Threads (These combine to simulate actual gaming conditions)
GPU Game test
CPU Game test
HDD: game HDD
Music Suite:
This test is all about how Vista can be used for digital music, with tests that cover Transcoding, browsing online music shops, adding music to a library, and of course Transcoding audio into different formats.
The Music suite is to show how well your system can handle being your digital jukebox and is made up of the following tests:
|
|
Music 1 -
Three simultaneous threads
Web page rendering (simulating browsing an online music shop)
Audio Transcoding – WAV format to WMA lossless
HDD: adding music to Windows Media Player
Music 2 -
Audio Transcoding – WAV format to WMA lossless
Music 3 -
Audio Transcoding – MP3 to WMA
Music 4 -
Two simultaneous threads
Audio Transcoding WMA to WMA
HDD: adding music to Windows Media Player
{mospagebreak title=The Suites III}
Communications Suite:
The purpose of this suite of tests is not obvious on the surface. It is designed to test the ability of your Vista system to handle data transfers, encryption, webpage rendering (with multiple pages or “tabs”) as well as “on-the-fly” scanning by antivirus and malware prevention applications.
It is comprised of the following tests:
|
|
Communications 1 –
Three simultaneous threads
Data encryption: CNG, AES, CBC
Data compression
Webpage rendering : Graphical pages in a 1024×768 window
Communications 2 -
Three simultaneous threads
Web page rendering: opening various simulated news pages from IE7 favorites in separate tabs then closing them one at a time.
Data encryption: CNG, AES and CBC
HDD: Windows Defender scan
Communications 3 -
Windows Mail: Search test
Communications 4 -
Two simultaneous threads
Data encryption
Audio Transcoding: WMA to WMA this is intended to simulate conversing using VoIP such as Skype
Productivity Suite:
This suite is one that I actually do not totally agree with. This is nothing to do with testing productivity on a system (I agree completely with that) it is more the implementation of the suite. For the most part when you purchase a system (or even when you build one) you will install some sort of productivity software. This will be in the form of Microsoft Office, Star Office suite, or any one of a dozen options out there for office productivity. Instead of using these Future Mark went with the built in text editing and mail applications for the testing engine. Although I am sure this was done to reduce programming and licensing costs it still does not give a completely accurate representation of how well your system will run when confronted with the bloat of Office 2007 or with the issues surrounding the Corel Office X3 suite. But enough of my rant here, this suite is designed to show you how well your system can handle office productivity work using the built-in tools that Microsoft’s Vista provides.
(On an interesting note, I found that with Vista x64 and Office 2007 installed and when running the 32 bit version of the test, the productivity suite would fail the text editing portion. This would not happen with the x64 version of PCMark Vantage or on the 32 bit version of Vista. The issue also went away if Word 2007 was uninstalled from the x64 version of Vista. It would appear that Word 2007 makes some changes to Vista x64 that affect this test.)
The tests are:
|
|
Productivity 1 -
Two simultaneous threads
Text editing (using the built in Word Pad provided with Vista)
HDD: Application loading (simulating information transferring from HDD to CPU and memory)
Productivity 2 -
Two simultaneous threads
Windows Contact search
HDD: Windows Defender (Malware scan)
Productivity 3 –
HDD: Windows Vista Startup (this test is similar to the PCMark05 XP startup test)
Productivity 4 –
Three simultaneous threads
Windows contacts search
Windows Mail: Run Message Rules
Web page rendering: opening various simulated news pages from IE7 favorites in separate tabs then closing them one at a time.
HDD Suite:
In the majority of systems people do not think about the speed of their hard drives, sure they want a bigger drive but how many people would get a slower drive to save money and get a few extra GB?
This is even worse when it comes to laptops and the box systems. In reality our hard drives are a huge part of performance, especially now in the days of real-time virus and malware scanning, full text indexing and media catalogs. FutureMark developed this suite with that in mind. The results of this test really do give a good impression of how your system can handle the constant data transfers that are going on all the time on a modern system.
The tests included in this suite are:
HDD 1
HDD: Windows Defender scan
HDD 2
HDD: Game HDD access (Simulated streaming data)
HDD 3
HDD: Picture Import
HDD 4
HDD: Windows Vista Start up
HDD 5
HDD: Video editing (Video editing is extremely HDD intensive especially the capture and rendering of the video)
HDD 6
HDD: Media Center (This test appears to be based on the ability of the system to “capture” video on the fly for DVR playback)
HDD 7
HDD: Adding music to Windows Media Player
HDD 8
HDD: Application loading
{mospagebreak title=Performance – The Systems}
Ok now that we have talked about all of the suites that you have let’s see how well this can differentiate between systems
Does PCMark Vantage give us a good performance evaluation tool?
I tested four different systems to see what I would find.
System 1
Intel V-8 System
2 Quad-Core Intel Xeon X5365 Processors – 3.0GHz 1333MHz FSB
1 Intel Workstation Board S5000XVN
2x 1GB Samsung 667MHz ECC FBDIMM memory
4x 2GB Kingston 667MHz ECC FBDIMM memory
2 Xeon HSF (Heatsink and Fan)
1x CoolerMaster 850Watt PSU (RS-850-EMBA)
1x Foxconn 8800GTX 768MB
3x Seagate Cheetah 15K.4 ST373454SS 74GB 15k RPM SAS HHDs (RAID 5)
3x 80GB Seagate Barracuda 7200.10 ST380815AS SATA II 3.0G HDDs (single drives)
LSI Logic LSI00118 PCI Express SAS MegaRAID 8344ELP 128MB
1 Sony DVD-RW drive
Microsoft Windows Vista x64 Ultimate
System 2
Intel X6850 1333 FSB 3.0GHz Quad core
2x 1GB Kingston Hyper-X PC2-9200 @ 1066MHz
2x 74GB Western Digital Raptor HDDs (Dual Boot)
ATi Radeon X1950XTX CF edition
eVGA 680i SLI mainboard
Mushkin 1KW PSU
Sony 16x DVD ROM
Dual Boot – Microsoft Windows Vista x64 Ultimate and Microsoft Windows Vista x86 Ultimate
System 3
AMD FX-62 (AM2)
4x 512MB Corsair PC-8500 DDR2 @1066MHz
1x 74GB Western Digital Raptor HDD
OCZ 520Watt ModStream PSU
ECS KA3-MVP Extreme mainboard
Leadtek 7900GTX 512MB
Sony 16x DVD-ROM
Microsoft Windows Vista x86 Ultimate
System 4
HP tx1000 entertainment tablet PC.
Turion 64 x2 (2.0GHz)
2GB PC2-5300 Hyundai RAM
Seagate Momentus 7200.1 200GB SATA150 HDD
nForce 6150 Go (128MB shared memory)
Microsoft Windows Vista x64 Ultimate
{mospagebreak title=Performance – The Tests}
With these four systems I felt I would get a good idea of whether or not PCMark Vantage would be able to give a good representation of a systems performance.
System 1
|
PCMark Vantage x86 |
PCMark Vantage x64 |
|
|
|
Looks like the V8, while powerful, is not getting the top score with either the 32 bit or 64 bit versions of Vantage. This is interesting but not alarming considering the RAM speed bottle neck and the fact that this system is not overclocked in anyway. This is one of the systems I had an issue with the x86 version of PCMark Vantage and Word 2007, after I uninstalled Word 2007 I was able to run the x86 version of the test without issue.
I was surprised that PCMark Vantage was able to max out ALL Eight cores during the gaming test.
|
|
System 2
On this system I had a dual boot setup so I was able to test the differences between the 32 bit version of Vista and the 64 bit version.
| Windows Vista x64 |
|
|
PCMark Vantage x86 |
PCMark Vantgae x64 |
|
|
|
|
PCMark Vantage x86 |
|
|
Looks like Vista x64 is faster than the 32bit version, even running the 32 bit version we see that Vista x64 is faster, granted not by much but it is still faster. Where we find the major speed increase is with the 64 bit version this gives us an almost 300 point improvement on the same system.
I know most of you will have noticed that the QX6850 outperformed the V8, Much of this is due to memory speed, 667MHz Vs. 1066MHz makes a huge difference.
System 3
This is a Vista x86 Ultimate installation on an AMD Athlon FX-62.
|
|
This system was intended to show how older hardware would fare with the new suite, even with the older CPU and GPU the score is still not bad.
System 4
This system is my work laptop and not meant to be a mega system. It has not real power for gaming and is only good for productivity work and entertainment (movies etc).
This is another system that I had a problem with the x86 version of PCMark Vantage and Word 2007.
On this system I did not uninstall Word 2007 like I did on the V8.
|
|
The tx1000 did not do a bad as I thought it would, in fact it did quite well considering that the GPU is not SM3.0 compliant.
By looking at the results I feel that FutureMark has given us a synthetic test that comes very close to showing real world performance and is a very capable benchmarking tool.
Editor’s Note:
It is interesting to see what others have said about this test. Although this review was supposed to be released on the 18th, due to unforeseen issues with an unruly ISP, I was not able to get everything finished up. I have had the chance to read a few other evals and am surprised that people have fallen into more MS and Vista bashing. Personally I know that NO OS regardless of who writes it is perfect, the general public should know this too. What gets me is the ready abuse they lavish on MS every time a new OS hits the streets. The people bad mouthing MS now are the same ones that did it when XP came out and did the same thing to 2000. If you really want to know how well Vista works, grab a trail and run it for yourself, be careful to make sure you have drivers for everything you need to work (this is the responsibility of the hardware vendors NOT Microsoft). Then you will be able to judge for yourself if Vista is right for you or not.
Well enough of my rate at this, back to the review, next up are my thoughts on PCMark Vantage.
{mospagebreak title=Conclusion}
Conclusion:
The problem with most synthetic testing suites is that they are only good for numbers. They do not come close to real-world usage. This is due to a synthetic benchmark’s inability to simulate all of the things that might be going on in any given situation. If you are running a gaming test using stock footage it is simply running the same video and sound loop over and over again. There are no AI calculations, no true particle effects are being run and physics calculations (if there are any) are all “canned” so while it can give you a repeatable number it is not a true measure of a system.
FutureMark has thought of that though, what they have done is to create a suite of tests that can run multiple threads (each one running a different calculation) simultaneously. This allows for much more of a real-world test and gives reproducible “numbers” that really do reflect what your system is capable of.
We here at Planetx64 will be using PCMark Vantage for our testing and to help provide you with more and more accurate evaluations.
Acknowledgments:
I would like to thank the team at FutureMark for helping us with this Evaluation and in particular Oliver Baltuch and ILkka Koho.
Discuss this article here