All New Gaming Test Suite

On the gaming side, the changes are rather more substantial. We’ve decided to wipe the slate clean and select all new titles for 2012. Actually, that’s not entirely true—I’ve been running tests with Civilization V and Total War: Shogun 2 for a while, but we’re now updating the settings for the benchmarks and all laptop editors will test with the following games and settings. Note that we tried to get a good selection of game genres in the list; depending on how you want to classify the games, we have four games representing first person/third person action games, two strategy games, one RPG, and one simulation/driving game. We also have representatives of several major engines—Unreal Engine 3, Frostbite 2, and Source being the most noteworthy. We’ve tried to overlap our desktop gaming suite, and while we won’t use identical test suites, we do overlap on six of the titles.

The other big change is that we’re ramping up the quality settings this year. Previously, we had both Low/Minimum and Medium settings at 1366x768. Unfortunately, we often ran into games where minimum quality looked, frankly, awful; in other games the difference between our Low and Medium settings ended up being negligible. For 2012, then, we’ve decided to skip the “minimum” detail testing and select settings that we feel most gamers will actually like using. [Update: we're changing the naming convention to avoid name space conflicts.]

Our Value setting for the test suite loosely corresponds to last year’s “Medium” settings, all run at 1366x768; our new Mainstream settings bump the resolution to 1600x900 and increase quality to roughly match last year’s “High” settings; finally, our Enthusiast settings enable 4x MSAA in all seven titles and increase the resolution to 1920x1080, basically matching the “Ultra” settings of 2011. We’ve tested each game on several setups with the goal of choosing settings that will result in reasonable quality and performance differences between the three settings. With that out of the way, here’s a rundown of the games.

Batman: Arkham City: The sequel to 2009’s Batman: Arkham Asylum, Arkham City continues the story of the Dark Knight with a free roaming playground to explore at your leisure. Graphically, the game is similar to the original title, only now the PC version has (properly working) DX11 support. The DX11 features come at a serious performance cost, however, so our Value test setting will leave them off. Note that even in the game settings, DX11 features are disabled unless you choose the maximum “Extreme” preset—and you’ll really need a beefy PC to handle the workload at that point!

We’re using the built-in benchmark so that readers can compare our results with their own hardware. Our Value settings use the Medium defaults; for Mainstream we switch to the High defaults and enable DX11 features; finally, for Enthusiast we use the Extreme preset and enable 4x MSAA. If you’re wondering, leaving DX11 disabled largely removes performance differences between the various settings, at least on moderate hardware. We tested at 1366x768 with Low, Medium, High, and Very High settings and found the average frame rates only dropped around 20%; enabling DX11 on any of the other modes results in a drop of around 40%.

Also worth noting is that Batman supports PhysX, but we won’t be testing with PhysX as that’s only available on NVIDIA hardware. That said, we do want to mention that PhysX definitely improves the gaming experience, with enhanced fog effects, more debris, cloth effects, and certain weapons (e.g. the freeze gun in one scene) fire polygons instead of sprites/textures. My own impression is that Batman with PhysX enabled and DX11 disabled generally looks better than Batman with DX11 enabled and PhysX disabled. If you want all of the goodies enabled, you’ll need very high-end hardware, beyond what most laptops can support—we’d suggest GTX 560M SLI as a bare minimum for the Extreme preset with PhysX enabled at 1080p.

Battlefield 3: We’re switching our Battlefield choice from Bad Company 2 to Battlefield 3, though in practice performance is frequently similar. For this title, we’re using FRAPS and using a two minute tank “on rails” sequence from the Thunder Run single player mission. Performance in the single player missions is highly variable depending on the level, and multi-player is even more so, but we need something that provides consistency between test runs. The Frostbite 2 engine puts quite a hefty load on your GPU, and if you want all the eye candy enabled you’ll need more than your typical mobile GPU. For BF3, our Value settings use the Medium preset; Mainstream uses the High preset; and Enthusiast uses the Ultra preset. BF3 also supports DX10 and DX11, and we leave the DirectX version support set to “Auto”; outside of Intel’s HD 2000/3000 hardware, that means all laptops will run in DX11 mode.

Civilization V: Civ5 is an interesting title in that the use of driver command lists allowed NVIDIA to optimize performance and get a healthy boost in frame rates not long after it launched. AMD has yet to implement command lists (AFAIK), but as we showed in our HD 7970 review, there may be other factors at play. We continue to use the built-in LateGameView benchmark, and it’s worth noting that the turn-based nature of Civ5 makes lower frame rates more palatable than in shooters. For our detail settings, Value has all of the video settings at Low; Mainstream uses High settings on everything (with the High detail strategic view enabled); Enthusiast is the same, only with 4x MSAA enabled. We use the DX10/11 executable and set the configuration file to allow the use of both SM41 and SM50 (Shader Model 4.1/DX10.1 and Shader Model 5.0/DX11).

DiRT 3: Our replacement to DiRT 2 is a simple update to the latest title in the series. As with BF3, this time we’re letting all systems use DX11 hardware—early indications are that DX11 improves performance at Low to High presets, but it creates a pretty massive performance drop at the Ultra preset. We run the in-game benchmark. Our Value setting will use the Medium preset; Mainstream will use the High preset, and Enthusiast will use the Ultra preset with 4x MSAA. (Note that just moving the detail slider from High to Ultra results in a ~40% drop in frame rate while adding 4xAA accounts for another 10-15%, so there’s a pretty sizeable gap between our Mainstream and Enthusiast results.)

Elder Scrolls: Skyrim: Skyrim is one of two titles in our updated list that doesn’t (currently) support DX11. There may be a patch at some point to improve the situation—there are some old conflicting statements from earlier this year where Skyrim was claimed to support DX11—but for now we’re using whatever the game has in the latest patch (e.g. as of early January, 2012). Texture quality is not one of the strong points of Skyrim, with frequently blurry textures (thanks to the console cross-platform nature of development), but at least dragons are now properly attacking with the latest updates.

As far as benchmarking goes, Skyrim appears to be far more taxing on the CPU side of things than on graphics, particularly for desktop gamers, but mobile graphics hardware is several rungs down the performance ladder so we’re going to use it. Our Value setting uses the Medium preset with antialiasing off, anisotropic filtering set to 4x, and texture quality set to medium with FXAA disabled—the latter basically uses a full screen blur filter to remove jaggies while increasing blurriness. For Mainstream, we use the High preset and turn antialiasing off. Last, for Enthusiast, we use the Ultra preset but drop antialiasing to 4xAA. Note that enabling antialiasing, at least on a GTX 560M, appears to have a minimal impact on performance; however, that may not always be the case so we’re sticking with our standard of no-AA at Value and Mainstream settings and 4xAA at the Enthusiast settings.

Update: The 1.4 patch of Skyrim dramatically improved performance, and Bethesda also released a high resolution texture pack for the PC. We will use the high resolution texture pack at the Mainstream and Enthusiast settings going forward.

Portal 2: Portal 2 is our representative of the Source engine, and like the other Source games released so far from Valve, that means no DX10 or DX11 support. That doesn’t mean the game isn’t graphically demanding, though it may have different bottlenecks than many of the other titles. We use an in-house demo file where the player combines speed gel with portals, switches, and an Excursion Funnel to advance through the map. Like most Source engine games, frame rates tend to be quite a bit higher than other titles. Our Value settings use trilinear filtering with multicore rendering enabled, and Shader/Effect/Model/Texture detail are all at Medium (with paged pool memory available set to High). Mainstream maxes out all of the settings with the exception of antialiasing, which remains off, and Enthusiast adds 4x MSAA to the mix.

Total War: Shogun 2: Wrapping up our gaming list is Total War: Shogun 2, a game which holds the dubious honor of being the slowest loading title in our test suite—by a large margin. Initially launched as a DX9 title, a patch later added DX10/11 support. Graphically, it’s difficult to tell what differences the various rendering modes have, but DX11/SM5.0 does appear to have substantially better SSAO. The patch that added DX11 features also added a built-in benchmark, the introduction to one of the scenarios, which we use for our testing. Our Value settings use the Medium preset, Mainstream will use the High preset, and Enthusiast uses the Very High preset with 4x MSAA enabled. In addition to enabling the DX11 engine, all of our settings files are set to use SM5.0 code where applicable.

2012: Meet Our New Mobile Benchmark Suite Benchmarks and Closing Thoughts


View All Comments

  • Ryan Smith - Saturday, January 7, 2012 - link

    Brink's really not any better. It's a multiplayer game that no one plays (and I say that as an owner), and because it's a MP game there's no practical way to structure a repeatable benchmark.

    We would have definitely liked to include an OpenGL game, but even I have to admit that OpenGL just isn't very important right now. Maybe Doom 4 or Prey 2 will change that, but with id not licensing Tech 5, OpenGL is quickly becoming an API that's only applicable to id games.
  • jjj - Friday, January 6, 2012 - link

    "as where many wouldn’t notice the difference between a web page loading in two seconds and a web page loading in one second"

    That's not all that true,there are some actual numbers from Google and Amazon about how page loading time relates to sales/searches:
    -Amazon : every 100 ms increase in load time of decreased sales by 1%
    -Google: a change in a 10-result page loading in 0.4 seconds to a 30-result page loading in 0.9 seconds decreased traffic and ad revenues by 20%.
    In the end it might be a more usefull test than some synthetic tests already on the list.

    Intel low power CPUs choke when both GPU and CPU are active and maybe the 17W Trinity SKUs will too.Maybe there should be a test that reflects that,besides the gaming ones since it's a rather important piece of info.

    Finally,the most important test that is missing is WIFI perf,it is a mobile device after all.
  • Conficio - Friday, January 6, 2012 - link

    It's so funny that Google knows about the numbers and does even provide tools to measure the load times and make suggestions how to improve. However when I run the performance analysis against one of my websites, some of the suggestions to improve are for the Google Analytics, and Google Adsense scripts (like not allowing caching of the script, or only allowing for less than a week, or scripts loaded from redirections, etc.).

    I also see many web pages waiting for serves that load google AdSense or analytics.

    Is it just me, or should Google start to eat it's own dog food?
  • JarredWalton - Friday, January 6, 2012 - link

    What would you like us to test with WiFi? Maximum throughput? Part of that will be determined by the chipset, a larger part more than likely will come from the choice of router, and the rest is noise and interference between runs. I do make a point of listing the WiFi chipset, which tells you about 90% of what you need to know.

    (Hint: 1x1:1 MIMO solutions are the bottom of the barrel and come in most laptops; the 2x2:2 solutions are okay, and if you have a 5GHz router that can really help remove interference. I've only tested about three laptops with 3x3:3 MIMO over the years, and sadly my current router can't even support the maximum throughput.)
  • jjj - Saturday, January 7, 2012 - link

    You guys are already testing WIFI for phones and tablets,it's easy to just apply the same methodology.I do have a hard time understanding why do it there and not here and this internet thing is something that tends to be used a lot.
    I guess testing notebooks started by doing the same thing as on desktops and it didn't seemed obvious that this should be tested. Listing the part used helps only folks that know what it means and you get what you expect only if the OEM does things right. I don't have a laptop example but look at Asus Transformer Prime and it's WIFI and GPS problems, can you be sure that there aren't a bunch of laptops that offer a lot less than they should? Poor WIFI perf could also be a deal breaker for many,if they knew about it and maybe,just maybe,some manufacturers would pay more attention to it.
  • JarredWalton - Saturday, January 7, 2012 - link

    For smartphones, you're looking at 1x1:1 2.4GHz solutions with a very small device that can easily run into connectivity problems if, for instance, the casing is all aluminum. For laptops, it's usually not much of a concern (e.g. they're plenty large to route wires in a sensible fashion for the antennae). WiFi speeds usually aren't a concern unless you're transferring large files. If you're doing that, then you're going to typically be at the limits of the chipset, not the laptop, and you'd be far better off with Ethernet regardless.

    Anyway, there are other problems with trying to test laptop wireless speeds. One is that we have at least two different test locations, so that necessitates getting identical hardware on both sides -- router for sure, and testing location won't be identical. Even with the right hardware, outside interference (from neighboring networks) is a potential problem.

    The better solution IMO is a separate review of wireless chipsets. I tried to do that with the Killer Wireless-N review a while back, and Brian is working on a more complete roundup of wireless chipsets. Outside of that review, I'll see what Anand thinks about getting us equipped with the necessary hardware to test wireless.
  • jjj - Saturday, January 7, 2012 - link

    maybe this helps
    Anyway design matters here too and ofc it impacts range too not only throughput.For using Ethernet,you don't always have it,maybe you use your laptop mostly as a desktop replacement but in the end that's not it's main purose for many.
    The differences are there and i don't know how many folks wouldn't care at least about range.
  • jalexoid - Friday, January 6, 2012 - link

    Please, please, please add OpenGL benchmarks for professional users. OpenCL would be quite good also, specifically data transfer between the main mem to graphics memory. Reply
  • JarredWalton - Saturday, January 7, 2012 - link

    We generally run SpecViewPerf on workstation GPUs (Quadro FX), but is there really a desire to see those tests run on consumer graphics as well? Even a basic Quadro FX will generally outperform the fastest consumer cards in professional OpenGL testing, simply because of the drivers/firmware.

    For OpenCL, Ryan tests that on GPUs to some extent, but I'm not sure how many people are seriously looking at OpenCL performance on laptops. Would you suggest using the same tests as Ryan is using, or do you have some specific OpenCL benchmark you'd like us to run?
  • ltcommanderdata - Saturday, January 7, 2012 - link

    PowerDirector 10 uses OpenCL to render it's video effects. Video editing seems like a good use case that most users can relate to rather than say fluid simulation or Monte Carlo calculations.

    PowerDirector 10 also supports AMD APP Acceleration (which is OpenCL I suppose), nVidia CUDA, and Intel QuickSync for final encoding so could be useful to compare each platform's ideal accelerated encoding method.

    The upcoming WinZip 16.5 is supposed to be OpenCL accelerated for compression, decompression, and encryption making another benchmark with a use case that is applicable to most users.

Log in

Don't have an account? Sign up now