Updated: December 31, 2008

Running virtualized operating systems on top of existing hosts is a great idea. It's fun, it's cool, it allows you to experiment with software before committing to your real platform, it allows you to do things you would not normally dare, it gives you enormous flexibility, and it significantly reduces costs of deployment.

But the one thing that most virtualization software lacks is the 3D acceleration for guest operating systems. Or used to lack ... 3D acceleration in virtual machines has become a reality.

Introduction

In this series of articles, I will show you the simple hacks required to have 3D acceleration enabled in your guest operating systems! This means that your virtual machines will not be limited to 2D applications. You will be able to enjoy rich, interactive desktops. Best of all, you'll be able to play games inside virtual machines!

In this first article, we will learn how to enable DirectX support in VMware Server. The solution is also valid for VMware Player or VMware Workstation. The choice of the host operating system does not matter. The guest will have to be Windows.

In the second article, we will use OpenGL support in VirtualBox. Again, the choice of the host operating system is not important (either Windows or Linux will do), but the guest will have to run Windows. As you can see, both of these solutions are limited to having a Windows guest running, though. So, in the third article, we'll try VMGL, a solution that is supposed to give you 3D acceleration for Linux guests on top of Linux hosts.

A word of caution ...

Please note that all the mentioned features (and therefore, the solutions) are currently experimental. This means that the setup might work for you - or it might not. It also means less than perfect integration and performance.

3D virtualization is still in its infancy, but it's getting better all the time. In near future, we will have perfect, full-blown 3D graphics support that we desire. For now, let's see what we can do. OK, let's rock.

Wait! I know very little about virtualization!

Ah, yes, sorry. If you're new to virtualization, this article (or rather, this series) may be slightly advanced for you. Therefore, I recommend you spend some time reading the articles in my Virtualization section until you get comfortable enough with the topics. 
In particular, the article that you should pay attention to:

How to install VMware Tools - Tutorial

Installing Windows XP - Full tutorial - Another article that may help you obtain what you need; this will teach you how to install Windows XP, a procedure that is identical for both real hosts and virtual machines. Nevertheless, as always, I'll try to make the tutorial as easy and detailed as possible, so it can be followed by just about anyone.

VMware Server & DirectX

What do we need?

You need a VMware product - Player, Server, Workstation, one of these. They are all mutually exclusive, so only one can be installed at any given time. For this tutorial, I have chosen:

1. VMware Server 1.0.5 on Windows XP3 as the host (RE: Windows host example)
2. VMware Server 1.0.8 on openSUSE 11.0 as the host (RE: Linux host example)
3. Windows XP SP3 as the guest operating system in both cases
4. VMware Tools installed in the guest operating system
5. A small tweak of the virtual machine configuration file
6. A game that requires DirectX to test the functionality

Note: If you intend to use the VMware Player for the task, make sure your Windows virtual machine already has the VMware Tools installed.

So, assuming that you have everything ready (steps 1-4), let us tweak the virtual machine configuration file. If you're unsure how to complete steps 1-4, please refer to my Virtualization section.

Enable 3D acceleration for the virtual machine

Virtual machines have configuration files. These can be identified by the .vmx extension. The .vmx files are simple text files containing directives, written in the syntax that the VMware programs can interpret and use. In the configuration files, we define how many CPUs our guest will use, how much RAM, hard disk etc.

Make changes to the configuration file only when the virtual machine is powered off! Open the folder containing your virtual machines (and their configuration files), find the relevant .vmx file and open it in a text editor.

So now, we need to add a section that enables 3D acceleration.

This is the first line we need:

This enables the 3D acceleration.

This is the second, optional line we need:

It specifies how much memory we want to dedicate to our virtual GPU. In my example, this is 64MB. If you want to be precise, let's say 32MB, then your calculation should be 32 x 1024 x 1024, because 1KB is 1024 bytes. The maximum is 128MB.

This is the third, optional line. It disables mouse integration. It is important for certain games, like full-screen real-time First Person Shooters (FPS).

Along with the third option, you should turn off motion grabbing for your virtual machines. In the main menu of the VMware (product) console, click Edit > Preferences, select the Input tab and under Cursor, untick Ungrab when cursor leaves window.

If you're wondering what this does, it will prevent your games from remembering the mouse motion when it leaves the virtual machine. So if you "ungrab" the mouse by moving it out of the virtual screen, let's say, on the right side, your game won't continue pushing to the right. In games where the mouse movement is used for map scrolling, it won't keep your maps scrolling. In games where the mouse movement is used for motion and targeting, like FPS, it won't keep your character spinning around like a record. Now, who was it that sang this? Dead or Alive or Youthquake?

Finally, your file should contain a section like this:

We're ready, let's boot and test.

Windows host, Windows guest

Boot virtual machine

Once booted, make sure you have the VMware Tools installed and running. You will know this by looking for a VMware icon in the system tray.

Next, run dxdiag to get the diagnostics for your display (Start > Run > dxdiag):

Pay attention to DirectX Features. We have DirectDraw and Direct3D enabled. AGP Texture Acceleration is not available, though. Good enough. Not the best, but good enough. Under Device above, we can see we have a VMware driver installed and it has 64MB RAM.

Test game

Our candidates were Max Payne and Scorched3D:

They worked OK. The performance was reasonable. It was not as smooth or fast as a native platform, but it's a step in the right direction.

Linux host, Windows guest

Similarly, it worked on the Linux machine. Not only that, I connected to a remote Windows machine, where the virtual machine was located, and ran it from there, using Wireless connection!

Conclusion

The setup is simple. 3D acceleration works. Currently, it is approx. equivalent to what you'd get running a GeForce 2/4 card. Maybe a little more. But think of this as a child, growing up to become a 3D bully. This is a tremendous and important step for all hardcore fans of virtualization. In the next article, we'll see the same done using VirtualBox and OpenGL applications.