I would like to play 3D games in a virtual machine

Quick answer: don't.

VMware Server or ESX do not have 3D support. Not at all.

VMware Workstation and Player have limited 3D support. They support 99% of DirectX 8.1. Some old games may work. VMware Fusion also has experimental support for DirectX 9 (without shaders). There are examples of working games on YouTube.

If you want to play games, rather dual boot, or if you are a Linux user, check out something like Cedega from http://www.transgaming.com/.

See also:

http://www.vmware.com/support/ws55/doc/ws_vidsound_d3d_enabling_host.html
http://www.vmware.com/support/ws55/doc/ws_vidsound_d3d_enabling_vm.html