VMware ESX Server provides dynamic control over the amount of physical memory allocated to each virtual machine. You may overcommit memory, if you wish, so the total size configured for all running virtual machines exceeds the total amount of available physical memory. The system manages the allocation of memory to virtual machines automatically based on allocation parameters and system load.
You may specify initial memory allocation values for a virtual machine in its configuration file. You may also modify most memory allocation parameters dynamically using the VMware Management Interface, the procfs interface on the service console or the VMware Scripting API. Reasonable defaults are used automatically when parameters are not specified explicitly.
You have access to information about current memory allocations and other status information through the management interface, the procfs interface on the service console and the VMware Scripting API.
If you have server with NUMA architecture, be sure to see www.vmware.com/support/ esx2/doc/esx20admin_res-NUMA-intro.html. Also, for additional information on memory management by VMware ESX Server, see the mem(8) man page.
Three basic parameters control the allocation of memory resources to each virtual machine:
The minimum size is a guaranteed lower bound on the amount of memory that is allocated to the virtual machine, even when memory is overcommitted. The system uses an admission control policy to enforce this guarantee. You cannot power on a new virtual machine if there isn't sufficient memory to reserve its minimum size.
The maximum size is the amount of memory configured for use by the guest operating system running in the virtual machine. This maximum size must be specified in the configuration file for the virtual machine. By default, virtual machines operate at their maximum allocation, unless memory is overcommitted.
Memory shares entitle a virtual machine to a fraction of physical memory. For example, a virtual machine that has twice as many shares as another is generally entitled to consume twice as much memory, subject to their respective minimum and maximum constraints, provided they are both actively using the memory they have been allocated.
The system automatically allocates an amount of memory to each virtual machine somewhere between its minimum and maximum sizes based on its shares and an estimate of its recent working set size.
VMware ESX Server uses an admission control policy to ensure that sufficient unreserved memory and swap space are available before powering on a virtual machine. Memory must be reserved for the virtual machine's guaranteed minimum size; additional overhead memory is required for virtualization. Thus the total required for each virtual machine is the specified minimum plus overhead.
The overhead memory size is determined automatically; it is typically 54MB for a single virtual CPU virtual machine, and 64MB for a dual-virtual CPU SMP virtual machine. Additional overhead memory is reserved for virtual machines larger than 512MB.
Note: To create SMP virtual machines with ESX Server, you must also have purchased the VMware Virtual SMP product. For more information on the VMware Virtual SMP product, contact VMware, Inc. or your authorized sales representative.
Swap space must be reserved on disk for the remaining virtual machine memory - that is the difference between the maximum and minimum settings. This swap reservation is required to ensure the system is able to preserve virtual machine memory under any circumstances. In practice, only a small fraction of the swap space may actually be used.
Similarly, while memory reservations are used for admission control, actual memory allocations vary dynamically, and unused reservations are not wasted.
The amount of swap space configured for the system limits the maximum level of overcommitment. A default swap file size equal to the physical memory size of the computer is recommended in order to support a reasonable 2x level of memory overcommitment. You may configure larger or smaller swap files. If you do not configure a swap file, memory may not be overcommitted. You may configure the swap file using the VMware Management Interface (Swap Configuration in the Options page) or from the service console using the vmkfstools command.
Virtual machines are allocated their maximum memory size unless memory is overcommitted. When memory is overcommitted, each virtual machine is allocated an amount of memory somewhere between its minimum and maximum sizes. The amount of memory granted to a virtual machine above its minimum size may vary with the current memory load. The system automatically determines allocations for each virtual machine based on two factors: the number of shares it has been given and an estimate of its recent working set size.
ESX Server uses a modified proportional-share memory allocation policy. Memory shares entitle a virtual machine to a fraction of physical memory. For example, a virtual machine that has twice as many shares as another is entitled to consume twice as much memory, subject to their respective minimum and maximum constraints, provided that they are both actively using the memory they have been allocated. In general, a virtual machine with S memory shares in a system with an overall total of T shares is entitled to receive at least a fraction S/T of physical memory.
However, virtual machines that are not actively using their currently allocated memory automatically have their effective number of shares reduced, by levying a tax on idle memory. This "memory tax" helps prevent virtual machines from unproductively hoarding idle memory. A virtual machine is charged more for an idle page than for a page that it is actively using.
The MemIdleTax configuration option provides explicit control over the policy for reclaiming idle memory. You may use this option, together with the MemSamplePeriod configuration option, to control how the system reclaims memory.
ESX Server estimates the working set for a virtual machine automatically by monitoring memory activity over successive periods of virtual machine virtual time. Estimates are smoothed over several time periods using techniques that respond rapidly to increases in working set size and more slowly to decreases in working set size. This approach ensures that a virtual machine from which idle memory has been reclaimed is be able to ramp up quickly to its full share-based allocation once it starts using its memory more actively. You can modify the default monitoring period of 30 seconds by adjusting the MemSamplePeriod configuration option.
ESX Server employs two distinct techniques for dynamically expanding or contracting the amount of memory allocated to virtual machines - a VMware-supplied vmmemctl module that is loaded into the guest operating system running in a virtual machine and swapping pages from a virtual machine to a server swap file without any involvement by the guest operating system.
The preferred mechanism is the vmmemctl driver, which cooperates with the server to reclaim those pages that are considered least valuable by the guest operating system. This proprietary technique provides predictable performance that closely matches the behavior of a native system under similar memory constraints. It effectively increases or decreases memory pressure on the guest operating system, causing the guest to invoke its own native memory management algorithms. When memory is tight, the guest operating system decides which particular pages to reclaim and, if necessary, swaps them to its own virtual disk. The guest operating system must be configured with sufficient swap space. Some guest operating systems have additional limitations. If necessary, you can limit the amount of memory reclaimed using vmmemctl by setting the sched.mem.maxmemctl option. This option specifies the maximum amount of memory that may be reclaimed from a virtual machine in megabytes (MB).
Swapping is used to forcibly reclaim memory from a virtual machine when no vmmemctl driver is available. This may be the case if the vmmemctl driver was never installed, has been explicitly disabled, is not running (for example, while the guest operating system is booting) or is temporarily unable to reclaim memory quickly enough to satisfy current system demands. Standard demand paging techniques swap pages back in when the virtual machine needs them.
The vmmemctl approach is used whenever possible for optimum performance. swapping is a reliable mechanism of last resort that the system uses to reclaim memory only when necessary.
If you choose to overcommit memory with ESX Server, then you need to be sure your guest operating systems have sufficient swap space. This swap space must be greater than or equal to the difference between the virtual machine's maximum and minimum sizes.
Caution: If memory is overcommitted, and the guest operating system is configured with insufficient swap space, the guest operating system in the virtual machine may fail.
To prevent virtual machine failure, increase the swap size in your virtual machines:
For more information, refer to your Windows documentation or search the Windows help files for "paging files." Follow the instructions for changing the size of the virtual memory paging file.
Guest operating systems with large memory and small virtual disks (for example, a virtual machine with 3.6GB RAM and a 2 GB virtual disk) are more susceptible to this problem.
Many ESX Server workloads present opportunities for sharing memory across virtual machines. For example, several virtual machines may be running instances of the same guest operating system, have the same applications or components loaded, or contain common data. In such cases, ESX Server uses a proprietary transparent page sharing technique to securely eliminate redundant copies of memory pages. With memory sharing, a workload running in virtual machines often consumes less memory than it would when running on physical machines. As a result, higher levels of overcommitment can be supported efficiently.
The ESX Server approach does not require any cooperation from the guest operating system. You may use the MemShareScanVM and MemShareScanTotal configuration options to control the rate at which the system scans memory to identify opportunities for sharing memory.
The VMware Management Interface provides information on the current use of RAM by the physical computer and the virtual machines running on it. View the Status Monitor page in the management interface.
The System Summary section at the top shows systemwide information. The Virtual Machines section below it shows information for particular virtual machines.
You can also read the current memory statistics for a virtual machine from its status file on the service console. For example, to view the statistics for the virtual machine with ID 103, use this command:
cat /proc/vmware/vm/103/mem/status
The results are displayed in the following format:
vm mctl? wait shares min max size/sizetgt
103 yes no 2560 131072 262144 217300/ 217300
memctl/mctltgt swapped/swaptgt shared active overhd/ovhdmax
39168/ 39168 5672/ 5672 38164 191756 14508/ 32768
The output above is shown with additional line breaks, in order to avoid wrapping long lines. All memory sizes are reported in kilobytes; 1 megabyte = 1024KB. The columns indicate:
vm
|
Virtual machine identifier
|
mctl?
|
vmmemctl driver active?
|
wait
|
Blocked in a memory wait state?
|
shares
|
Memory shares associated with the virtual machine
|
min
|
Minimum size
|
max
|
Maximum size
|
size
|
Current size
|
sizetgt
|
Target size
|
memctl
|
Currently reclaimed using vmmemctl
|
mctltgt
|
Target to reclaim using vmmemctl
|
swapped
|
Currently swapped to VMFS swap file
|
swaptgt
|
Target to swap to VMFS swap file
|
shared
|
Memory shared via transparent page sharing
|
active
|
Current working set estimate
|
overhd
|
Current overhead memory size
|
ovhdmax
|
Maximum overhead memory size
|
In this example, the virtual machine with ID 103 is running the vmmemctl driver and is not currently blocked waiting for memory. The virtual machine is configured to use between 128MB and 256MB and has been allocated 2560 memory shares. It is currently allocated about 212MB. Approximately 44MB has been reclaimed for use by other virtual machines - 38MB via vmmemctl and nearly 6MB via swapping to the ESX server swap file. Of the 212MB allocated to the virtual machine, more than 37MB is shared - for example with other virtual machines. The current working set estimate for the virtual machine is approximately 187MB. About 14MB of overhead memory is currently being used for virtualization, out of a maximum of 32MB.
VMware supplies vmmemctl drivers for Windows Server 2003, Windows XP, Windows 2000, Windows NT 4.0, and Linux. The appropriate vmmemctl driver is installed automatically when you install VMware Tools in the guest operating system. The system uses swapping to reclaim memory from virtual machines running other guest operating systems and from virtual machines that do not have VMware Tools installed.
The maximum amount of memory that the system may attempt to reclaim using vmmemctl is restricted automatically based on known limitations of the type of guest operating system. Alternatively, you may specify the configuration file option sched.mem.maxmemctl manually. See the description of the ESX Server option MemCtlMax[OSType] for appropriate limits.
feature; you should upgrade the driver to the current version. Alternatively, you may specify the configuration file option sched.mem.maxmemctl manually. See the description of the ESX Server option MemCtlMax[OSType] for appropriate limits.