Hi MagicHut,
Lot of good info. We ran into this with pairs of two ESXi vSphere v5.1 hosts for Oracle as a VM performance benchmarking when we separated our OLTP database to one pair, and the DW database onto the other pair. Oracle recommendation is to properly determine your Oracle SGA required space, then create a VM reservation equal to the Oracle SGA. In our case, we had 5 environments -- 1 Production, and 4 various versions of "non-production (QA, DEV, etc)" -- and each environment had two VMs -- one for DW VM destined for pair #1, and one for OLTP VM destined for pair #2. All VMs were running RHEL 6.3 (FYI) and the ".vmx" was on the vmdk used for RH Root (/) partition. Out of scope for this answer is the RH Swap, Oracle software binary and a detailed strategy for optimized ASM performance on our new SAN.
DW Pair#1, each host has 256GB, 5 VMs, each DW VM with 64GB vRAM, but only 42GB SGA
OLTP Pair#2, each host has 128GB, 5 VMs, each OLTP VM with 32GB vRAM, but only 22GB SGA.
Initially, we created 80GB LUN with a 60GB DataStore for RHROOT. That left 20GB left over. [ Note we had an issue here because late in the game in January RHROOT went from 32GB to 60GB ].
We created all 10 VMs quickly (quite an assembly line -- build RH VM from GOLD image, install Oracle, tune it, place on ESXi target host).
We went to deploy, the plan was to reduce the 8 non-production VMs down to the Oracle SGA only
When we attempted to reduce the VM memory reservation on a Data Warehouse VM, we got a vSphere kernel panic because ESXi had no place to put the 22GB (64GB vRAM - 42GB SGA) -- see math calculation below:
MATH: VM Overhead when memory reservation set to unlimited:
RHROOT vmdk was 60GB as expected
I confirmed that the average .vmx vswp size was 112MB to 116MB
Remaining files were about 1-2 MB
Free space was around 18.9 GB <=== TOO SMALL TO HOLD 22GB .VSWP file
Normal operations for VM placement on across four ESXi hosts (2 separate pairs, LUNs visible only within pair)
PAIR#1 (256GB physical RAM per host)
DW01 --- only production DW VM, 64GB vRAM, 64GB memory reservation (set to unlimited)
64GB of 256GB used, no default plan for non-production VMs to restart on DW01
DW02 --- four non-production DW VMs, each 64GB vRAM, planned 42GB memory reservation
Without SGA-based reservation: 256GB of 256GB used
With SGA-based reservation: 168GB of 256GB used;
SGA-based res. requires 80GB LUN to be expanded to 140GB
Why? allows up to 0% reservation
Default plan is for PROD VM to immediately failover to DW02 ESXi host as needed (maint, other).
OLTP01 --- only production OLTP VM, 32GB vRAM, 32GB memory reservation (set to unlimited)
32GB of 128GB used, no default plan for non-production VMs to restart on OLTP01
OLTP02 --- four non-production OLTP VMs, each 32GB vRAM, planned 22GB memory reservation
Without SGA-based reservation: 128GB of 128GB used
With SGA-based reservation: 88GB of 128GB used;
SGA-based res. requires 80GB LUN to be expanded to 140GB
Why? allows up to 0% reservation -AND- OLTP can expand from 32GB to 64GB in future
Default plan is for PROD VM to immediately failover to DW02 ESXi host as needed (maint, other).
OPTIONS / SOLUTION:
Note: With vSphere v5.1, we set the VMs running RHEL 6.3 to allow hot-add of memory and CPUs.
Option 1: Local ESXi swap: We have 300GB unused mirrored drives on each ESXi host
Decision: Rejected. We didn't want solution to catch up with us if we added more DB VMs
Option 2: Expand Data Store where 1st vmdk / .vmx file
Decision: Rejected. For VM replica reasons at SAN, we have LUN per DataStore and the LUN was too small.
Option 3: On SAN, create 140GB LUNs and perform transparent LUN migration from 80GB LUN.
[Requires other steps, expanding data store, changing reservation not discussed here].
Decision: Accepted. We dedicated an entire EMC VNX 5300 for these 10 VMs and this was a good use.
We don't expect any problems with option 3. We want future flexibility since we have been launching these environments in stages and the last ones will be launched on 5/31.
l a rr y -dot- t i m mi ns <at> g m a il. co m for more info.
Larry