This objective focuses on host design, what sort of hardware to choose, what is the amount if CPU and Memory required and so on….
Before you decide what sort of Hardware you should purchase, you need to take a Current State Analysis:
- How many VMs?
- How much total CPU load?
- How much total RAM load?
- How much total disk capacity?
- How much totalk Disk performance?
- How much total Network Performance?
Use 2 socket, 12 cores as the minimum standard
- Use 4 sockets in large farm where have more than 5000 VMs
- Use 4 socket for large VMs
- If your largest VM needs >6 core, go for 8 core pCPU.
- Ideally, a VM should fit inside socket to minimise NUMA effect
- If your largest VM needs 44 GB of RAM, then each socket should have 48 GB
- ESXi host should be >2x than your largest VM
Factor in the future demand, because hardware once implemented is not easy to change:
- How much more CPU/RAM do you need in future?
- Cyclic Peak?
- Purchasing lead time for unplanned demand?
- Silos for security or political reasons?
One of my very good mate Iwan Rahabok has prepared an excellent slide, let me share the CPU and RAM selection details:
Look at overall cost, not just the cost of ESX host
- Larger host = larger VM or more VM/host.
Xeon 5600 delivers 9x improvement over Xeon 5100
- Clock speed only improves by 0.1x and # core by 3x
- Fujitsu delivers VMmark result of 38.39 at 27 tiles on July 2010
AMD Opteron at 2 sockets 24 cores is 32.44@22 tiles. An impressive number too.
- Xeon is around 18% faster, not a huge margin anymore.
- But Xeon uses 12 cores, while Opteron uses 24 cores. So each core is around 2x faster.
Recommendation (for Intel)
- Use Xeon 2803 if budget is the constraint and you don’t need to run >6 vCPU VM.
- Use Xeon 2820 if you need to run 8 vCPU VM.
- Use Xeon 2850 if you need to run 10 vCPU VM
Recommendation (for Intel, 4 socket box)
- Use 4807, then 4820, then 4850
Do not buy the highest end CPU
- Premium too high. Use the savings and buy more hosts instead
How much RAM? It depends on the # core in previous slide.
- Not so simple anymore. Each vendor is different.
- As at Nov 2011, 8 GB DIMM is cheaper than 2x 4 GB DIMM.
- 8 GB per core. So 12 core means around 96 GB.
- Consider the channel best practice
- Don’t leave some empty. This bring benefits of memory interleaving.
- Check with the server vendor on the specific model.
- Some models now comes with 16 slots per socket, so you might be able to use lower DIMM size.
- Some vendors like HP has similar price between 4 GB and 8 GB.
- Dell R710 has 18 DIMM slots (?)
- IBM x3650 M3 has 18 DIMM slots
- HP DL 380 G7 and BL490cG6/G7 have 18 DIMM slots
- Cisco has multiple models
VMkernel has Home Node concept in NUMA system.
For ideal performance, fit a VM within 1 CPU-RAM “pair” to avoid “remote memory” effect.
- # of vCPUs + 1 <= # of cores in 1 socket. So running a 5 vCPU VM in a quad-core will force remote memory situation
- VM memory <= memory of one node
Turn on Large Page, especially for Tier 1.
- This means no TPS. That’s why the RAM sizing is relatively large
- Need application-level support
Total GB for all VMs + headroom / Total GB per host = Minimum host count for workload
ESXi Host: IO & Management:
- 2 FC ports and 10-12 GE NIC ports
- Get 2x quad-port NIC. Since the built-in NIC is either 2-4, the box will end up with 10 – 12 ports
- Not supported: mixing hardware iSCSI and software iSCSI
Use a NIC that supports the following:
- Checksum offload
- Capability to handle high memory DMA (64-bit DMA addresses)
- Capability to handle multiple scatter/gather elements per Tx frame
- Jumbo frames
- We require NetQueue to achieve 10 GE inside the VM. Without it, VM gets only 3–4 Gbps
- Lights-out management, So you don’t have to be in front of physical server to do certain thing (e.g. go into CLI as requested by VMware Support)
- Hardware agent is properly configured, Very important to monitor hardware health due to many VMs in 1 box
PCI Slot on the motherboard
- Since we are using 8 Gb FC HBA, make sure the physical PCI-E slot has sufficient bandwidth
- A single-dual port FC port makes more sense if the saving is high and you need the slot. But there is a risk of bus failure. Also, double check to ensure the chip can handle the throughput of both ports
- If you are using blade, and have to settle for a single 2-port HBA (instead of two 1-port HBA), then ensure the PCI slot has bandwidth for 16 Gb. When using a dual-port HBA, ensure the chip & bus in the HBA can handle the peak load of 16 Gb
Blades vs Rack Mount Servers
- Blades have less space inside
- Less I/O of RAM Slots
- Cost Doesn’t scale linearly, $50K for new enclosure
- take up more space
- Require more cables
- Less power efficient
Scale Up vs Scale Out:
- Fewer larger Servers
- Accomodates larger VMs
- Larger impact for failure
- For large VMs
- More smaller servers
- Fewer VMs affected by host failure
- Easier growth
vSphere Auto Deploy: vSphere Auto Deploy can provision hundreds of physical hosts with ESXi software. You can specify the image to deploy and the hosts to provision with the image. Optionally, you can specify host profiles to apply to the hosts, and a vCenter Server location (folder or cluster) for each host. Auto Deploy requires the following to work:
- vCenter Server
- Install Auto Deploy Server, preferably on vCenter Server
- Install PowerCLI, before that ensure you install windows .Net 2 and PowerShell Version 2
- Setup a TFPT server
- DHCP Scope with scope options 66 and 67
Best Practices around Auto Deploy can be found here