Objective 3.4 – Determine Appropriate Compute Resources for a vSphere 5 Physical Design

This objective focuses on host design, what sort of hardware to choose, what is the amount if CPU and Memory required and so on….

Before you decide what sort of Hardware you should purchase, you need to take a Current State Analysis:

  • How many VMs?
  • How much total CPU load?
  • How much total RAM load?
  • How much total disk capacity?
  • How much totalk Disk performance?
  • How much total Network Performance?

Use 2 socket, 12 cores as the minimum standard

  • Use 4 sockets in large farm where have more than 5000 VMs
  • Use 4 socket for large VMs
  • If your largest VM needs >6 core, go for 8 core pCPU.
  • Ideally, a VM should fit inside socket to minimise NUMA effect
  • If your largest VM needs 44 GB of RAM, then each socket should have 48 GB
  • ESXi host should be >2x than your largest VM

Factor in the future demand, because hardware once implemented is not easy to change:

  • How much more CPU/RAM do you need in future?
  • Growth?
  • Cyclic Peak?
  • Purchasing lead time for unplanned demand?
  • Silos for security or political reasons?

One of my very good mate Iwan Rahabok has prepared an excellent slide, let me share the CPU and RAM selection details:


Look at overall cost, not just the cost of ESX host

  • Larger host = larger VM or more VM/host.

Xeon 5600 delivers 9x improvement over Xeon 5100

  • Clock speed only improves by 0.1x and # core by 3x
  • Fujitsu delivers VMmark result of 38.39 at 27 tiles on July 2010

AMD Opteron at 2 sockets 24 cores is 32.44@22 tiles. An impressive number too.

  • Xeon is around 18% faster, not a huge margin anymore.
  • But Xeon uses 12 cores, while Opteron uses 24 cores. So each core is around 2x faster.

Recommendation (for Intel)

  • Use Xeon 2803 if budget is the constraint and you don’t need to run >6 vCPU VM.
  • Use Xeon 2820 if you need to run 8 vCPU VM.
  • Use Xeon 2850 if you need to run 10 vCPU VM

Recommendation (for Intel, 4 socket box)

  • Use 4807, then 4820, then 4850

Do not buy the highest end CPU

  • Premium too high. Use the savings and buy more hosts instead
Total MHz for all VMs + Headroom / Total MHz per host = Minimum house count for workload


How much RAM? It depends on the # core in previous slide.

  • Not so simple anymore. Each vendor is different.
  • As at Nov 2011, 8 GB DIMM is cheaper than 2x 4 GB DIMM.
  • 8 GB per core. So 12 core means around 96 GB.
  • Consider the channel best practice
  • Don’t leave some empty. This bring benefits of memory interleaving.
  • Check with the server vendor on the specific model.
  • Some models now comes with 16 slots per socket, so you might be able to use lower DIMM size.
  • Some vendors like HP has similar price between 4 GB and 8 GB.
  • Dell R710 has 18 DIMM slots (?)
  • IBM x3650 M3 has 18 DIMM slots
  • HP DL 380 G7 and BL490cG6/G7 have 18 DIMM slots
  • Cisco has multiple models

VMkernel has Home Node concept in NUMA system.
For ideal performance, fit a VM within 1 CPU-RAM “pair” to avoid “remote memory” effect.

  • # of vCPUs + 1 <= # of cores in 1 socket.  So running a 5 vCPU VM in a quad-core will force remote memory situation
  • VM memory <= memory of one node

Turn on Large Page, especially for Tier 1.

  • This means no TPS. That’s why the RAM sizing is relatively large
  • Need application-level support

Total GB for all VMs + headroom / Total GB per host = Minimum host count for workload

ESXi Host: IO & Management:

I/O card

  • 2 FC ports and 10-12 GE NIC ports
  • Get 2x quad-port NIC. Since the built-in NIC is either 2-4, the box will end up with 10 – 12 ports
  • Not supported: mixing hardware iSCSI and software iSCSI

Use a NIC that supports the following:

  • Checksum offload
  • Capability to handle high memory DMA (64-bit DMA addresses)
  • Capability to handle multiple scatter/gather elements per Tx frame
  • Jumbo frames
  • We require NetQueue to achieve 10 GE inside the VM. Without it, VM gets only 3–4 Gbps


  • Lights-out management, So you don’t have to be in front of physical server to do certain thing (e.g. go into CLI as requested by VMware Support)
  • Hardware agent is properly configured, Very important to monitor hardware health due to many VMs in 1 box

PCI Slot on the motherboard

  • Since we are using 8 Gb FC HBA, make sure the physical PCI-E slot has sufficient bandwidth
  • A single-dual port FC port makes more sense if the saving is high and you need the slot. But there is a risk of bus failure. Also, double check to ensure the chip can handle the throughput of both ports
  • If you are using blade, and have to settle for a single 2-port HBA (instead of two 1-port HBA), then ensure the PCI slot has bandwidth for 16 Gb. When using a dual-port HBA, ensure the chip & bus in the HBA can handle the peak load of 16 Gb

Blades vs Rack Mount Servers

  • Blades have less space inside
  • Less I/O of RAM Slots
  • Cost Doesn’t scale linearly, $50K for new enclosure

Rack Servers

  • take up more space
  • Require more cables
  • Less power efficient

Scale Up vs Scale Out:

Scale Up:

  • Fewer larger Servers
  • Accomodates larger VMs
  • Larger impact for failure
  • For large VMs

Scale out:

  • More smaller servers
  • Fewer VMs affected by host failure
  • Easier growth

vSphere Auto Deploy: vSphere Auto Deploy can provision hundreds of physical hosts with ESXi software. You can specify the image to deploy and the hosts to provision with the image. Optionally, you can specify host profiles to apply to the hosts, and a vCenter Server location (folder or cluster) for each host. Auto Deploy requires the following to work:

  • vCenter Server
  • Install Auto Deploy Server, preferably on vCenter Server
  • Install PowerCLI, before that ensure you install windows .Net 2 and PowerShell Version 2
  • Setup a TFPT server
  • DHCP Scope with scope options 66 and 67

Best Practices around Auto Deploy can be found here

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s