A lot of VMware vSphere architects and engineers are designing their vSphere clusters for some overbooking ratios to define some level of the service (SLA or OLA) and differentiate between different compute tiers. They usually want to achieve something like
- Tier 1 cluster (mission-critical applications) - 1:1 vCPU / pCPU ratio
- Tier 2 cluster (business-critical applications) - 3:1 vCPU / pCPU ratio
- Tier 3 cluster (supporting applications) - 5:1 vCPU / pCPU ratio
- Tier 4 cluster (virtual desktops) - 10:1 vCPU / pCPU ratio
Terminology:
- vCPU - virtual CPU configured for Virtual Machine
- pCPU - physical CPU core (hyper-threading is not considered)
Before vSphere 6.5 we have to monitor it externally by vROps or some other monitoring tool. Some time ago I have blogged how to achieve it with PowerCLI and LogInsight - ESXi host vCPU/pCPU reporting via PowerCLI to LogInsight.
vSphere 6.5 DRS has introduced additional option to set maximum CPU over-commitment. It limits the number of vCPUs per pCPU in particular DRS cluster. However, it is good to know that there are two different advanced DRS options (configuration parameters) how to specify vCPU:pCPU ratio and each setting behaves differently. See table below ...
It is worth to mention a little bit tricky setting of these additional options via GUI. It is good to know how GUI setting of "CPU Over-Commitment" is mapped to DRS cluster advanced options.
In my lab, I have VCSA 6.5 U1c (build 7119157) so I did some tests.
If I set "CPU Over-Commitment" in vSphere Web Client (Flash/Flex) it sets MaxVcpusPerClusterPct so it is the setting per the whole vSphere Cluster.
However, in vSphere Client (HTML5) it sets MaxVCPUsPerCore so it is per ESXi host.
Therefore, it is good to know what you would like to achieve and double check DRS Advanced Options.
See different behavior in screenshots below
So this is how it should work. Now let's do some test to understand real behavior.
TEST 1: MaxVcpusPerClusterPct = 0
Let's set MaxVcpusPerClusterPct to 0 so we are saying to allow 0 : 1 vCPU / pCPU ratio. In other words, no VM can be running in the cluster. And it works as expected. When I try to run VM I get the error "The total number of virtual CPUs present or requested in virtual machines' configuration has exceeded the limit on the host: 0". Well, it is a little bit misleading because it should be cluster-wide rule but it works as expected.
TEST 2: MaxVcpusPerClusterPct = 100
Let's set MaxVcpusPerClusterPct to 100% so we are saying to allow 1 : 1 vCPU / pCPU ratio. I have 4 node DRS cluster where each ESXi host has two cores (pCPUs), therefore I have 8 pCPUs available in the cluster. And I can really start only four VMs because each has 2 vCPUs so I can run up to 8 vCPUs in DRS cluster.
It is great, but it is worth to mention that VMs were started on single ESXi host even I have 4 ESXi hosts in DRS cluster. So vCPU / pCPU ratio is compliant per cluster but not per ESXi host as I have 8 vCPUs on single ESXi hosts having just 2 pCPUs. But that's expected behavior. So far so good.
TEST 3: MaxVcpusPerCore = 0
MaxVcpusPerCore should solve the problem observed in previous two test because it should set vCPU / pCPU ratio per host which is much better from the predictability point of view.
Let's set MaxVcpusPerCore to 0. My expectation was that I will not be able to start any VM but that was NOT the case. I was able to start a lot of VMs and exceed the expected vCPU / pCPU ratio. This is unexpected behavior.
TEST 4: MaxVcpusPerCore = 1
Let's set MaxVcpusPerCore to 1. My expectation was that I will not be able to start more than 2 vCPUs per ESXi host so only one VM with 2 vCPUs. Unfortunately, I was able to start much more vCPUs per ESXi host. This is again unexpected behavior.
TEST 5: MaxVcpusPerCore = 4
I have been informed by DRS Engineer that the minimum value allowed by host over-commitment ratio option (MaxVCPUsPerCore) is 4:1.
So, let's set MaxVcpusPerCore to 4. I have prepared nine VMs with 4 vCPUs each and my expectation is that I will not be able to start more than 8 vCPUs per ESXi host so only two VMs with 4 vCPUs per ESXi host. And because I have 4 ESXi hosts per cluster I should be able to start the maximum of 8 VMs.
UPDATE 2018-04-20: The issue with MaxVcpusPerCore is fixed in vSphere 6.5 U2. This is written in Release Notes: The advanced vSphere DRS parameter MaxVcpusPerCore might not work as expected and the desired ratio of virtual CPUs per physical CPU or core will not take effect in configurations below 4:1. MaxVcpusPerCore supported ratios now start from 1:1.
Please note, that MaxVcpusPerCore does not support ratio 0:1 in contrast with MaxVcpusPerClusterPct where 0:1 is possible and it will effectively disable to run any VM on the cluster.
Conclusion
Advanced DRS setting MaxVcpusPerClusterPct supports values between 0 and 500 and represents percentage between vCPUs and pCPUs across the whole DRS cluster. If the value is higher then 500, enforcing does not work. So, vCPU / pCPU percentage ratio can be enforced between 0:1 to 5:1. vCPU:pCPU ratio 0:1 is a special setting where no VM can be PowerOn on vSphere Cluster. This is little bit risky setting but it can be used to put the whole cluster in kind of "maintenance mode" and forbid anyone to run VMs there.
Advanced DRS setting MaxVcpusPerCore currently supports values between 0 and 32 but it works only with values between 4 and 32. This setting enforces vCPU / pCPU per each ESXi host within DRS cluster. This means that the minimum vCPU / pCPU ration configurable by this option is 4 : 1.
To be honest, I think for vSphere architects/designers MaxVcpusPerCore makes more sense than MaxVcpusPerClusterPct because vCPU/pCPU overbooking ratio defines CPU quality per ESXi host.
VMware is internally considering to align MaxVcpusPerClusterPct and MaxVcpusPerCore behavior and allow lower vCPU / pCPU ratios when MaxVcpusPerCore is used. I will track how this topic will evolve in the future. Stay tuned.
vSphere 6.5 DRS has introduced additional option to set maximum CPU over-commitment. It limits the number of vCPUs per pCPU in particular DRS cluster. However, it is good to know that there are two different advanced DRS options (configuration parameters) how to specify vCPU:pCPU ratio and each setting behaves differently. See table below ...
| DRS Advanced Option | Scope | Min-Max Value |
|---|---|---|
| MaxVcpusPerClusterPct | cluster | 0% - 500% |
| MaxVCPUsPerCore | host | 0 - 32 |
It is worth to mention a little bit tricky setting of these additional options via GUI. It is good to know how GUI setting of "CPU Over-Commitment" is mapped to DRS cluster advanced options.
In my lab, I have VCSA 6.5 U1c (build 7119157) so I did some tests.
If I set "CPU Over-Commitment" in vSphere Web Client (Flash/Flex) it sets MaxVcpusPerClusterPct so it is the setting per the whole vSphere Cluster.
However, in vSphere Client (HTML5) it sets MaxVCPUsPerCore so it is per ESXi host.
Therefore, it is good to know what you would like to achieve and double check DRS Advanced Options.
See different behavior in screenshots below
 |
| 6.5 U1c (build 7119157) vSphere Web Client (Flash/Flex) sets MaxVcpusPerClusterPct |
 | |
| 6.5 U1c (build 7119157) vSphere Client (HTML5) sets MaxVCPUsPerCore |
TEST 1: MaxVcpusPerClusterPct = 0
Let's set MaxVcpusPerClusterPct to 0 so we are saying to allow 0 : 1 vCPU / pCPU ratio. In other words, no VM can be running in the cluster. And it works as expected. When I try to run VM I get the error "The total number of virtual CPUs present or requested in virtual machines' configuration has exceeded the limit on the host: 0". Well, it is a little bit misleading because it should be cluster-wide rule but it works as expected.
 |
| Error when MaxVcpusPerClusterPct is set to 0 |
Let's set MaxVcpusPerClusterPct to 100% so we are saying to allow 1 : 1 vCPU / pCPU ratio. I have 4 node DRS cluster where each ESXi host has two cores (pCPUs), therefore I have 8 pCPUs available in the cluster. And I can really start only four VMs because each has 2 vCPUs so I can run up to 8 vCPUs in DRS cluster.
 |
| Error when MaxVcpusPerClusterPct is set to 100 |
TEST 3: MaxVcpusPerCore = 0
MaxVcpusPerCore should solve the problem observed in previous two test because it should set vCPU / pCPU ratio per host which is much better from the predictability point of view.
Let's set MaxVcpusPerCore to 0. My expectation was that I will not be able to start any VM but that was NOT the case. I was able to start a lot of VMs and exceed the expected vCPU / pCPU ratio. This is unexpected behavior.
TEST 4: MaxVcpusPerCore = 1
Let's set MaxVcpusPerCore to 1. My expectation was that I will not be able to start more than 2 vCPUs per ESXi host so only one VM with 2 vCPUs. Unfortunately, I was able to start much more vCPUs per ESXi host. This is again unexpected behavior.
TEST 5: MaxVcpusPerCore = 4
I have been informed by DRS Engineer that the minimum value allowed by host over-commitment ratio option (MaxVCPUsPerCore) is 4:1.
So, let's set MaxVcpusPerCore to 4. I have prepared nine VMs with 4 vCPUs each and my expectation is that I will not be able to start more than 8 vCPUs per ESXi host so only two VMs with 4 vCPUs per ESXi host. And because I have 4 ESXi hosts per cluster I should be able to start the maximum of 8 VMs.
 |
| Expected error message when MaxVcpusPerCore is set to 4 and vCPU:pCPU is over 4:1 per ESXi host. |
Please note, that MaxVcpusPerCore does not support ratio 0:1 in contrast with MaxVcpusPerClusterPct where 0:1 is possible and it will effectively disable to run any VM on the cluster.
Conclusion
Advanced DRS setting MaxVcpusPerClusterPct supports values between 0 and 500 and represents percentage between vCPUs and pCPUs across the whole DRS cluster. If the value is higher then 500, enforcing does not work. So, vCPU / pCPU percentage ratio can be enforced between 0:1 to 5:1. vCPU:pCPU ratio 0:1 is a special setting where no VM can be PowerOn on vSphere Cluster. This is little bit risky setting but it can be used to put the whole cluster in kind of "maintenance mode" and forbid anyone to run VMs there.
To be honest, I think for vSphere architects/designers MaxVcpusPerCore makes more sense than MaxVcpusPerClusterPct because vCPU/pCPU overbooking ratio defines CPU quality per ESXi host.
VMware is internally considering to align MaxVcpusPerClusterPct and MaxVcpusPerCore behavior and allow lower vCPU / pCPU ratios when MaxVcpusPerCore is used. I will track how this topic will evolve in the future. Stay tuned.
2 comments:
Hi, thanks for article. Could you clarify? What does pCPU mean when we talking about Hyper Threading enabled? Is it whole core (2 thread) or is it one logical CPU (1 thread)? Best regards, Andrei.
Hi,
vCPU - virtual CPU configured for Virtual Machine
pCPU - physical CPU core (hyper-threading is not considered)
Blog post updated with this terminology to avoid misunderstanding.
Post a Comment