Heads Up! VMware virtual disk IOPS limit bad behavior in VMware ESX 5.5

I've been informed about strange behavior of  VM virtual disk IOPS limits by one my customer for whom I did vSphere design recently. If you don't know how VM vDisk IOPS limits can be useful in some scenarios read my another blog post - "Why use VMware VM virtual disk IOPS limit?". And because I designed this technology for some of my customers they are very impacted by bad vDisk IOPS limit behavior in ESX 5.5

I've tested VM IOPS limits in my lab to see it by myself. Fortunately I have two labs. Older vSphere 5.0 lab with Fibre Channel Compellent storage and newer vSphere 5.5 lab with iSCSI storage EqualLogic. First of all let's look how it works in ESX 5.0. Same behavior is in ESX 5.1 and this behavior make perfect sense.

By default VM vDisks doesn't have limits as seen on next screen shot.

When I run IOmeter with single worker (thread) on unlimited vDisk I can achieve 4,846 IOPS. That's what datastore (physical storage) is able to give to single thread.

When I run IOmeter with two workers (threads) on unlimited vDisk I can achieve 7,107 IOPS. That's ok because all shared storages have implemented algorithms to limit performance for threads. That's actually protection against single thread abuse of all storage performance.

Now let's try to setup SIOC to 200 IOPS limits on both vDisks on VM as depicted on  picture below.  

Due to settings above IOmeter single worker generated workload is limited to 400 IOPS (2 x 200) per whole VM because all limit values are consolidated per virtual machine per LUN. For more info look at http://kb.vmware.com/kb/1038241. So it behaves as expected because IOmeter IOPS was oscillating between 330 and 400 IOPSes as you can see in picture below.

We can observe similar behavior with two workloads.

So in ESX 5.0 lab everything works as expected. Now let's move to another lab where I have vSphere 5.5. There is iSCSI storage so first of all we will run IOmeter without vDisk IOPS limits to see maximal performance we can get. On picture below we can see that single thread is able to get 1741 IOPSes.

... and two workers can get 3329 IOPSes.

So let's setup vDisk IOPS limits to 200 IOPS limits on both vDisks on VM as in test on ESX 5.0. I have also 2 disks on this VM. Due to these settings IOmeter single worker generated workload should be also limited to 400 IOPS (2 x 200) per whole VM. But unfortunately it is not limited and it can get 2000 IOPSes. It is strange and in my opinion bad behavior.

 

But even worse behavior can be observe when there are more threads. In examples below you can see two and four workers (threads) behavior. VM is getting really slow performance.

ESX 5.5 VM vDisk behavior is really strange and because all typical OS storage workloads (even OS booting) are multi-threaded than VM vDisk IOPS limits technology is unusable. My customer has opened support request so I believe it is a bug and VMware Support will help to escalate it in to VMware engineering.

UPDATE 2014-07-14 (Workaround): 
I've tweet about this issue to @DuncanYB @CormacJHogan @MostafaVMW and Duncan moved me immediately in to the right direction.  He reveal me the secret ... ESXi has two disk schedulers old one and new one (aka mClock).  ESXi 5.5 uses new one (mClock) by default. If you switch back to the old one, disk scheduler behaves as expected.  Below is the setting how to switch to the old one.

Go to ESX Host Advanced Settings and set Disk.SchedulerWithReservation=0

This will switch back to the old disk scheduler.

Kudos to Duncan.

Switching back to old scheduler is good workaround which will probably appear in VMware KB but there is definitely some reason why VMware introduced new disk scheduler in 5.5. I hope we will get more information  from VMware engineering so stay tuned for more details ...

UPDATE 2015-12-3:
Here are some references to more information about disk schedulers in ESXi 5.5 and above ...

ESXi 5.5
http://www.yellow-bricks.com/2014/07/14/new-disk-io-scheduler-used-vsphere-5-5/
http://cormachogan.com/2014/09/16/new-mclock-io-scheduler-in-vsphere-5-5-some-details/
http://anthonyspiteri.net/esxi-5-5-iops-limit-mclock-scheduler/

ESXi 6
http://www.cloudfix.nl/2015/02/02/vsphere-6-mclock-scheduler-reservations/

Local and shared storage performance comparison

I have just answered typical question received by email. Here is the email I have got ...

Working with a customer to validate and tune their environment. We're running IOMETER and pointing at an R720 with local storage as well as an MD3200i. Local storage is 6 15k disks in RAID10. MD has 2 disk groups w/ 6 drives in each group w/ 6 15k drives. ISCSI running through a pair of Dell PC Switches that look to be properly configured. Tried MRU and RR PSP. The local disks are absolutely blowing away the MD3200i, IOPS, MB/s and Latency in a variety of specifications.

I haven't had the chance to play w/ such a well provisioned local array lately, but am surprised by the numbers, like w/ a 512k 50%/50% spec we're seeing 22,000 iops local and 5000 iops on the MD....

Maybe I will write information you know but I believe it can be useful to get the full context.

6x15k physical disks can give you physically around 6x180 IOPS = 1080.

But ...

1/ each IOPS is different – IO depends on block size and other access specifications like sequence/random, outstanding I/O (asynch I/O not waiting for queue ack), etc.

2/ each architecture is different:

• local virtual disk (LUN) is connected via PERC having cache
• SAN virtual disk (LUN) is connected over SAN which brings another complexity & latency (NIC/HBA queues, switches, storage controller queues or LUN queues, …)   

3/ Each storage controller is different

• Local RAID controller is designed for single server workload => single thread can get full performance of disk performance and if more threads are used then performance drop down
• Shared RAID controller is designed for multiple workloads (servers/threads) => each thread can get only portion of full storage performance but each other thread will get same performance. This is fair policy/algorithm for shared environment.

The cache and particular controller IO optimization can give you significantly better IOPSes so that’s why you get 5,000 from MD and 22,000 from local disk/PERC. But 22,000 is too high number to believe it works directly with disks so there is definitely cache magic.

Here are widely used average IOPSes for different type of disks:

• 15k disk = 180 IOPS
• 10k disk = 150 IOPS
• 7k disk = 80 IOPS
• SSD/MLC = 2500 IOPS
• SSD/SLC = 5000 IOPS

Please note that

• these are average numbers used for sizing. I have seen SATA/7k disk in Compellent handling over 200 IOPses but it was sequential access and disks were quite overloaded because latency was very high!!!
• SSD numbers significantly differs among different manufacturers

All these calculations can give you available IOPSes for read or write to non-redundant virtual disk (LUN/volume). This means single disk or RAID 0. If you use redundant RAID you have to calculate RAID write penalty

• RAID 10 = 2
• RAID 5 =4
• RAID 6 = 6

So you can see this is a quite complex topic and if you really want to show the customer the truth (who knows what is pure true? :-) ) then you have to consider all statements above.

Typical issues of IOmeter measuring without enough experience:

• Small target disk file (entered in blocks = 512B). The disk target file must  be bigger than cache. I usually use the file between 20000000 (approx. 20GB) and 80000000 blocks (approx. 40GB).
• Small number of threads (in IOmeter terminology workers)
• Workload generated from single server. Do you know you can run dynamo on another computer and connect it to IOmeter over network? Then you will see more managers (servers) and you can define workers and access specifications from single GUI.

Hope this helps at least to someone and I would appreciate deeper discussion on this topic.