HP Flex-10 Design, Plan, Implement, Test

Before design phase of VMware vSphere Infrastructure I recommend to read blog post "Understanding HP Flex-10 Mappings with VMware ESX/vSphere" to get general overview about server infrastructure and advanced  network interconnect. During design phase prepare detail test plan (aka operational verification) and test it during implementation phase. You can use blog post "Testing Scenario's VMware / HP c-Class Infrastructure" as a template for your test plan. I don't doubt that you normally test infrastructure before put it into production :-)

MSCS RDMs causing long boot of ESX

That's because RDM LUN attached to MSCS cluster has permanent SCSI reservation initiated by active node of cluster.

In ESX 5 you have to mark all such LUNs as perennially reserved and your ESX boot can be fast as usual.

Here is CLI command to mark LUN
esxcli storage core device setconfig -d naa.id --perennially-reserved=true

This has to be changed on all ESX hosts with visibility to the LUN.

More info at http://kb.vmware.com/kb/1016106

How to calculate storage performance from host perspective

Storage performance is usually quantified as IOPS (I/O transactions per second). The performance from storage perspective is quite easy. It really depends on speed of each particular disk - also known as spindle. Each disk has some speed and bellow are written average values which are usually used for storage performance calculation

• SATA disk = 80 IOPS
• SCSI DISK(SAS or FC) 10k RPM = 150 IOPS
• SCSI DISK(SAS or FC) 15k RPM = 180 IOPS
• SSD disk (SLC aka EFD) = 6000 IOPS

So when we need higher performance we have to bundle disks. Disks can be bundled with standard RAID technology.

Here are most common RAID types used on standard disk arrays:

• RAID 0 - no redundancy, disk bundle, higest performance => WRITE PENALTY = 0
• RAID 1 - disk mirror, max bundle of 2 disks, high performance => WRITE PENALTY = 2
• RAID 10 - RAID 1 + RAID 0 for bundling disk pairs, max disk bundle depends on disk array limits, high performance => WRITE PENALTY = 2
• RAID 5 - block level striping with rotated parity, max disk bundle depends on disk array limits, moderate performance => WRITE PENALTY = 4
• RAID 6 - block level striping with double parity, max disk bundle depends on disk array limits, lower performance => WRITE PENALTY = 6

So performance from storage perspective and from host perspective are different. Performance from storage perspective is simply summation of speed of all disks in RAID group. Performance from host perspective depends on selected RAID type.

To calculate estimated storage performance from host perspective we need to use the formula of several variables.

First of all let's define variables

P=write penalty of selected RAID type
R=Read % of disk workload
W=Write % of disk workload

H=IOPS from host perspective

S=IOPS from storage perspective

and now we can write formula to calculate storage performance from host perspective

H = S / (R+W*P)

Do you want to know all steps how to get this formula? It is simple. Start from another formula which describes storage behavior.
R*(1*H) + W*(P*H) = S
Above formula says - each host read IOPS generates single storage IOPS but each write IOPS generates multiple IOPS based on RAID type penalty (P).

Does it make sense? If not example can help you to understand.

My RAID group has 9 SAS disks 600GB/15k RPM and I use RAID 5 (8+1).
So from storage perspective I have 9 disks where each can perform 180 IOPS which means I have performance 1620 IOPS from storage perspective. Let's assume I have strange read/write ratio 20:80.

S = 1620
P = 4 (because of RAID 5)
R = 20% = 0.2
W= 80% = 0.8
I need to know H ... storage performance from host perspective.

H = 1620 / (0.2 + 0.8 * 4) = 1620 / 3.4 = 476.47 IOPS from host perspective.

Note: Modern disk arrays often offer AST (Automated Storage Tiering). The calculation described in this blog post is valid even for those disk arrays. You have to fully understand internal architecture and design of particular storage but generally all storage pools are build from some sub disk groups bundled and protected by some RAID type. So if you have 125 disks bundled by 5 disks in RAID 5 (4+1) then the principle is the same. We have 125 spindles and write penalty is 4 because of RAID 5.

Cisco Custom Image for ESXi 5

Cisco Custom Image for ESXi 5.1.0 GA Install CD

https://t.co/EGNxWJ5p

https://my.vmware.com/web/vmware/details?downloadGroup=CISCO-ESXI-5.1.0-GA-25SEP2012&productId=285#product_downloads

Set the Scratch Partition from the vSphere Client

If a scratch partition is not set up, you might want to configure one, especially if low memory is a concern. When a scratch partition is not present, vm-support output is stored in a ramdisk.

Prerequisites

The directory to use for the scratch partition must exist on the host.

Procedure

1	Use the vSphere Client to connect to the host.
2	Select the host in the Inventory.
3	In the Configuration tab, select Software.
4	Select Advanced Settings.
5	Select ScratchConfig. The field ScratchConfig.CurrentScratchLocation shows the current location of the scratch partition.
6	In the field ScratchConfig.ConfiguredScratchLocation, enter a directory path that is unique for this host. Example of directory path is /vmfs/volumes/NFS-SYNOLOGY-SSD/scratch/esx21.home.uw.cz In the example above, I have datastore with name NFS-SYNOLOGY-SSD where I have subdirectory scratch having another subdirectory esx21.home.uw.cz
7	Reboot the host for the changes to take effect. (copy from vSphere documentation)

For automated scratch partition configuration you can use vCLI, PowerCLI. For details see. VMware KB 1033696.

And here is my PowerCLI script inspired by KB above to set scratch location on all ESXi hosts in particular vSphere clusters.

ESXi strange related log entry in /var/log/vmkernel.log

I've just found in /var/log/vmkernel.log lot of following storage errors


2012-12-19T01:34:02.010Z cpu2:4098)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x93 (0x412401965f00, 5586) to dev "naa.60060e80102d5f500511c97d000000d4" on path "vmhba2:C0:T0:L2" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x96 0x32. Act:NONE
2012-12-19T01:34:02.010Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412401965f00) 0x93, CmdSN 0xc6fd5 from world 5586 to dev "naa.60060e80102d5f500511c97d000000d4" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x96 0x32.

The main part of log entry is "failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x96 0x32"

If I understand correctly
D: 0x2 = DEVICE CHECK CONDITION
Sense code 0x5 = ILLEGAL REQUEST

What is it? What doe's it mean?

I have ESXi 5.0 build 768111, storage HDS AMS 2300, CISCO UCS blade system, CISCO FC switches.

Update 1:
I've thought more about the root cause ... important detail is that it is happen when storage vMotion or other data migration is happening. So I've a hypotheses that it is related to VAAI. Storage is VAAI enabled and VAAI is supported. However disk block size is different on datastores (we are just in the middle of migration from VMFS-3 to VMFS-5).

So I've to do deeper diagnostic and root cause troubleshooting.

Stay tuned.


Update 2:
Solved, VAAI primitives must be enabled also on HDS Host Masking. For more information check
http://www.hds.com/assets/pdf/optimizing-the-hitachi-ams-2000-family-in-vsphere-4-environments.pdf

Storage Queues and Performance

VMware recently published a paper titled Scalable Storage Performance that delivered a wealth of information on storage with respect to the  ESX Server architecture.  This paper contains details about the storage  queues that are a mystery to many of VMware's customers and partners.   I  wanted to start a wiki article on some aspects of this paper that may  be interesting to storage enthusiasts and performance freaks.

Blog post for more information is at http://communities.vmware.com/docs/DOC-6490

These information are very useful for deep understanding of full storage stack.

Best Practices for Faster vSphere SDK Scripts

Source at http://www.virtuin.com/2012/11/best-practices-for-faster-vsphere-sdk.html 

The VMware vSphere API is one of the more powerful vendor SDKs available in the Virtualization Ecosystem.  As adoption of VMware vSphere has grown over the years, so has the size of Virtual Infrastructure environments.  In many larger enterprises, the increasing number of VirtualMachines and HostSystems is driving the architectural requirement to deploy multiple vCenter Servers.
In response, the necessity for automation tooling has grown just as quickly.  Automation to create daily reports, perform bulk operations, and aggregate data from large, distributed Virtual Infrastructure environments is a common requirement for managing the increasing virtual sprawl.
In a Virtual Infrastructure comprised of thousands of objects, even a simple script to list all VirtualMachines and their associated HostSystem and Datastores can result in very slow runtime execution.  Developing automation with the following, simple best practices can take orders of magnitude off your vSphere API tool's runtime.

 READ FULL ARTICLE

DELL Active System Manager

DELL Active System is managed by DELL Active System Manager. This is DELL converged infrastructure solution (blade server, networking, storage) to achieve "mainframe of 21st century" with leveraging server virtualization (hypervisors) to have enough flexibility to achieve required infrastructure SLAs.

http://www.youtube.com/watch?v=xU1I93wEHuU

Configuring a Chassis in Dell Active System Manager
http://www.youtube.com/watch?v=cRO0546yJ8U

IBM PureFlex

IBM Pure Flex System is probably another next generation computing system leveraging converged infrastructure concept. IBM Flex System Manager manages Pure Flex System. Who can honestly and precisely compare it with HP Virtual Connect, CISCO UCS, and DELL Active System?

Introduction video is available at
http://www.youtube.com/watch?v=GDGpzkQm8kU

VAAI - VMware API for Array Integration deep dive

http://www.vmware.com/files/pdf/techpaper/VMware-vSphere-Storage-API-Array-Integration.pdf

Correlating vCenter Server and ESXi/ESX host build numbers to update levels

VMware software versions can be found on VMware KB Article 1014508.

Very nice list of VMware ESX server build numbers and versions mappings together with mapping to VMware tools (aka vmtools) versions is at https://packages.vmware.com/tools/versions

Brocade Secure SAN Zoning Best Practices

White Paper
http://www.brocade.com/downloads/documents/white_papers/Zoning_Best_Practices_WP-00.pdf

This paper describes and clarifies Zoning, a security feature in Storage
Area Network (SAN) fabrics. By understanding the terminology and
implementing Zoning best practices, a Brocade®
 SAN fabric can be
easily secured and scaled while maintaining maximum uptime.
The following topics are discussed:
• Zoning defined and LUN security in the fabric
• Identifying hosts and storage members of a zone
• How do SAN switches enforce Zoning?
• Avoiding Zoning terminology confusion
• Approaches to Zoning, how to group hosts and storage in zones
• Brocade Zoning recommendations and summary


What is Zoning?
Zoning is a fabric-based service in Storage Area Networks that groups host and storage nodes
that need to communicate. Zoning creates a situation in which nodes can communicate with
each other only if they are members of the same zone. Nodes can be members of multiple
zones--—allowing for a great deal of flexibility when you implement a SAN using Zoning.
Zoning not only prevents a host from unauthorized access of storage assets, but it also stops
undesired host-to-host communication and fabric-wide Registered State Change Notification
(RSCN) disruptions. RSCNs are managed by the fabric Name Server and notify end devices of
events in the fabric, such as a storage node or a switch going offline. Brocade isolates these
notifications to only the zones that require the update, so nodes that are unaffected by the
fabric change do not receive the RSCN. This is important for non-disruptive fabric operations,
because RSCNs have the potential to disrupt storage traffic. When this disruption was more
common, that is, with older Host Bus Adapter (HBA) drivers, RSCNs gained an undeserved
negative reputation. However, since that time most HBA vendors have addressed the issues.
When nodes are zoned into small, granular groupings, the occurrences of disruptive RSCNs
are virtually eliminated. See a discussion of single HBA zoning in the section of this paper
entitled, “Approaches to Zoning.”

ESX and disk issues

ESX 4 & 5: Resolving SCSI reservation conflicts
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002293
In this KB article is described the process how to find which ESX host has SCSI reservation on LUN

ESX 5: Vmware vSphere 5 dead LUN and pathing issues and resultant SCSI errors
http://raj2796.wordpress.com/2012/03/14/vmware-vsphere-5-dead-lun-and-pathing-issues-and-resultant-scsi-errors/

All ESX versions: After repeated SAN path failovers, operations that involve VMFS changes might fail for all hosts accessing a particular LUN
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1009899

ESX 4.x: ESX/ESXi hosts in APD may appear Not Responding in vCenter Server
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1030980

ESX 4.1: Virtual machines stop responding when any LUN on the host is in an all-paths-down (APD) condition
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1016626

ESX 5.1 has significant improvements with APD a PDL
http://www.vmware.com/files/pdf/techpaper/Whats-New-VMware-vSphere-51-Storage-Technical-Whitepaper.pdf

vCenter Server 5.1 Required Ports

http://www.vladan.fr/vcenter-server-5-1-required-ports/

ESX Automated Provisioning on CISCO UCS

This is the demo of automation showing how VMware vSphere ESX host can be
 automatically deploy to CISCO UCS Service Profile which is booted from SAN.


If you want to know more don't hesitate to write comment bellow the blog post.

Home wind power station

http://www.taawin.cz/

VAAI Comparison – Block versus NAS

http://cormachogan.com/2012/11/08/vaai-comparison-block-versus-nas/

VMware Support Alert – Implementing CA signed SSL certificates with vSphere 5.1

http://vmwaretips.com/wp/2012/10/31/vmware-support-alert-implementing-ca-signed-ssl-certificates-with-vsphere-5-1/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+VMwareTips+%28VMwareTips%29

How to defend against ARP poisoning/spoofing attack in vSphere infrastructure

There are few vSphere Infrastructure enterprise possibilities how to deal with this type of attack.
I know about two ... Vmware vShield  and CISCO Nexus1000v.

However here I would like to share idea how to do it with open source tools integrated into enterprise infrastructure.

Disclaimer: 
Please be aware that this is not out of box enterprise solution and you have to know what you are doing and you have full responsibility for all impacts.

How we can simulate the attack?
Bellow is tutorial inspired by another tutorial from
http://blog.facilelogin.com/2011/01/arp-poisoning-with-dsniff.html
You can simply change installation procedures based on your OS distribution.

ARP poisoning with dsniff
dsniff is a collection of tools for network auditing and penetration testing. dsniff, filesnarf, mailsnarf, msgsnarf, urlsnarf, and webspy passively monitor a network for interesting data (passwords, e-mail, files, etc.). arpspoof, dnsspoof, and macof facilitate the interception of network traffic normally unavailable to an attacker (e.g, due to layer-2 switching). sshmitm and webmitm implement active monkey-in-the-middle attacks against redirected SSH and HTTPS sessions by exploiting weak bindings in ad-hoc PKI.

To install dsniff on CentOS 6.

yum -y install wget

cd /usr/src
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-7.noarch.rpm
rpm -ivh epel-release-6-7.noarch.rpm
yum –y install openssl gcc flex bison libpcap-devel libnet

yum install dsniff

Now you need to find out two things,

1. IP address of the target machine - say 192.168.1.4
2. IP address of the Gateway - say 192.168.1.1

Let's start ARP poisoning from the attacker's machine - with arpspoof tool which comes with dsniff.

$ sudo arpspoof -i en1 -t 192.168.1.4 192.168.1.1

This will update target machine's ARP table with attacker's MAC address against the IP address of the gateway.

Now - start a tcpdump on the same interface from your machine - start viewing all the traffic going to and from the target machine.

$ sudo tcpdump -i en1

How we can detect the attack?
We can use aprwatch for example on my favorite OS FreeBSD running in virtual machine,

Installation is simple as
cd /usr/ports/net-mgmt/arpwatch/
make install
...
then you have to add
arpwatch_enable="YES"
in to your /etc/rc.conf
...
 
And last but not least is to enable promiscuous mode on VMware vSwitch portgroup where arpwatch vm is connected to. The best way is to create another portgroup (single port is enough) with the same VLAN ID as protected VLAN and in Security we have to set Promiscuous Mode to Accept. 

... then arpwatch send message to syslog. So you can see something similar in your /var/log/messages
Oct 31 09:08:02 ips arpwatch: flip flop a.b.c.d 0:50:56:8d:2e:bc (54:52:0:fe:47:95)

Arpwatch can also send a e-mail message about incident. The message looks like

hostname:
ip address: 95.80.240.1
ethernet address: 54:52:0:fe:47:95
ethernet vendor:
old ethernet address: 0:50:56:8d:2e:bc
old ethernet vendor: VMWare, Inc.
timestamp: Wednesday, October 31, 2012 8:57:33 +0100
previous timestamp: Wednesday, October 31, 2012 8:57:33 +0100
delta: 0 seconds

How we can protect against the attack?

Well this is another story. It really depends on your environment but in my environment I have vSphere Distributed Switch and all virtual machines have VMtools installed so I trust VMware MAC:IP associations and based on these information (MAC) I can find the attacker port in distributed switch and disable it.

And that's just a small step to do it in automated way by leveraging VMware vCLI (aka VMware Perl SDK).

WARNING!!!

Currently described solution works only on single ESX host and doesn't work among more ESX hosts because of virtual networking principles. VMware virtual switch (or module of distributed switch) is not a switch but port extender. The difference is significant. The ARP poisoning attack is based on permanently unicasting ARP replies to the victim so when Arpwatch server is not on the same ESX as the attacker or the victim then even promiscuous port on distributed switch portgroup will not help us to catch it because arp reply packet is not visible on Arpwatch server.

The solution would be to have arpwatch VM on each ESX host. VMware introduced the concept of ESX agents (aka Agent Virtual Machines) so I believe this is the right use case for ESX agent implementation.
For more information about "Deploying vSphere Solutions, vServices, and ESX Agents" read this document.

I have to test this architecture ... so stay tuned.

Any comments or thoughts are appreciated.