No data visibility for vSphere Admin

Recently I did very quick (time constrained) conceptual/logical design exercise for one customer who had virtualization first strategy and was willing to virtualize his Tier 1 business critical applications. One his requirement was to preclude data visibility for VMware vSphere admins.

I was quickly thinking how to fulfill this particular requirement and my first general answer was ENCRYPTION. The customer asked me to tell him more about encryption possibilities and I listed him following options.

Option 1/ Encryption in the Guest OS 

Product examples Microsoft BitLocker, HyTrust, SafeNet, etc.
Very nice comparison of disk encryption softwares is here.

Option 2/ Application level encryption

Product examples Database Encryption in SQL Server 2008 and higher, Oracle Database Transparent Encryption, etc.

Option 3/ Encryption in the Fibre Channel SAN

Example is Brocade SAN Encryption Solution or Cisco MDS 9000 Family Storage Media Encryption.

Option 4/ Encryption in the Disk Array

Data encryption behind storage controllers. Usually leveraging Self Encrypted Disks (aka SED).

Next logical question was ... what is the performance impact?.
My quick answer was that there is definitely performance overhead in software encryption but no performance overhead with hardware encryption as it is offloaded into the special ASICs.

Hmm... Right, the most appropriate answer would be that hardware solutions are designed to have none or negligible performance impact. I always recommend to do testing before any real use in production but that's what hardware vendors claim at least in their white papers. Specifically in option (3) storage IO has to be redirected to the encryption module/appliance in the SAN which should be order of magnitude less that typical IO response time therefore impact on storage latency should be theoretically none or negligible.

However the problem with my recommended options is not  performance claim.
The problem is that only option 1 and 2 are applicable to fulfill customer's requirement because option 3 and 4 do encryption and decryption on lower levels and data are decrypted and visible on vSphere layer. Therefore vSphere admin would have visibility into data.

Options 1 and 2 has definitely some performance overhead nowadays generally somewhere between 20%-100% depending on software solution, CPU family, encryption algorithm strength, encryption key length, etc.

For completeness let's say that options 3 and 4 are good for different use cases.

• Option 3 can help you to secure data from storage admin not having access to SAN network or from someone having physical access to disks. 
• Option 4 can help you to secure data on disks against theft of physical storage or disks.

It is worth to say that security is always trade-off.

Software based solutions has some negative impact on performance, medium negative impact on price and also negative impact on manageability. Performance of software based solutions can be significantly improved by leveraging AES hardware offload to modern Intel Processors and performance overhead will be mitigated year by year.

Pure hardware based solutions are not applicable options for our specific requirement but even it would be applicable and they will have none or negligible impact on performance there are drawbacks like huge impact on cost and also some impact on scalability and manageability.

Conclusion
I was very quick during my consulting and I didn't realize what options really fulfill specific customer's requirement. I'm often saying that I don't trust anybody nor my self. This was exactly the case - unfortunately :-(

Time constrained consulting usually doesn't offer the best results. Good architecture need some time for review and better options comparison :-)

ESXi root password complexity

Warning: This is just for lab experimenting and not for production use. 

When experimenting with ESXi in the lab sometimes you have to reset ESXi to default settings. After "Reset System Configuration"from DCUI your password is removed and you have to set the new one.  I prefer to have simple root password in the lab. However ESXi requires pretty strength password complexity and below is procedure how to decrease it.

1/ Login to ESXi shell console.

2/ Edit /etc/pam.d/passwd  (vi /etc/pam.d/passwd)
By default password complexity is set like that
password     requisite    /lib/security/$ISA/pam_passwdqc.so retry=3 min=disabled, disabled,disabled,7,7

3/ Change password requisite to
password     requisite    /lib/security/$ISA/pam_passwdqc.so retry=3 min=disabled, disabled,disabled,1,1

4/ Change root password by command passwd

For  more information look at vSphere documentation.

Using SSL certificates for VMware vSphere Components

Streaming the certificate replacement and management process in a VMware environment can be challenging at times. For instance, changing certificates for a vCenter 5.1 is a hugely laborious process. And in a typical environment where there are a large number of hosts running, tracking and managing their certificates is difficult and time consuming. More importantly, security breaches due to lapsed certificates can prove to be very expensive to the organization. vCert Manager from VSS Labs provides fully automated management of SSL Certificates in a VMware environment across the entire lifecycle.

VSS Labs has solution to simplify SSL management. For more info look at http://vsslabs.com/vCert.html

To be honest I had no chance to test it because I avoid signed SSL certificates if possible. However when I'll have a customer who requires SSL I definitely have to evaluate VSS Labs solution.

How to defend against ARP poisoning/spoofing attack in vSphere infrastructure

There are few vSphere Infrastructure enterprise possibilities how to deal with this type of attack.
I know about two ... Vmware vShield  and CISCO Nexus1000v.

However here I would like to share idea how to do it with open source tools integrated into enterprise infrastructure.

Disclaimer: 
Please be aware that this is not out of box enterprise solution and you have to know what you are doing and you have full responsibility for all impacts.

How we can simulate the attack?
Bellow is tutorial inspired by another tutorial from
http://blog.facilelogin.com/2011/01/arp-poisoning-with-dsniff.html
You can simply change installation procedures based on your OS distribution.

ARP poisoning with dsniff
dsniff is a collection of tools for network auditing and penetration testing. dsniff, filesnarf, mailsnarf, msgsnarf, urlsnarf, and webspy passively monitor a network for interesting data (passwords, e-mail, files, etc.). arpspoof, dnsspoof, and macof facilitate the interception of network traffic normally unavailable to an attacker (e.g, due to layer-2 switching). sshmitm and webmitm implement active monkey-in-the-middle attacks against redirected SSH and HTTPS sessions by exploiting weak bindings in ad-hoc PKI.

To install dsniff on CentOS 6.

yum -y install wget

cd /usr/src
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-7.noarch.rpm
rpm -ivh epel-release-6-7.noarch.rpm
yum –y install openssl gcc flex bison libpcap-devel libnet

yum install dsniff

Now you need to find out two things,

1. IP address of the target machine - say 192.168.1.4
2. IP address of the Gateway - say 192.168.1.1

Let's start ARP poisoning from the attacker's machine - with arpspoof tool which comes with dsniff.

$ sudo arpspoof -i en1 -t 192.168.1.4 192.168.1.1

This will update target machine's ARP table with attacker's MAC address against the IP address of the gateway.

Now - start a tcpdump on the same interface from your machine - start viewing all the traffic going to and from the target machine.

$ sudo tcpdump -i en1

How we can detect the attack?
We can use aprwatch for example on my favorite OS FreeBSD running in virtual machine,

Installation is simple as
cd /usr/ports/net-mgmt/arpwatch/
make install
...
then you have to add
arpwatch_enable="YES"
in to your /etc/rc.conf
...
 
And last but not least is to enable promiscuous mode on VMware vSwitch portgroup where arpwatch vm is connected to. The best way is to create another portgroup (single port is enough) with the same VLAN ID as protected VLAN and in Security we have to set Promiscuous Mode to Accept. 

... then arpwatch send message to syslog. So you can see something similar in your /var/log/messages
Oct 31 09:08:02 ips arpwatch: flip flop a.b.c.d 0:50:56:8d:2e:bc (54:52:0:fe:47:95)

Arpwatch can also send a e-mail message about incident. The message looks like

hostname:
ip address: 95.80.240.1
ethernet address: 54:52:0:fe:47:95
ethernet vendor:
old ethernet address: 0:50:56:8d:2e:bc
old ethernet vendor: VMWare, Inc.
timestamp: Wednesday, October 31, 2012 8:57:33 +0100
previous timestamp: Wednesday, October 31, 2012 8:57:33 +0100
delta: 0 seconds

How we can protect against the attack?

Well this is another story. It really depends on your environment but in my environment I have vSphere Distributed Switch and all virtual machines have VMtools installed so I trust VMware MAC:IP associations and based on these information (MAC) I can find the attacker port in distributed switch and disable it.

And that's just a small step to do it in automated way by leveraging VMware vCLI (aka VMware Perl SDK).

WARNING!!!

Currently described solution works only on single ESX host and doesn't work among more ESX hosts because of virtual networking principles. VMware virtual switch (or module of distributed switch) is not a switch but port extender. The difference is significant. The ARP poisoning attack is based on permanently unicasting ARP replies to the victim so when Arpwatch server is not on the same ESX as the attacker or the victim then even promiscuous port on distributed switch portgroup will not help us to catch it because arp reply packet is not visible on Arpwatch server.

The solution would be to have arpwatch VM on each ESX host. VMware introduced the concept of ESX agents (aka Agent Virtual Machines) so I believe this is the right use case for ESX agent implementation.
For more information about "Deploying vSphere Solutions, vServices, and ESX Agents" read this document.

I have to test this architecture ... so stay tuned.

Any comments or thoughts are appreciated.

vSphere 5.0 Hardening Guide - Public Draft

http://communities.vmware.com/docs/DOC-19056