Set the Scratch Partition from the vSphere Client

If a scratch partition is not set up, you might want to configure one, especially if low memory is a concern. When a scratch partition is not present, vm-support output is stored in a ramdisk.

Prerequisites

The directory to use for the scratch partition must exist on the host.

Procedure

1	Use the vSphere Client to connect to the host.
2	Select the host in the Inventory.
3	In the Configuration tab, select Software.
4	Select Advanced Settings.
5	Select ScratchConfig. The field ScratchConfig.CurrentScratchLocation shows the current location of the scratch partition.
6	In the field ScratchConfig.ConfiguredScratchLocation, enter a directory path that is unique for this host. Example of directory path is /vmfs/volumes/NFS-SYNOLOGY-SSD/scratch/esx21.home.uw.cz In the example above, I have datastore with name NFS-SYNOLOGY-SSD where I have subdirectory scratch having another subdirectory esx21.home.uw.cz
7	Reboot the host for the changes to take effect. (copy from vSphere documentation)

For automated scratch partition configuration you can use vCLI, PowerCLI. For details see. VMware KB 1033696.

And here is my PowerCLI script inspired by KB above to set scratch location on all ESXi hosts in particular vSphere clusters.

ESXi strange related log entry in /var/log/vmkernel.log

I've just found in /var/log/vmkernel.log lot of following storage errors


2012-12-19T01:34:02.010Z cpu2:4098)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x93 (0x412401965f00, 5586) to dev "naa.60060e80102d5f500511c97d000000d4" on path "vmhba2:C0:T0:L2" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x96 0x32. Act:NONE
2012-12-19T01:34:02.010Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412401965f00) 0x93, CmdSN 0xc6fd5 from world 5586 to dev "naa.60060e80102d5f500511c97d000000d4" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x96 0x32.

The main part of log entry is "failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x96 0x32"

If I understand correctly
D: 0x2 = DEVICE CHECK CONDITION
Sense code 0x5 = ILLEGAL REQUEST

What is it? What doe's it mean?

I have ESXi 5.0 build 768111, storage HDS AMS 2300, CISCO UCS blade system, CISCO FC switches.

Update 1:
I've thought more about the root cause ... important detail is that it is happen when storage vMotion or other data migration is happening. So I've a hypotheses that it is related to VAAI. Storage is VAAI enabled and VAAI is supported. However disk block size is different on datastores (we are just in the middle of migration from VMFS-3 to VMFS-5).

So I've to do deeper diagnostic and root cause troubleshooting.

Stay tuned.


Update 2:
Solved, VAAI primitives must be enabled also on HDS Host Masking. For more information check
http://www.hds.com/assets/pdf/optimizing-the-hitachi-ams-2000-family-in-vsphere-4-environments.pdf

Storage Queues and Performance

VMware recently published a paper titled Scalable Storage Performance that delivered a wealth of information on storage with respect to the  ESX Server architecture.  This paper contains details about the storage  queues that are a mystery to many of VMware's customers and partners.   I  wanted to start a wiki article on some aspects of this paper that may  be interesting to storage enthusiasts and performance freaks.

Blog post for more information is at http://communities.vmware.com/docs/DOC-6490

These information are very useful for deep understanding of full storage stack.

Best Practices for Faster vSphere SDK Scripts

Source at http://www.virtuin.com/2012/11/best-practices-for-faster-vsphere-sdk.html 

The VMware vSphere API is one of the more powerful vendor SDKs available in the Virtualization Ecosystem.  As adoption of VMware vSphere has grown over the years, so has the size of Virtual Infrastructure environments.  In many larger enterprises, the increasing number of VirtualMachines and HostSystems is driving the architectural requirement to deploy multiple vCenter Servers.
In response, the necessity for automation tooling has grown just as quickly.  Automation to create daily reports, perform bulk operations, and aggregate data from large, distributed Virtual Infrastructure environments is a common requirement for managing the increasing virtual sprawl.
In a Virtual Infrastructure comprised of thousands of objects, even a simple script to list all VirtualMachines and their associated HostSystem and Datastores can result in very slow runtime execution.  Developing automation with the following, simple best practices can take orders of magnitude off your vSphere API tool's runtime.

 READ FULL ARTICLE

DELL Active System Manager

DELL Active System is managed by DELL Active System Manager. This is DELL converged infrastructure solution (blade server, networking, storage) to achieve "mainframe of 21st century" with leveraging server virtualization (hypervisors) to have enough flexibility to achieve required infrastructure SLAs.

http://www.youtube.com/watch?v=xU1I93wEHuU

Configuring a Chassis in Dell Active System Manager
http://www.youtube.com/watch?v=cRO0546yJ8U

IBM PureFlex

IBM Pure Flex System is probably another next generation computing system leveraging converged infrastructure concept. IBM Flex System Manager manages Pure Flex System. Who can honestly and precisely compare it with HP Virtual Connect, CISCO UCS, and DELL Active System?

Introduction video is available at
http://www.youtube.com/watch?v=GDGpzkQm8kU

VAAI - VMware API for Array Integration deep dive

http://www.vmware.com/files/pdf/techpaper/VMware-vSphere-Storage-API-Array-Integration.pdf

Correlating vCenter Server and ESXi/ESX host build numbers to update levels

VMware software versions can be found on VMware KB Article 1014508.

Very nice list of VMware ESX server build numbers and versions mappings together with mapping to VMware tools (aka vmtools) versions is at https://packages.vmware.com/tools/versions

Brocade Secure SAN Zoning Best Practices

White Paper
http://www.brocade.com/downloads/documents/white_papers/Zoning_Best_Practices_WP-00.pdf

This paper describes and clarifies Zoning, a security feature in Storage
Area Network (SAN) fabrics. By understanding the terminology and
implementing Zoning best practices, a Brocade®
 SAN fabric can be
easily secured and scaled while maintaining maximum uptime.
The following topics are discussed:
• Zoning defined and LUN security in the fabric
• Identifying hosts and storage members of a zone
• How do SAN switches enforce Zoning?
• Avoiding Zoning terminology confusion
• Approaches to Zoning, how to group hosts and storage in zones
• Brocade Zoning recommendations and summary


What is Zoning?
Zoning is a fabric-based service in Storage Area Networks that groups host and storage nodes
that need to communicate. Zoning creates a situation in which nodes can communicate with
each other only if they are members of the same zone. Nodes can be members of multiple
zones--—allowing for a great deal of flexibility when you implement a SAN using Zoning.
Zoning not only prevents a host from unauthorized access of storage assets, but it also stops
undesired host-to-host communication and fabric-wide Registered State Change Notification
(RSCN) disruptions. RSCNs are managed by the fabric Name Server and notify end devices of
events in the fabric, such as a storage node or a switch going offline. Brocade isolates these
notifications to only the zones that require the update, so nodes that are unaffected by the
fabric change do not receive the RSCN. This is important for non-disruptive fabric operations,
because RSCNs have the potential to disrupt storage traffic. When this disruption was more
common, that is, with older Host Bus Adapter (HBA) drivers, RSCNs gained an undeserved
negative reputation. However, since that time most HBA vendors have addressed the issues.
When nodes are zoned into small, granular groupings, the occurrences of disruptive RSCNs
are virtually eliminated. See a discussion of single HBA zoning in the section of this paper
entitled, “Approaches to Zoning.”

ESX and disk issues

ESX 4 & 5: Resolving SCSI reservation conflicts
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002293
In this KB article is described the process how to find which ESX host has SCSI reservation on LUN

ESX 5: Vmware vSphere 5 dead LUN and pathing issues and resultant SCSI errors
http://raj2796.wordpress.com/2012/03/14/vmware-vsphere-5-dead-lun-and-pathing-issues-and-resultant-scsi-errors/

All ESX versions: After repeated SAN path failovers, operations that involve VMFS changes might fail for all hosts accessing a particular LUN
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1009899

ESX 4.x: ESX/ESXi hosts in APD may appear Not Responding in vCenter Server
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1030980

ESX 4.1: Virtual machines stop responding when any LUN on the host is in an all-paths-down (APD) condition
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1016626

ESX 5.1 has significant improvements with APD a PDL
http://www.vmware.com/files/pdf/techpaper/Whats-New-VMware-vSphere-51-Storage-Technical-Whitepaper.pdf

vCenter Server 5.1 Required Ports

http://www.vladan.fr/vcenter-server-5-1-required-ports/

ESX Automated Provisioning on CISCO UCS

This is the demo of automation showing how VMware vSphere ESX host can be
 automatically deploy to CISCO UCS Service Profile which is booted from SAN.


If you want to know more don't hesitate to write comment bellow the blog post.

Home wind power station

http://www.taawin.cz/

VAAI Comparison – Block versus NAS

http://cormachogan.com/2012/11/08/vaai-comparison-block-versus-nas/

VMware Support Alert – Implementing CA signed SSL certificates with vSphere 5.1

http://vmwaretips.com/wp/2012/10/31/vmware-support-alert-implementing-ca-signed-ssl-certificates-with-vsphere-5-1/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+VMwareTips+%28VMwareTips%29

How to defend against ARP poisoning/spoofing attack in vSphere infrastructure

There are few vSphere Infrastructure enterprise possibilities how to deal with this type of attack.
I know about two ... Vmware vShield  and CISCO Nexus1000v.

However here I would like to share idea how to do it with open source tools integrated into enterprise infrastructure.

Disclaimer: 
Please be aware that this is not out of box enterprise solution and you have to know what you are doing and you have full responsibility for all impacts.

How we can simulate the attack?
Bellow is tutorial inspired by another tutorial from
http://blog.facilelogin.com/2011/01/arp-poisoning-with-dsniff.html
You can simply change installation procedures based on your OS distribution.

ARP poisoning with dsniff
dsniff is a collection of tools for network auditing and penetration testing. dsniff, filesnarf, mailsnarf, msgsnarf, urlsnarf, and webspy passively monitor a network for interesting data (passwords, e-mail, files, etc.). arpspoof, dnsspoof, and macof facilitate the interception of network traffic normally unavailable to an attacker (e.g, due to layer-2 switching). sshmitm and webmitm implement active monkey-in-the-middle attacks against redirected SSH and HTTPS sessions by exploiting weak bindings in ad-hoc PKI.

To install dsniff on CentOS 6.

yum -y install wget

cd /usr/src
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-7.noarch.rpm
rpm -ivh epel-release-6-7.noarch.rpm
yum –y install openssl gcc flex bison libpcap-devel libnet

yum install dsniff

Now you need to find out two things,

1. IP address of the target machine - say 192.168.1.4
2. IP address of the Gateway - say 192.168.1.1

Let's start ARP poisoning from the attacker's machine - with arpspoof tool which comes with dsniff.

$ sudo arpspoof -i en1 -t 192.168.1.4 192.168.1.1

This will update target machine's ARP table with attacker's MAC address against the IP address of the gateway.

Now - start a tcpdump on the same interface from your machine - start viewing all the traffic going to and from the target machine.

$ sudo tcpdump -i en1

How we can detect the attack?
We can use aprwatch for example on my favorite OS FreeBSD running in virtual machine,

Installation is simple as
cd /usr/ports/net-mgmt/arpwatch/
make install
...
then you have to add
arpwatch_enable="YES"
in to your /etc/rc.conf
...
 
And last but not least is to enable promiscuous mode on VMware vSwitch portgroup where arpwatch vm is connected to. The best way is to create another portgroup (single port is enough) with the same VLAN ID as protected VLAN and in Security we have to set Promiscuous Mode to Accept. 

... then arpwatch send message to syslog. So you can see something similar in your /var/log/messages
Oct 31 09:08:02 ips arpwatch: flip flop a.b.c.d 0:50:56:8d:2e:bc (54:52:0:fe:47:95)

Arpwatch can also send a e-mail message about incident. The message looks like

hostname:
ip address: 95.80.240.1
ethernet address: 54:52:0:fe:47:95
ethernet vendor:
old ethernet address: 0:50:56:8d:2e:bc
old ethernet vendor: VMWare, Inc.
timestamp: Wednesday, October 31, 2012 8:57:33 +0100
previous timestamp: Wednesday, October 31, 2012 8:57:33 +0100
delta: 0 seconds

How we can protect against the attack?

Well this is another story. It really depends on your environment but in my environment I have vSphere Distributed Switch and all virtual machines have VMtools installed so I trust VMware MAC:IP associations and based on these information (MAC) I can find the attacker port in distributed switch and disable it.

And that's just a small step to do it in automated way by leveraging VMware vCLI (aka VMware Perl SDK).

WARNING!!!

Currently described solution works only on single ESX host and doesn't work among more ESX hosts because of virtual networking principles. VMware virtual switch (or module of distributed switch) is not a switch but port extender. The difference is significant. The ARP poisoning attack is based on permanently unicasting ARP replies to the victim so when Arpwatch server is not on the same ESX as the attacker or the victim then even promiscuous port on distributed switch portgroup will not help us to catch it because arp reply packet is not visible on Arpwatch server.

The solution would be to have arpwatch VM on each ESX host. VMware introduced the concept of ESX agents (aka Agent Virtual Machines) so I believe this is the right use case for ESX agent implementation.
For more information about "Deploying vSphere Solutions, vServices, and ESX Agents" read this document.

I have to test this architecture ... so stay tuned.

Any comments or thoughts are appreciated.

Accessing Microsoft SQL Server from Linux using DBD::Sybase

Citation from: http://www.perlmonks.org/?node_id=392385
Author:  Lindsay Leeds (2004 Sep 20)

Recently, I made yet another attempt to get Perl to access Microsoft SQL Server using DBD.  Usually, when I want to connect to a Microsoft SQL Server, it is from Perl on Windows.  So I take the easy route and use DBD::ODBC and use an ODBC connection.  This time though, I wanted to connect to Microsoft SQL Server 2000 from a Linux box.  Having no ODBC to fall back on, I looked for native DBD driver of some sort.
It took me several hours of struggling to make it work.  I almost gave up several times, so I am writing outline to help anyone else trying to accomplish this same task.
In the end, we will use the DBD::Sybase perl module from CPAN to access the Microsoft SQL Server.  Before we can do that however, we must first compile the freetds library.

Note: From now on I will refer to Microsoft SQL Server as SQL Server.  Please do not confuse this with a generic sql server.  We can all now pause to gripe about the lack of imagination in product naming at Microsoft.
Compiling Freetds
Download and compile freetds from http://www.freetds.org/.

once you unzip and untar it, enter the directory and run:

./configure --prefix=/usr/local/freetds --with-tdsver=7.0
make
make install

Configuring Freetds
Now we have the freetds compiled, but we still have configure it.  This is the part that threw me off and is so different from other DBD drivers.  The DBD::Sybase driver will ultimately be affected by the contents of the /usr/local/freetds/etc/freetds.conf file.  If that file is not configured correctly, your DBD::Sybase connection will fail.
Okay, now that we have established there is a  relationship between the freetds.conf file and the DBD::Sybase module, let's edit the freetds.conf file.
The strategic modifications I made to the freetds.conf file were:
1) uncomment the following lines and modify if necessary:

try server login = yes
try domain login = no

Note: this forces the module to attempt a database login instead of a domain login.  I could not get domain login to work, though I will admit I did not try very hard.
2) uncomment the following line and modify if necessary:

tds version = 7.0

This supposedly sets the default tds version to establish a connection with.  I have only SQL Server 2000 servers, and they won't talk at any lower version.  So I set it to 7.0.  If for some reason you had older SQL Servers, you might leave it at the default 4.2.
3) create a server entry for my server sql1:

[sql1]
        host = sql1
        port = 1433
        tds version = 8.0
    


[download]

Note: My server here is sql1.  Ping sql1 worked, so I am sure I can resolve it using DNS.  You can also specifcy an ip address instead of the host name.  The sql1 in the brackets is just a descriptor.  It could be 'superduperserver' and it would still work as long as my 'host =' is set correctly.  I tried 'tds version 7.0' for my SQL Sever 2000 and it worked.  Version 5.0 though resulted in an error.  You might want to verify your SQL Server is listening on port 1433 with a 'netstat -a -n' run from the command line on the SQL Server.
At this point you can verify your configuration.

/usr/local/freetds/bin/tsql -S sql1 -U sqluser

You will then be prompted for a password and if everything is well, you will see a '1)' waiting for you to enter a command.  If you can't get the 1) using tsql, I doubt your DBD::Sybase perl code is going to work.  Please note that sqluser is not an Active Directory/Windows Domain user, but an SQL Server user.
Compiling DBD::Sybase
Now that we have the freetds library prerequisite for DBD::Sybase installed and configured, we can compile the DBD::Sybase perl module.  Obtain it from www.cpan.org if you haven't already.
once you have untarred it and are in the directory, run:

export SYBASE=/usr/local/freetds
perl Makefile.PL
make
make install

Note: The export line is to let the compilation process know where to find the freetds libraries.

Using DBD::Sybase
You are now ready to test your DBD::Sybase module.

#!/usr/bin/perl

use DBI;

$dsn = 'DBI:Sybase:server=sql1';

my $dbh = DBI->connect($dsn, "sqluser", 'password');
die "unable to connect to server $DBI::errstr" unless $dbh;

$dbh->do("use mydatabase");

$query = "SELECT * FROM MYTABLE";
$sth = $dbh->prepare ($query) or die "prepare failed\n";
$sth->execute( ) or die "unable to execute query $query   error $DBI::errstr";

$rows = $sth->rows ;
print "$row rows returned by query\n";

while ( @first = $sth->fetchrow_array ) {
   foreach $field (@first) {
      print "field: $field\n";
   }
}

Good luck!

Adding ESXi 5.1 to "5.1 vCenter Appliance" fail

Finally I found time to install vSphere 5.1 in my home lab. I have 5.0 environment running so I've bought another old DELL PE 2950 on czech "ebay like" system Aukro (www.aukro.cz) for just 6.500 CZK (approx. 330 USD) to leave my current lab untouched and try 5.1.

So, I upgraded BIOS and DRAC to latest firmwares and installed DELL version of ESXi 5.1 (embedded) on my DELL PE 2950. Then I installed vCenter appliance (OVF) on top of this new ESX.

I was able to add my old ESX5.0 to this new vCenter but not the new one.

WHAT'S WRONG???

Troubleshooting process ...

In vCenter event log I can see message: "A general system error occurred: Timed waiting for vpxa to start".

In /var/log/vpxa.log ...

2012-10-07T08:33:29.941Z [FFFE1B90 error 'SoapAdapter'] Unsupported namespace "urn:vpxa3" in content of SOAP body
-->
--> while parsing SOAP body
--> at line 9, column 0
-->
--> while parsing SOAP envelope
--> at line 2, column 0
-->
--> while parsing HTTP request before method was determined
--> at line 1, column 0

So, what  versions am I running?


ESXi 5.1.0 (799733)
vCenter Appliance 5.0.0 (755629)


Oooops ... I believed I installed vCenter 5.1 because in my local install archive it is stored as
VMware-vCenter-Server-Appliance-5.1.0.5100-799730_OVF10


So the cause is clear ... bad installation image management cost me one hour of troubleshooting :-(

NAKIVO - another virtual infrastructure backup software

NAKIVO (http://nakivo.com) is another virtual infrastructure backup software. It can be installed on Windows or Linux (Ubuntu) server. Linux installation is something which interest me. I have to test it and compare it against Veeam Backup and Replication. 

New Nexus 1000v (2.1) will be also available as free edition

Source

Nexus 1000v version 2.1 will have (2.1 is currently beta) two editions. Essential edition is free of charge so VMware Enterprise Plus customers can leverage CISCO virtual networking. Advanced edition is paid version but with significantly enhanced features. The most interesting thing is that VSG (Virtual Security Gateway) is also included in Nexus 1000v advanced edition.