Recovering from a Forgotten Password on the Force10 S series switch

I've just spent several hours to find the recovery procedure from forgotten password. Google returned just one relevant result to the Force10 tech tip page "How Do I Reset the S-Series to Factory Defaults?". However the procedure doesn't work because there is not "Option menu" during system boot. It is most probably old and deprecated procedure.

Here is the new procedure so I hope google will return it for other people looking for correct procedure.

Procedure to recovery from forgotten password on Force10 S-series switches:

1. Use serial console
2. Power off and then Power on all of the power modules
3. Wait for message similar to "Hit Esc key to interrupt autoboot: 5" and press Esc key to go to Boot Loader (uBoot) interactive shell
4. Change environment variable – setenv stconfigignore true - (uBoot - boot loader interactive shell)
5. Save the changes - saveenv - (uBoot - boot loader interactive shell)
6. Continue to boot the system – boot  - (uBoot - boot loader interactive shell)
7. Default configuration is loaded so console login authentication is disabled by default
8. Go to EXEC mode - en - (FTOS command line)
9. Load startup configuration -  copy startup-config running-config - (FTOS command line)
10. Now you can reconfigure the switch to change or add user login credentials
11. Save configuration -  copy running-config startup-config - (FTOS command line)
12. Reload the switch just for verification -  reload - (FTOS command line)

This procedure was tested on Force10 S4810 and S60.

Simple TFTP server for windows

Anybody working with networking equipment need simple tftp server. Typical use case is to download and/or upload switch configuration and to perform firmware upgrades.

I generally like simple tools which allow me to do my work quickly and efficiently.  That's the reason I really like portable version of TFTP32.

Fore more information about TFTP32 go here.

Microsoft Cluster Service (MSCS) support on VMware vSphere

Microsoft Cluster Service (MSCS) is Microsoft cluster technology required shared storage supporting SCSI reservation mechanism. Microsoft has introduced new - perhaps more modern and more descriptive - name for the same technology. New name is "Microsoft Failover Cluster" so don't be confused with different names.

VMware has supplementary documentation called "Setup for Failover Clustering and Microsoft Cluster Service" covering the subject. Here is online documentation for vSphere 5.5 and here for vSphere 5.1. This documentation is a must to read to fully understand what is and what is not possible.

However documentation is relatively old so if you want to know up-to-date information you have to leverage VMware KB. Here are two KB articles related to the topic

• Microsoft Cluster Service (MSCS) support on ESXi/ESX (1004617)
• Microsoft Clustering on VMware vSphere: Guidelines for supported configurations (1037959)

And here is general advice.
Plan, plan, plan ... design ... review ... implement ... verify.
I hope you know what I mean.

DELL Force10 : Initial switch configuration

[ Previous | DELL Force10 : Series Introduction ]

---

I assume you have serial console access to the switch unit to perform initial switch configuration. I guess it will not impressed you that to switch from read mode to configuration mode you  have to use command

conf

... before continue I would like to recap some important basic FTOS commands we will use later in this blog post. If you want to exit from configuration mode or even from deeper configuration hierarchy you can do it with one or several

exit 

commands which will jump to the upper level of configuration hierarchy and eventually exit conf mode. However the easiest way to leave configuration mode is to use command

end

which will exit configuration mode immediately.

The last but very important and very often used command is

write mem

which will write your running switch configuration to the flash and therefore configuration will survive the switch reload. You can do the same with more general command

copy running-config startup-config

If you want to display running configuration you can use command

show running-config

Whole configuration can be pretty long, so if you are interested only on some part of running configuration you can use following commands

show running-config interface managementethernet 0/0
show running-config interface gigabitethernet 0/2
show running-config spanning-tree rstp
show running-config boot

As you can see FTOS command line interface (cli) is very similar to CISCO.

Ok, so after basics let's start with initial configuration. Switch configuration usually begins with host name configuration. It is generally good practice to use unique host names because you know on which system you are logged in.

hostname f10-s60

As a next step I usually configure management IP settings and enable remote access. You have to decide if you will use in-band management leveraging normal IP settings usually configured on dedicated VLAN interface just for system management or you will leverage dedicated out-of-band management port. In example below you can see

• out-of-band management port for system management IP settings
• how to create admin user
• how to enable ssh to allow remote system management

interface ManagementEthernet 0/0
  ip address 192.168.42.101/24
  no shutdown
exit
management route 0.0.0.0/0 192.168.42.1

username admin password YourPassword privilege 15

ip ssh server enable

Now you have to decide if you want to enforce login for users connected via local console. By default there is no login required which can by security risk especially in environments without strict physical security rules. Below is configuration which enforce local login credentials when using serial console.

 aaa authentication login default local

At this point I would like to note that Force10 switch has all capabilities and features disabled in default factory configuration. That's the reason why for example each switch interface must be explicitly enabled before usage because all interfaces are in shutdown state by default.

Before you enable any switch interface it is good practice to enable spanning tree protocol as security mechanism against potential loops in the network. Once again, spanning tree feature is not enabled by default so you have to do it explicitly. Force10 FTOS has implemented all standard and even some non-standard (CISCO proprietary) spanning tree protocols like PVSTP+. On the latest FTOS version following spanning tree protocols are supported:

• STP (Spanning Tree Protocol)
• RSTP (Rapid Spanning Tree Protocol)
• MSTP (Multiple Spanning Tree Protocol)
• PVSTP+ (Per-VLAN Spanning Tree Plus)

Bellow is configuration example which enables standard rapid spanning tree protocol (aka RSTP) ...

protocol spanning-tree rstp
  no disable 

Another decision you have to do before implementation is the location from where do you want to boot your switch operating system. On some Force10 models (for example on S60) is default primary boot location TFTP server  ...

boot system stack-unit 0 primary tftp://192.168.128.1/FTOS-SC-8.3.3.7.bin
boot system stack-unit 0 secondary system: A:
boot system stack-unit 0 default system: B:
boot system gateway 192.168.128.1

You can see that primary boot location is TFTP server. If you don't have tens or hundreds of switches you usually don't want to load FTOS remotely from TFTP server but from internal flash in the switch. Although default switch configuration would work because if TFTP server boot fails switch boot sequence continue with secondary location  but it's better to configure the switch boot sequence explicitly base on your requirements. Below is typical boot sequence configuration.

boot system stack-unit 0 primary system: A:boot system stack-unit 0 secondary system: B:boot system stack-unit 0 default system: A: no boot system gateway

Next thing you should check is what FTOS version do you have. Below is the command how you can check it ...

f10-s60#show version
Dell Force10 Networks Real Time Operating System Software
Dell Force10 Operating System Version: 1.0
Dell Force10 Application Software Version: 8.3.3.7
Copyright (c) 1999-2011 by Dell Inc.
Build Time: Sat Nov 26 01:23:50 2011
Build Path: /sites/sjc/work/build/buildSpaces/build20/E8-3-3/SW/SRC
f10-s60 uptime is 4 minute(s)
System image file is "system://A"
System Type: S60
Control Processor: Freescale MPC8536E with 2147483648 bytes of memory.
128M bytes of boot flash memory.
  1 48-port E/FE/GE (SC)
 48 GigabitEthernet/IEEE 802.3 interface(s)
  2 Ten GigabitEthernet/IEEE 802.3 interface(s)

You can see FTOS version 8.3.3.7 which is not the latest one as the latest FTOS version at the time of writing this article is 8.3.3.9 and boot loader 1.0.0.5. It is generally good practice to upgrade FTOS to the latest version before performing verification test and going into production. For the latest version you have to go to http://www.force10networks.com and sign in. If you don't have Force10 account you can register there. Please note that each Force10 switch model use different FTOS versions. So there can be FTOS 9.4.x for model S4810 and 8.3.x for S60.

Now I'll show you how to do FTOS and boot loader upgrade.
FTOS should be upgraded first and Boot Loader later ...

upgrade system tftp: A:
upgrade system stack-unit all A: (applicable only if you have stack configured)
upgrade boot ftp: (applicable only if  new bootloader compatible with FTOS code exists)
reload

You can check current FTOS version

show version

and if you want to know what FTOS version do you have on which boot bank you can 

show boot system stack-unit 0

By the way, have I told you there are two boot banks? Boot bank A: and boot bank B:so you can choose primary and secondary boot location. We have already covered boot configuration but here it is again ...

conf

  boot system stack-unit 0 primary system: A:
  boot system stack-unit 0 secondary system: B:

FTOS is loaded by boot loader and current Boot Loader can be displayed by command below

show system stack-unit 0

Hope this post is helpful for IT community. In case you have any question, suggestion or idea on improvements please share your thoughts in in the comments.

Stay tuned and wait for next article ...

---

[ Next | DELL Force10 : Interface configuration and VLANs ]

DELL Force10 : Series Introduction

I have just decided to write dedicated blog post series about DELL Force10 networking. Why?

Who knows me in person is most probably aware that my primary professional focus is on VMware vSphere infrastructure and datacenter enterprise hardware. Sometimes I have discussion with infrastructure experts, managers and other IT folks what is the most important/complex/critical/expensive vSphere component. vSphere component is meant by compute (servers), storage and networking.  I thing it is needless discussion because all components are important and have to be integrated into single integrated system fulfilling all requirements, dealing with known constraints and mitigating all potential risks. Such integrated infrastructures are very often called POD which stands, as far as I know, for Performance Optimized Datacenter. These integrated systems are, from my point of view, new datacenter computers having dedicated but distributed computing, storage and networking components. I would prefer to call such equipment as "Optimized Infrastructure Block" or "Datacenter Computer" because it is not only about performance but also about reliability, capacity, availability, manageability, recoverability and security. We call these attributes infrastructure qualities and whole infrastructure block inherits qualities from sub-components. Older IT folks often compare this concept with main frame architectures however nowadays we usually use commodity x86 hardware "a little bit" optimized for enterprise workloads. 

By the way that's one of the reason I like current DELL datacenter product portfolio because DELL has everything I need to build POD - server, storage systems and now also networking so I'm able to design single vendor infrastructure block with unified support, warranty, etc. Maybe someone don't know but DELL acquired EqualLogic and Compellent storage vendors some time ago, but more importantly for this blog post, DELL also acquired well known (at least in US) datacenter networking producer Force10. For official acquisition details look here.

But back to the networking. Everybody would probably agree that networking is very important part of vSphere infrastructure because of several reasons. It provides interconnect between clustered components - think about vSphere networks like Management, vMotion, Fault Tolerance, VSAN, etc. It also routes network traffic to the outside world. And sometimes it even provides storage fabrics (iSCSI, FCoE, NFS). That's actually the reason why I'm going to write this series of blog posts about DELL Force10 networking - because of networking importance. However I don't want to write about  legacy networking but modern networking approach for next generation virtualized and software defined datacenters.

Modern physical networking is not only about hardware (burned intelligence in ASICs with high bandwidth, fast and low latency interfaces) but also in software. The main software sits inside DELL Force10 switches. It is switch firmware called FTOS - Force10 Operating System (see. for more general information about FTOS look here).  However, today it is not only about switch embedded firmwares but also about whole software ecosystem - managements, centralized control planes, virtual distributed switches, network overlays, etc.
 
In future articles I would like to deep dive into FTOS features, configuration examples and virtualization related integrations.

Next, actually first technical article in this series will be about about typical initial configuration of Force10 switch. I know it is not rocket science but we have to know basics before taking off. In the future I would like to write about more complex designs, capabilities and configurations like

• Multi-Chassis Link Aggregation (aka MC-LAG). In Force10 terminology we call it VLT - Virtual Link Trunking.
• Virtual Routing and Forwarding (aka VRF). Some S-series Force10 models with FTOS 9.4 support VRF-lite.
• some Software Define Networking (aka SDN) capabilities like python/perl scripting inside the switch, REST API, VXLAN hardware VTEP, Integration with VMware Distributed Virtual Switch, Integration with VMware NSX, OpenFlow, etc.

If you prefer some topics please let me know in comments and I'll try to prioritize it. Otherwise I'll write future posts based on my preferences.   

So let's finish blog post series introduction and start some technical stuff and begin with deep dive into switch initial configuration! Just click next and below ...

---

[ Next | DELL Force10 : Initial switch configuration ]

Potential Network Black Hole Issue

When I do vSphere and hardware infrastructure health checks very often I meet misconfigured networks usually but not only in blade server environments. That's the reason I've decided to write blog post about this issue. The issue is general and should be considered and checked for any vendor solution but because I'm very familiar with DELL products I'll use DELL blade system and I/O modules to show you deeper specification and configurations.  

Blade server chassis typically have switch modules as depicted on figure below.

When blade chassis switch modules are connected to another network layer (aggregation or core) than there is possibility of network black hole which I would like to discuss deeply on this post.

Let's assume you will lost single uplink from I/O module A. This situation is depicted below.

In this situation there is not availability problem because network traffic can flow via second I/O module uplink port. Indeed, there is only half of uplink bandwidth so there is potential throughput degradation and therefore congestion can occur but everything works and it is not availability issue.

But what happen when second I/O switch module uplink port fails? Look at figure below.

   
If I/O switch module is in normal switch mode then uplink ports are in link-down state but downlink server ports are in link-up state and therefore ESX host NIC ports are also up and ESX teaming don't know that something is wrong down the path and traffic is sending to both NIC uplinks. We call this situation "black hole" because traffic routed via NIC1 will never reach the destination and your infrastructure is in trouble.

To overcome this issue some I/O modules in blade systems can be configured as I/O Aggregator. Some other modules are designed as I/O Aggregators by default and it cannot be changed.

Here are examples of DELL blade switch modules which are switches by default but can be configure to work as I/O Aggregators (aka Simple Switch Mode):

• DELL PowerConnect M6220
• DELL PowerConnect M6348
• DELL PowerConnect M8024-k

Example of implicit I/O Aggregator is DELL Force10 IOA.

Another I/O Aggregator like option is to use Fabric Extender architecture implemented in DELL Blade System as CISCO Nexus B22. CISCO FEX is little bit different topic but is also help you to effectively avoid our black hole issue.

When you use "simple switch mode"you have limited configuration possibilities. For example you can use the module just for L2 and you cannot use advanced features like access control lists (ACLs). That can be reason you would like to leave I/O module in normal switch mode. But even you have I/O modules in normal switch mode you can configure your switch  to overcome potential "black hole" issue. Here are examples of DELL blade switches and technologies to overcome this issue:

• DELL PowerConnect M6220 (Link Dependency)
• DELL PowerConnect M6348 (Link Dependency)
• DELL PowerConnect M8024-k (Link Dependency)
• DELL Force10 MXL (Uplink Failure Detection)
• CISCO 3130X (Link State Tracking)
• CISCO 3130G (Link State Tracking)
• CISCO 3032 (Link State Tracking)
• CISCO Nexus B22 (Fabric Extender)

If you leverage any of technology listed above then link states of I/O module switch uplink ports are synchronized to the configured downlink ports and ESX teaming driver can effectively do ESX uplink high availability. Such situation is depicted in figure below.

Below are examples of detail CLI configurations of some port tracking technologies described above.


DELL PowerConnect Link Dependency

Link dependency configuration on both blade access switch modules can solve "Network Black Hole" issue.

 ! Server port configuration  
 interface Gi1/0/1  
 switchport mode general  
 switchport general pvid 201  
 switchport general allowed vlan add 201  
 switchport general allowed vlan add 500-999 tagged  
 ! Physical Uplink port configuration   
 interface Gi1/0/47  
 channel-group 1 mode auto  
 exit  
 ! Physical Uplink port configuration   
 interface Gi1/0/48  
 channel-group 1 mode auto  
 exit  
 ! Logical Uplink port configuration (LACP Port Channel)  
 interface port-channel 1  
 switchport mode trunk  
 exit   
 ! Link dependency configuration  
 link-dependency group 1  
 add Gi1/0/1-16  
 depends-on port-channel 1  

Force10 Uplink Failure Detection (UFD)

Force 10 call link dependency feature UFD and here is configuration example

 FTOS#show running-config uplink-state-group  
 !  
 uplink-state-group 1  
 downstream TenGigabitEthernet 0/0  
 upstream TenGigabitEthernet 0/1  
 FTOS#  

The status of UFD can be displayed by "show configuration" command

 FTOS(conf-uplink-state-group-16)# show configuration  
 !  
 uplink-state-group 16  
 description test  
 downstream disable links all  
 downstream TengigabitEthernet 0/40  
 upstream TengigabitEthernet 0/41  
 upstream Port-channel 8  

CISCO Link State Tracking

Link state tracking is a feature available on Cisco switches to manage the link state of downstream ports (ports connected to Servers) based on the status of upstream ports (ports connected to Aggregation/Core switches).

How to fix broken VMFS partition?

I had a need of more storage space in my lab. The redundancy was not important so I changed RAID configuration of local disk from RAID 1 to RAID 0. After this change the old VMFS partition left on the disk volume. That was the reason I have seen just half of all disk space when I was trying to create new datastore. Another half was still used by old VMFS partition. You can ssh to ESXi host end check disk partition by partedUtil command. 

 ~ # partedUtil get /dev/disks/naa.690b11c0034974001ae597300bc5b3aa  
 Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)? This will also fix the last usable sector as per the new size. diskSize (1169686528) AlternateLBA (584843263) LastUsableLBA (584843230)  
 Warning: Not all of the space available to /dev/disks/naa.690b11c0034974001ae597300bc5b3aa appears to be used, you can fix the GPT to use all of the space (an extra 584843264 blocks) or continue with the current setting? This will also move the backup table at the end if is is not at the end already. diskSize (1169686528) AlternateLBA (584843263) LastUsableLBA (584843230) NewLastUsableLBA (1169686494)  
 72809 255 63 1169686528  
 1 2048 584843230 0 0  

Old datastore didn't contain any data so I had no problem to try fix command option by following command.

 ~ # partedUtil fix /dev/disks/naa.690b11c0034974001ae597300bc5b3aa  

When I checked disk partition on disk device I get  following output

 ~ # partedUtil get /dev/disks/naa.690b11c0034974001ae597300bc5b3aa  
 72809 255 63 1169686528  

... and then I was able to create datastore on whole disk.

DELL recommended BIOS Settings for VMware vSphere Hypervisor

Here are a list of BIOS settings  specifically regarding Dell PowerEdge servers:

• Hardware-Assisted Virtualization: As the VMware best practices state, this technology provides hardware-assisted CPU and MMU virtualization.
  In the Dell PowerEdge BIOS, this is known as “Virtualization Technology” under the “Processor Settings” screen. Depending upon server model, this may be Disabled by default. In order to utilize these technologies, Dell recommends setting this to Enabled.
• Intel® Turbo Boost Technology and Hyper-Threading Technology: These technologies, known as "Turbo Mode" and "Logical Processor" respectively in the Dell BIOS under the "Processor Settings" screen, are recommended by VMware to be Enabled for applicable processors; this is the Dell factory default setting.
• Non-Uniform Memory Access (NUMA): VMware states that in most cases, disabling "Node Interleaving" (which enables NUMA) provides the best performance, as the VMware kernel scheduler is NUMA-aware and optimizes memory accesses to the processor it belongs to. This is the Dell factory default.
• Power Management: VMware states “For the highest performance, potentially at the expense of higher power consumption, set any BIOS power-saving options to high-performance mode.” In the Dell BIOS, this is accomplished by setting "Power Management" to Maximum Performance.
• Integrated Devices: VMware states “Disable from within the BIOS any unneeded devices, such as serial and USB ports.” These devices can be turned off under the “Integrated Devices” screen within the Dell BIOS.
• C1E: VMware recommends disabling the C1E halt state for multi-threaded, I/O latency sensitive workloads. This option is Enabled by default, and may be set to Disabled under the “Processor Settings” screen of the Dell BIOS.
• Processor Prefetchers: Certain processor architectures may have additional options under the “Processor Settings” screen, such as Hardware Prefetcher, Adjacent Cache Line Prefetch, DCU Streamer Prefetcher, Data Reuse, DRAM Prefetcher, etc. The default settings for these options is Enabled, and in general, Dell does not recommend disabling them, as they typically improve performance. However, for very random, memory-intensive workloads, you can try disabling these settings to evaluate whether that may increase performance of your virtualized workloads.

 Source: http://en.community.dell.com/techcenter/virtualization/w/wiki/dell-vmware-vsphere-performance-best-practices.aspx

Long network lost during vMotion

We have observed strange behavior of vMotion during vSphere Design Verification tests after successful vSphere Implementation. By the way that's the reason why Design Verification tests are very important before putting infrastructure into production.But back to the problem. When VM was migrated between ESXi hosts leveraging VMware vMotion we have seen long network lost of VM networking somewhere between 5 and 20 seconds. That's really strange because usually you didn't observe any network lost or maximally lost of single ping from VM perspective.

My first question to implementation team was if  virtual switch option "Notify Switch" is enabled as is documented in vSphere design. The importance of this option is relatively deeply explained for example here and here. The answer from implementation team was positive so problem has to be somewhere else and most probably on physical switches. The main reason of option "Notify Switch" is to quickly update physical switch mac-address-table (aka CAM) and help physical switch to understand on which physical port VM vNIC is connected. So as another potential culprit was intended physical switch. In our case it was  virtual stack of two HP blade modules 6125XLG.

At the end it has been proven by implementation team that HP switch was the culprit. Here is very important HP switch global configuration setting to eliminate this issue.

 mac-address mac-move fast-update  

I didn't find any similar blog post or KB article about this issue so I hope it will be helpful for VMware community. 

DELL official response to OpenSSL Heartbleed

The official Dell response to OpenSSL Heartbleed for our entire portfolio of products is listed here.

Code Formatter

From time to time i'm publishing programming code source or configurations on my blog running on google blog platform blogger.com.  I'm always struggling with formatting the code.

I've just found http://codeformatter.blogspot.com/ and I'll try it next time when needed.

PRTG alerts phone call notifications

I have been asked by someone how to do phone call notification of critical alerts in PRTG monitoring system. Advantage of phone call notification against Email or SMS is that it can wake up sleeping administrator in night when he has support service and critical alert appears in central monitoring system.

My conceptual answer was ... use PRTG API to monitor alerts and make a phone call when critical alerts exist.

New generation of admins doesn't have problem with APIs but don't know how to dial voice call. That's because they are IP based generation and don't have experience with modems we played extensively back in 80's and 90's ;-)

At the end I promised them to prepare embedded system integrated with PRTG over API and dialing phone configured number in case of critical alerts.

Here is the picture of hardware prototype leveraging soekris computing platform running FreeBSD OS in RAM disk and making phone calls via RS-232 GSM modem.

Soekris computing platform and RS-232 GSM modem.

Here are relevant blog posts describing some technical details little bit deeper

• http://blog.igics.com/2013/12/freebsd-running-from-read-only-compact.html
• http://blog.igics.com/2014/01/gsmgprs-modem-siemens-es75-usefull-at.html

Difference between ESXi Shell time-outs

ESXi Advanced Settings have two timeout parameters to manage ESXi Shell timeout:

• UserVars.ESXiShellTimeOut 
• UserVars.ESXiShellInteractiveTimeOut

Both parameters are by default set to 0 which means the time-outs are disabled. However, it is good practice to set these timeouts as it has a positive impact on security.

But what values should be set there?
What is the difference between ESXiShellTimeOut and ESXiShellInteractiveTimeOut?

Let's start with description of both parameters available in ESXi hosts.

UserVars.ESXiShellTimeOut
Key: UserVars.ESXiShellTimeOut
Description: Time before automatically disabling local and remote shell access (in seconds, 0 disables). Takes effect after the services are restarted.
Value: 0
Default: 0
Read only: No
Range: 0 ≤ x ≤ 86400

UserVars.ESXiShellInteractiveTimeOut
Key: UserVars.ESXiShellInteractiveTimeOut
Description: Idle time before an interactive shell is automatically logged out (in seconds, 0 disables). Takes effect only for newly logged in sessions.
Value: 0
Default: 0
Read only: No
Range: 0 ≤ x ≤ 86400

Is it clear? No? So, here is a more human-readable explanation ...

On ESXi host are following two shells:

• ESXi Shell (Local Troubleshooting Shell available in Local Console) 
• SSH Service (Remote Shell)

What does UserVars.ESXiShellTimeOut setting?
If you want to troubleshoot ESXi locally or remotely, you have to enable services ESXi Shell or SSH. We are talking about ESXi services highlighted in the figure below.

ESXi Shell and SSH services

Advanced setting parameter UserVars.ESXiShellTimeOut is the time in seconds how long are these services (ESXi Shell and SSH) running when explicitly started by vSphere admin for troubleshooting. The default value of UserVars.ESXiShellTimeOut is 0, therefore these services will stay up and running until vSphere admin stop them manually. There is a potential risk that vSphere admin will forget to stop these services after troubleshooting and it would have a negative impact on security as these services should be running only for troubleshooting. That's the reason for best practice to set this timeout setting to some reasonable time for troubleshooting and to stop services automatically after configure time. I think the reasonable time for troubleshooting is for example 4 hours which is 14400 seconds.

What does UserVars.ESXiShellInteractiveTimeOut setting?
If you have started ESXi services ESXi Shell or SSH Service you can do local or remote troubleshooting. Advanced settings parameter UserVars.ESXiShellInteractiveTimeOut configures timeout from the active (interactive) session. In other words, if you do a troubleshooting via SSH session (SSH service must be running) this setting will close the active SSH session in case of idle (no interaction) is higher then UserVars.ESXiShellInteractiveTimeOut value. So, it will close your interactive SSH session but SSH service will stay up and running so you can reconnect and continue with troubleshooting. The same principle applies to local troubleshooting with ESXi Shell.

Do these settings have any impact on Direct Console UI?

No. None of the above two advanced settings have any impact on Direct Console UI (aka DCUI, ESXi Text-based Menu available in Local Console). DCUI is not considered as a shell.

Hope the explanation above help you to understand the difference. If not, please leave the comment below the blog post.

Script to create VMware SRM database

VMware SRM installer creates tables in database automatically but you must prepare MS-SQL database, DB schema and ODBC data source before SRM installation.

Note: SRM has technical requirement to use database schema having the same name as DB user.

Here is the script to prepare MS-SQL database (SITE-A-SRM), schema (SRMlogin) and DB user (SRMlogin) with password (SRMpassword) for SRM:

CREATE DATABASE [SITE-A-SRM];
GO
USE [SITE-A-SRM];
GO
CREATE LOGIN [SRMlogin] WITH PASSWORD=N'SRMpassword', DEFAULT_DATABASE=[SITE-A-SRM], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF
BEGIN
EXEC sp_changedbowner 'SRMlogin'
EXEC sp_executesql N'CREATE SCHEMA SRMlogin'
END

I hope it is obvious to everybody that the same procedure (probably with little bit different colored items) must be done in database on SITE-B for second SRM instance.

I generally believe that scripts are better, faster, more consistent and less error-prone then manually creating something via GUI.

Credits: SQL script has been created  by customer's database admin (Ladislav Hajek) contributing with me on SRM project.

VMware Site Recovery Manager network ports

Here are documented network port numbers and protocols that must be open for Site Recovery Manager, vSphere Replication, and vCenter Server. Very nice and useful VMware KB article however during my last SRM implementation I have realized that some ports are not documented on KB article mentioned above.

We spent some time with customer's network admin to track what other ports are required so here they are. These other ports must be opened for full functionality of  SRM + vSphere Replication.

Source	Target	Protocol_Port
SRM SERVER	VCENTER SERVER	http_80, https_443, tcp_80, tcp_8095
SRM SERVER	ESX HOSTS	tcp/udp_902
VCENTER SERVER	SRM SERVER	http_9085, https_9086, tcp_8095, tcp_9085
REPLICATION APPLIANCE	VCENTER SERVER	tcp_80
REPLICATION APPLIANCE	ESX HOSTS	http_80, tcp/udp_902
ESX HOSTS	REPLICATION APPLIANCE	tcp_31031, tcp_44046
VCENTER SERVER	VCENTER SERVER	http_80, tcp_10443, https_443

If you use external MS-SQL database don't forget to allow network communication to database server. It is typically udp_1434 (MS-SQL Resolver) and tcp port of MS-SQL instance.

Credits: Network protocols and ports has been grabbed by customer's network admins (Ladislav Hajek and Ondrej Safranek) contributing with me on SRM project.

Storage design verification - performance test

I had a unique chance to work with relatively big customer on VMware vSphere Architecture Design from the scratch. I prepared vSphere Architecture Design based on their real business and technical requirements and the customer used the outcome to prepare hardware RFI and RFP to buy the best hardware technology on the market from technical and also cost point of view. Before design I did capacity and performance monitoring of customer's current environment and we used these numbers for capacity sizing of new infrastructure. I designed the logical hardware architecture of fully integrated compute/storage/network infrastructure blocks (aka PODs - Performance Optimized Datacenter) where PODs are leveraged as vSphere Clusters with predefined and well known performance characteristics and ratios among CPU, memory, storage and network.

We all know the most complicated is storage performance sizing especially with leveraging automated storage tiering technology existing in almost all modern storage systems. I was able to prepare some estimations based on standard storage calculations and my experience however we left final responsibility on hardware vendors and their technical pre-sales teams. Our requirement was pretty easy - 60TB of capacity and 25,000 IOPSes generated from servers in R/W ratio 70/30.

Validation and acceptance test of storage was clearly defined. The storage systems must be able to handle a 25,000 IOPS workload synthetically generated leveraging free tool IOmeter. Test environment was composed from 250 linux VMs with single Worker (IOmeter dynamo). All these workers were connected to single IOmeter GUI reporting total workload nicely in single place. Each of 250 workloads were defined as described below:

• Outstanding IO: 1
• IO size: 64KB
• Workload pattern Random/Sequence ratio: 70:30
• Read/Write Ratio: 70:30

Hardware vendor was informed that we will run this workload during 24 hours and we want to see average performance of 25,000 IOPSes with response times below 25 ms.

Selected hardware vendor delivered storage with following disk tiers:

• Tier 1: 8x 400GB 6G SAS 2.5” MLC SSD R5 (3+1)
• Tier 2: 128x 300GB 6G SAS 15K 2.5” HDD R5 (7+1)
• Tier 3: 40x 900GB 6G SAS 10K 2.5” HDD R5 (7+1)

We asked hardware vendor how LUNs for vSphere datastores should be created to fulfil our capacity and performance requirement. The vendor recommended to leverage automated storage tiering and stretch the LUN across all disk tiers. We were able to choose particular disk tier for first write into LUN. It was selected to Tier 2. It is important to mention that process for automated storage tiering runs by default one a day and it can be changed. However from my experience it is usually even worse because if you generate continuous storage workload and AST background process starts then it generate another workload on already highly loaded storage and response times becomes unpredictable and sometimes it makes even bigger problem. AST is good technology for typical enterprise workloads when you have good capacity and performance ratio among Tiers and you have tiering window when your storage is lightly loaded and you can run AST background process to optimize your storage system. It's important to mention that AST requires really good planning and it is not good technology for continuous stress workload. But that's what hardware vendor pre-sales team has to know, right?

The result where we are right now is that we are able to achieve 15,600 front-end IOPSes which can be simply recalculated into back-end IOPSes based on read/write ratio and write penalty which is 4 for RAID 5. On figure below is screenshot from IOmeter just for illustration but final achievement was really 15,600 IOPS average from the beginning of the test.

Backend IOPSes = 10920 reads + ( 4 x 4680 writes) = 29,640 which can be recalculated into IOPSes per disk = 29640/128 = 231 IOPS. 231 IOPSes per 15k/rpm disk is pretty high and other Storage tiers are not leveraged so we are calling hardware vendor and asking how we can achieve our numbers.

BTW: this is acceptance hardware test and vendor has to prove this numbers otherwise he has to upgrade his storage (at his expense) or take the hardware out and return money.

To be continued ... stay tuned ...

UPDATE: Long story short ... at the end of the day storage vendor had to add additional disk enclosures with more spindles. Storage vendor had to pay it and it is worth to mention that it was significant additional cost covered 100% from his margin!!! No additional single cent paid by my customer. It is just another reason to engage subject matter expert for Infrastructure Design because when logical infrastructure design along with test plan is prepared before RFI and RFP your RFP strict requirements can be properly written and clearly articulated to all tender participants. 

VMware vSphere 5 Memory Management and Monitoring

Do you think you fully understand VMware vSphere ESXi memory management?
Compare your understanding with memory diagram at  VMware KB 2017642.

Now another question. Do you still think you are able to exactly know how much memory is used and how much is available? Do you? It is very important to know that this task is complex in any operating system because of lot of memory virtualization layers, memory garbage algorithm, caching, buffering, etc .... therefore nobody is able to know exact numbers. Of course you can monitor ESX memory usage but that is always estimated number.

Real memory over allocation and potential memory issue can be monitored by several mechanisms

• Running VMs ballooning - because ballooning starts only when there is not enough memory
• VMs (ESX) swapping - mainly swap in/out rate higher then 0 because that's the real indication you have memory problem

DELL Force10 S4810 fans

The S4810 comes from the factory with one power supply and two fan modules installed in the chassis. Both the fan module and the integrated fan power supply are hot-swappable if a second (redundant) power supply is installed and running. With redundant power supplies, traffic will not be interrupted if a fan module is removed. In addition to the integrated fan power-supply modules, fan modules can be ordered separately and additional modules can be inserted in the chassis.

The S4810 system fans are supported with two air-flow options. Be sure to order the fans that are suitable to support proper ventilation for your site. Use a single type of fan in your system. Do not mix Reverse and Normal air-flows in a single chassis. The system will shut down in one minute if the airflow directions are mismatched.

Air-flow options:

•     Normal is airflow from I/O panel to power supply
•     Reversed is airflow from power supply to I/O panel

So if you want to use S4810 as a top of rack switch for servers in the server rack you probably want to have ports (I/O panel) on the rear of the rack to simplify cable management. The reversed air-flow option is the way to go for this use case.

VMware vShiled Manager - VXLAN limit

We all know that all technologies has some limits. Only important thing is to know about particular limits limiting your solution.

Do you know VMware vShield Manager has limit for number of virtual networks?

There is the limit 5,000 networks even you use VXLAN network virtualization. So even VXLAN can have theoretically up to 16M segments (24-bit segment ID) you are effectively limited to 5,000 which is not significantly more then old VLAN ID limit of 4,096 (12-bit segment ID).

The most strange thing is that this limit is not documented on vSphere Configuration Maximums. There are documented only following network limits:

• Static/Dynamic port groups per distributed switch = 6,500
• Ports per distributed switch = 60,000
• vCloud Director "Number of networks" = 10,000

Thanks Tom Fojta for this information and link to VMware KB 2042799.

On top of that the current VMware VXLAN implementation provide VXLAN based network overlay only in single vCenter domain so it will not help you with network tunnel for DR (VMware SRM) use case where two vCenters are required.   

So only two benefits of current VMware VXLAN implementation I see today are:

• software defined network segments in single vCenter domain allowing automation of VXLAN provisioning. Nice blog about it is here.
• split between physical networking segments (system VLANs, internet VLANs, MPLS VLANs, ...) and multi-tenant virtual network segments used for tenant's private use.

To be honest even those two benefits are very useful and limits will increase year by year as technology evolves and matures. That's usual technology behavior.

Good or Bad Backup Job?

Veeam is very good backup software specialized on agent-less VM backups. But we all  know that bugs are everywhere and Veeam is not the exception. If you have VMware vSphere VM with independent disk Veeam cannot successfully perform a backup. That's logical because independent disks cannot have snapshots which are mandatory for agent-less VM backups leveraging VMware API for Data Protection (aka VADP). However the problem is that backup job of independent virtual disk is green. That can give you impression that everything is OK. But it is not. You have false expectation that you have correct backup. But you haven't and if you don't check logs you can find it really late ... during restore procedure which is not possible.

You can see what happen below on the screenshot.     

Click to enlarge

The correct behavior would be if backup job fails and backup administrator can repair the issue. This behavior was seen in Veeam version 6.5. Veeam support has been informed about this wrong behavior so it hopefully will be repaired in the future.

Performance Data charts for datastore LUNs report the message: No data available

Performance Data charts for datastore LUNs are extremely useful to have clue to understand storage performance trend.

However sometimes you can see message like this

"Performance Data charts for datastore LUNs report the message: No data available"

I didn't know the root cause. Recently colleague of mine told me he has found what is the root cause which is described at VMware KB 2054403.

Workaround is to not use network virtual adapter E1000E. If you have larger environment it's not big fun to search these adapters. My colleague wrote useful PowerCLI one-liner to find VM with E1000E which should be manually changed. Here is the my colleague's script:

Get-VM | Get-NetworkAdapter | Where-object {$_.Type -like "Unknown" -or $_.Type -like "E1000E" } | Select @{N="VM";E={$_.Parent.Name}},Name,Type | export-Csv  c:\VM-Network_Interface.csv -NoTypeInformation 

He asked me to share this information with community so enjoy it.

Reserved IP addresses

Very useful Wikipedia page with list of  all "Reserved IP addresses".
http://en.wikipedia.org/wiki/Reserved_IP_addresses

VMware vSphere: Migration of RDM disk to VMDK

I have received following question from my customer ...

"We have business critical application with MS-SQL running in virtual machine on top of VMware vSphere. OS disk is vmdk but data disk is on RDM disk. We want to get rid of RDM and migrate it into normal vmdk disk. We know there are several methods but we would like to know the safest method. We cannot accept too long service downtime but we prefer certainty against shorter down time."

Let's write down customer requirements

• migrate RDM into VMDK
• migrate business critical application
• service downtime as shorter as possible
• guarantee seamless migration

So here are my recommended options ...

IMPORTANT: First of all you have to do backup before any migration.

Assumptions

• RDM disk is in virtual mode (if not, you have to remove physical RDM from VM and connect RDM in virtual mode)
• Latest system and data backup exist
• At least two datastores exists. One where VM currently resides and second one where you will do migration.
• Just for Option 1: Experience with VMware Cold Migration
• Just for Option 2: Experience with VMware live Disk Migration (aka Storage vMotion)
• Just for Option 2: Availability of VMware vSphere Storage vMotion licence

Option 1 - Cold Migration

Procedure

1. Shutdown operating system
2. Use VMware Migrate function and migrate VM in power off state to another datastore. You must select another virtual disk format (for example Lazy Zeroed) and another datastore than VM current datastore. It will convert RDM to VMDK during migration.
3. Power On VM and validate system functionality

Option 2 - Live Migration without server downtime

Procedure

1. Use VMware Migrate function and migrate VM in power on state to another datastore. You must select another virtual disk format (for example Lazy Zeroed) and different datastore than VM currently resides. It will convert RDM to VMDK during data migration.
2. Validate system functionality

Options comparison

Option 1

Advantages

• system is in power off so it is just disk conversion which is very safe method

Drawbacks

• offline migration which means service downtime

Option 2

Advantages

• No service downtime because of online disk migration without service disruption
• Leveraging your investment into VMware enterprise capabilities 

Drawbacks

• potential issues specially on disks with high load
• if there is high disk load on RDM migration will generate another I/O which can lead into worse response times and overall service quality and availability
• migration of system where all services are running so there is potential risk of data corruption but the risk is very low and mitigated by existing data backup

Dear Mr. customer. Final decision what method is better for your particular use case is up to you. Both methods are relatively safe but Option 1 is probably little bit safer and Option 2 is absolutely without downtime and totally transparent for running services inside VM.

There are even other methods how to convert RDM to VMDK but these two options are relatively easy, fast, save and doesn't require any special software. It simply leverage native vSphere capabilities.

Hope this helps.

Configure default settings on a VMware virtual distributed switch

Original blog post and full text is here. All credits go to http://kickingwaterbottles.wordpress.com

Here is the PowerCLI script that will set the ‘Teaming and Failover’ defaults on the vDS to work with etherchannel and two active uplinks.

connect-viserver vCenter
$vDSName = “”
$vds = Get-VDSwitch $vDSName
$spec = New-Object VMware.Vim.DVSConfigSpec
$spec.configVersion = $vds.ExtensionData.Config.ConfigVersion
$spec.defaultPortConfig = New-Object VMware.Vim.VMwareDVSPortSetting
$uplinkTeamingPolicy = New-Object VMware.Vim.VmwareUplinkPortTeamingPolicy
# Set load balancing policy to IP hash
$uplinkTeamingPolicy.policy = New-Object VMware.Vim.StringPolicy
$uplinkTeamingPolicy.policy.inherited = $false
$uplinkTeamingPolicy.policy.value = “loadbalance_ip”
# Configure uplinks. If an uplink is not specified, it is placed into the ‘Unused Uplinks’ section.
$uplinkTeamingPolicy.uplinkPortOrder = New-Object VMware.Vim.VMwareUplinkPortOrderPolicy
$uplinkTeamingPolicy.uplinkPortOrder.inherited = $false
$uplinkTeamingPolicy.uplinkPortOrder.activeUplinkPort = New-Object System.String[] (2) # (#) designates the number of uplinks you will be specifying.
$uplinkTeamingPolicy.uplinkPortOrder.activeUplinkPort[0] = “dvUplink1″
$uplinkTeamingPolicy.uplinkPortOrder.activeUplinkPort[1] = “dvUplink2″
# Set notify switches to true
$uplinkTeamingPolicy.notifySwitches = New-Object VMware.Vim.BoolPolicy
$uplinkTeamingPolicy.notifySwitches.inherited = $false
$uplinkTeamingPolicy.notifySwitches.value = $true
# Set to failback to true
$uplinkTeamingPolicy.rollingOrder = New-Object VMware.Vim.BoolPolicy
$uplinkTeamingPolicy.rollingOrder.inherited = $false
$uplinkTeamingPolicy.rollingOrder.value = $true
# Set network failover detection to “link status only”
$uplinkTeamingPolicy.failureCriteria = New-Object VMware.Vim.DVSFailureCriteria
$uplinkTeamingPolicy.failureCriteria.inherited = $false
$uplinkTeamingPolicy.failureCriteria.checkBeacon = New-Object VMware.Vim.BoolPolicy
$uplinkTeamingPolicy.failureCriteria.checkBeacon.inherited = $false
$uplinkTeamingPolicy.failureCriteria.checkBeacon.value = $false
$spec.DefaultPortConfig.UplinkTeamingPolicy = $uplinkTeamingPolicy
$vds.ExtensionData.ReconfigureDvs_Task($spec)

and here is simplified version

$vDSName = “XXX”  ## << dvSwitch name

$vds = Get-VDSwitch $vDSName

$spec = New-Object VMware.Vim.DVSConfigSpec

$spec.configVersion = $vds.ExtensionData.Config.ConfigVersion

$spec.defaultPortConfig = New-Object VMware.Vim.VMwareDVSPortSetting

$uplinkTeamingPolicy =  New-Object VMware.Vim.VmwareUplinkPortTeamingPolicy

# Set load balancing policy to IP hash

$uplinkTeamingPolicy.policy = New-Object VMware.Vim.StringPolicy

$uplinkTeamingPolicy.policy.inherited = $false

$uplinkTeamingPolicy.policy.value = “loadbalance_ip”   ## << Teaming Policy Type

$spec.DefaultPortConfig.UplinkTeamingPolicy = $uplinkTeamingPolicy

$vds.ExtensionData.ReconfigureDvs_Task($spec)

 

Network Port list of vSphere 5.5 Components

Year by year vSphere platform becomes more complex. It is pretty logical as Virtualization is de facto standard on modern datacenters and new enterprise capabilities are required by VMware users.

At the beginning of Vmware Server Virtualization there were just vCenter (Virtual Center, database and simple integration with active directory). Today vSphere management plane is composed from more software components integrated over network. So it becomes more complex ... 

Although using, consulting and architecting vSphere daily, sometimes I get lost in the network ports of vSphere components.

That's the reason I have created and will maintain following vSphere Component network ports table.

Component	L7 Protocol	L3 Protocol/Port
vCenter  Single Sign-On	https	tcp/7444
vSphere Web Client HTTPS port https://WebClient_host_FQDN_or_IP:9443	https	tcp/9443
vSphere Web Client HTTP port	http	tcp/9090
vCenter Inventory Service https://Inventory_Service_host_FQDN_or_IP:10443	https	tcp/10443
vCenter Inventory Service management port	unknown	tcp/10109
vCenter Inventory Service Linked Mode communication port	unknown	tcp/10111
vCenter SSO Lookup Service https://SSO_host_FQDN_or_IP:7444/lookupservice/sdk	https	tcp/7444
vCenter Server HTTPS port	https	tcp/443
vCenter Server HTTP port	http	tcp/80
vCenter Server Management Web Services HTTP	http	tcp/8080
vCenter Server Management Web Services HTTPS	https	tcp/8080
vCenter Server Web Service - Change Service Notification	https	tcp/60099
vCenter Server Appliance (VCSA) - VAMI management GUI https://VCSA_host_FQDN_or_IP:5480	https	tcp/5480

I'll add other components to the list as needed in the future ...

DELL Storage useful links

Shared storage is essential  and common component in today's era of modern virtualized datacenters. Sorry hyper-converged evangelists, that's how it is today :-) DELL has two very popular datacenter storage products EqualLogic and Compellent. Useful links for datacenter architects and/or administrators are listed below.
 

EqualLogic

• EqualLogic Compatibility Matrix
• EqualLogic Configuration Guide
• Rapid EqualLogic Configuration Portal
• EqualLogic Best Practices Whitepapers

Switch Configuration Guides for EqualLogic SANs provide step-by-step instructions for configuring Ethernet switches for use with EqualLogic PS Series storage using Dell Best Practices.

Another switch configuration guides are in "Rapid EqualLogic Configuration Portal by SIS"

Compellent

• Compellent Compatibility Matrix (this is DELL internal resource - ask your DELL representative)
• Compellent on Dell TechCenter