Force10 Operating System 9.10 changes maximum MTU size

Force10 operating system (aka FTOS, DNOS) always had the maximal configurable MTU size per port 12000 bytes. I have just been informed by former colleague of mine that it is not the case since FTOS 9.10 and above. Since FTOS 9.10 the maximum MTU size per switch port is 9216. If you used MTU 12000 then after upgrade to firmware 9.10 the MTU should be adjusted automatically. But I have been told that it is automatically adjusted to standard MTU 1500 therefore if you use Jumbo Frames (9000 bytes payload) it is necessary to change configuration before upgrade from 12000 to 9216.

Disclaimer: I had no chance to test it so I don't guarantee all information on this post are correct. 

UPDATE: Please read comments below this article for further information and great Martin's explanation of real MTU behavior. Thanks Martin and Kaloyan for your comments.

Martin's comment:

MTU 12000 in configuration was not reflecting real hardware MTU of underlaying chipset, after upgrade to 9.10 it's just adjusted to reflect real hardware MTU. Tested on S4048 9.10(0.1). When you boot into 9.10 you can see log messages saying that configuration is adjusted to reflect real maximum hardware MTU.
Also in configuration

 ethswitch1(conf-if-te-1/47)#mtu ?  
 <594-12000> Interface MTU (default = 1554, hardware supported maximum = 9216)  
 ethswitch1(conf-if-te-1/47)#mtu   

                                 

iSCSI and Ethernet

Each manufacturer of Ethernet switch may implement features unique to their specific model. Below are some general tips to look for when implementing an iSCSI network infrastructure. Each tip may or may not apply to a specific installation. Be aware that this is list is inspired by DELL Compellent iSCSI bets practices and it is not an all-inclusive list.

• Bi-Directional Flow Control enabled for all Switch Ports that carry iSCSI traffic, including any inter switch links.
• Separate networks or VLANs from data.
• Separate iSCSI traffic multi-path traffic also.
• Unicast storm control disabled on every switch that handles iSCSI traffic.
• Multicast disabled at the switch level for any iSCSI VLANs - Multicast storm control enabled (if available) when multicast cannot disabled.
• Broadcast disabled at the switch level for any iSCSI VLANs - Broadcast storm control enabled (if available) when broadcast cannot disabled.
• Routing disabled between regular network and iSCSI VLANs - Use extreme caution if routing any storage traffic, performance of the network can be severely affected. This should only be done under controlled and monitored conditions.
• Disable Spanning Tree (STP or RSTP) on ports which connect directly to end nodes (the server or Dell Compellent controller's iSCSI ports.) You can do it by enabling PortFast or EdgePort option  on these ports so that they are configured as edge ports.
• Ensure that any switches used for iSCSI are of a non-blocking design.
• Hard set for all switch ports and server ports for Gigabit Full Duplex if applicable.
• When deciding which switches to use, remember that you are running SCSI traffic over it. Be sure to use a quality managed enterprise class networking equipment. It is not recommended to use SBHO (small business/home office) class equipment outside of lab/test environments.

Do you want configuration examples for DELL PowerConnect and DELL Force10 switches? Leave a comment with particular switch model and firmware version and I'll try my best to prepare it for you.

CISCO UDLD alternative on Force10

I've been asked by one DELL System Engineer if we support CISCO's UDLD feature because it was required in some RFI. Well, DELL Force10 Operating System have similar feature solving the same problem and it is called FEFD.

Here is the explanation from FTOS 9.4 Configuration Guide ...

FEFD (Far-end failure detection) is supported on the Force10 S4810 platform. FEFD is a protocol that senses remote data link errors in a network. FEFD responds by sending a unidirectional report that triggers an echoed response after a specified time interval. You can enable FEFD globally or locally on an interface basis. Disabling the global FEFD configuration does not disable the interface configuration.

Figure caption: Configuring Far-End Failure Detection

The report consists of several packets in SNAP format that are sent to the nearest known MAC address. In the event of a far-end failure, the device stops receiving frames and, after the specified time interval, assumes that the far-end is not available. The connecting line protocol is brought down so that upper layer protocols can detect the neighbor unavailability faster.

Update 2015-05-20:
If I understand it correctly CISCO's UDLD main purpose is to detect potential uni-directional links and mitigate the risk of loop in the network because STP cannot help in this scenario. Force10 has another feature to prevent a loop in such situation - STP loop guard.

The STP loop guard feature provides protection against Layer 2 forwarding loops (STP loops) caused by a hardware failure, such as a cable failure or an interface fault. When a cable or interface fails, a participating STP link may become unidirectional (STP requires links to be bidirectional) and an STP port does not receive BPDUs. When an STP blocking port does not receive BPDUs, it transitions to a Forwarding state. This condition can create a loop in the network.

DELL Force10 : Initial switch configuration

[ Previous | DELL Force10 : Series Introduction ]

---

I assume you have serial console access to the switch unit to perform initial switch configuration. I guess it will not impressed you that to switch from read mode to configuration mode you  have to use command

conf

... before continue I would like to recap some important basic FTOS commands we will use later in this blog post. If you want to exit from configuration mode or even from deeper configuration hierarchy you can do it with one or several

exit 

commands which will jump to the upper level of configuration hierarchy and eventually exit conf mode. However the easiest way to leave configuration mode is to use command

end

which will exit configuration mode immediately.

The last but very important and very often used command is

write mem

which will write your running switch configuration to the flash and therefore configuration will survive the switch reload. You can do the same with more general command

copy running-config startup-config

If you want to display running configuration you can use command

show running-config

Whole configuration can be pretty long, so if you are interested only on some part of running configuration you can use following commands

show running-config interface managementethernet 0/0
show running-config interface gigabitethernet 0/2
show running-config spanning-tree rstp
show running-config boot

As you can see FTOS command line interface (cli) is very similar to CISCO.

Ok, so after basics let's start with initial configuration. Switch configuration usually begins with host name configuration. It is generally good practice to use unique host names because you know on which system you are logged in.

hostname f10-s60

As a next step I usually configure management IP settings and enable remote access. You have to decide if you will use in-band management leveraging normal IP settings usually configured on dedicated VLAN interface just for system management or you will leverage dedicated out-of-band management port. In example below you can see

• out-of-band management port for system management IP settings
• how to create admin user
• how to enable ssh to allow remote system management

interface ManagementEthernet 0/0
  ip address 192.168.42.101/24
  no shutdown
exit
management route 0.0.0.0/0 192.168.42.1

username admin password YourPassword privilege 15

ip ssh server enable

Now you have to decide if you want to enforce login for users connected via local console. By default there is no login required which can by security risk especially in environments without strict physical security rules. Below is configuration which enforce local login credentials when using serial console.

 aaa authentication login default local

At this point I would like to note that Force10 switch has all capabilities and features disabled in default factory configuration. That's the reason why for example each switch interface must be explicitly enabled before usage because all interfaces are in shutdown state by default.

Before you enable any switch interface it is good practice to enable spanning tree protocol as security mechanism against potential loops in the network. Once again, spanning tree feature is not enabled by default so you have to do it explicitly. Force10 FTOS has implemented all standard and even some non-standard (CISCO proprietary) spanning tree protocols like PVSTP+. On the latest FTOS version following spanning tree protocols are supported:

• STP (Spanning Tree Protocol)
• RSTP (Rapid Spanning Tree Protocol)
• MSTP (Multiple Spanning Tree Protocol)
• PVSTP+ (Per-VLAN Spanning Tree Plus)

Bellow is configuration example which enables standard rapid spanning tree protocol (aka RSTP) ...

protocol spanning-tree rstp
  no disable 

Another decision you have to do before implementation is the location from where do you want to boot your switch operating system. On some Force10 models (for example on S60) is default primary boot location TFTP server  ...

boot system stack-unit 0 primary tftp://192.168.128.1/FTOS-SC-8.3.3.7.bin
boot system stack-unit 0 secondary system: A:
boot system stack-unit 0 default system: B:
boot system gateway 192.168.128.1

You can see that primary boot location is TFTP server. If you don't have tens or hundreds of switches you usually don't want to load FTOS remotely from TFTP server but from internal flash in the switch. Although default switch configuration would work because if TFTP server boot fails switch boot sequence continue with secondary location  but it's better to configure the switch boot sequence explicitly base on your requirements. Below is typical boot sequence configuration.

boot system stack-unit 0 primary system: A:boot system stack-unit 0 secondary system: B:boot system stack-unit 0 default system: A: no boot system gateway

Next thing you should check is what FTOS version do you have. Below is the command how you can check it ...

f10-s60#show version
Dell Force10 Networks Real Time Operating System Software
Dell Force10 Operating System Version: 1.0
Dell Force10 Application Software Version: 8.3.3.7
Copyright (c) 1999-2011 by Dell Inc.
Build Time: Sat Nov 26 01:23:50 2011
Build Path: /sites/sjc/work/build/buildSpaces/build20/E8-3-3/SW/SRC
f10-s60 uptime is 4 minute(s)
System image file is "system://A"
System Type: S60
Control Processor: Freescale MPC8536E with 2147483648 bytes of memory.
128M bytes of boot flash memory.
  1 48-port E/FE/GE (SC)
 48 GigabitEthernet/IEEE 802.3 interface(s)
  2 Ten GigabitEthernet/IEEE 802.3 interface(s)

You can see FTOS version 8.3.3.7 which is not the latest one as the latest FTOS version at the time of writing this article is 8.3.3.9 and boot loader 1.0.0.5. It is generally good practice to upgrade FTOS to the latest version before performing verification test and going into production. For the latest version you have to go to http://www.force10networks.com and sign in. If you don't have Force10 account you can register there. Please note that each Force10 switch model use different FTOS versions. So there can be FTOS 9.4.x for model S4810 and 8.3.x for S60.

Now I'll show you how to do FTOS and boot loader upgrade.
FTOS should be upgraded first and Boot Loader later ...

upgrade system tftp: A:
upgrade system stack-unit all A: (applicable only if you have stack configured)
upgrade boot ftp: (applicable only if  new bootloader compatible with FTOS code exists)
reload

You can check current FTOS version

show version

and if you want to know what FTOS version do you have on which boot bank you can 

show boot system stack-unit 0

By the way, have I told you there are two boot banks? Boot bank A: and boot bank B:so you can choose primary and secondary boot location. We have already covered boot configuration but here it is again ...

conf

  boot system stack-unit 0 primary system: A:
  boot system stack-unit 0 secondary system: B:

FTOS is loaded by boot loader and current Boot Loader can be displayed by command below

show system stack-unit 0

Hope this post is helpful for IT community. In case you have any question, suggestion or idea on improvements please share your thoughts in in the comments.

Stay tuned and wait for next article ...

---

[ Next | DELL Force10 : Interface configuration and VLANs ]

DELL Force10 : Series Introduction

I have just decided to write dedicated blog post series about DELL Force10 networking. Why?

Who knows me in person is most probably aware that my primary professional focus is on VMware vSphere infrastructure and datacenter enterprise hardware. Sometimes I have discussion with infrastructure experts, managers and other IT folks what is the most important/complex/critical/expensive vSphere component. vSphere component is meant by compute (servers), storage and networking.  I thing it is needless discussion because all components are important and have to be integrated into single integrated system fulfilling all requirements, dealing with known constraints and mitigating all potential risks. Such integrated infrastructures are very often called POD which stands, as far as I know, for Performance Optimized Datacenter. These integrated systems are, from my point of view, new datacenter computers having dedicated but distributed computing, storage and networking components. I would prefer to call such equipment as "Optimized Infrastructure Block" or "Datacenter Computer" because it is not only about performance but also about reliability, capacity, availability, manageability, recoverability and security. We call these attributes infrastructure qualities and whole infrastructure block inherits qualities from sub-components. Older IT folks often compare this concept with main frame architectures however nowadays we usually use commodity x86 hardware "a little bit" optimized for enterprise workloads. 

By the way that's one of the reason I like current DELL datacenter product portfolio because DELL has everything I need to build POD - server, storage systems and now also networking so I'm able to design single vendor infrastructure block with unified support, warranty, etc. Maybe someone don't know but DELL acquired EqualLogic and Compellent storage vendors some time ago, but more importantly for this blog post, DELL also acquired well known (at least in US) datacenter networking producer Force10. For official acquisition details look here.

But back to the networking. Everybody would probably agree that networking is very important part of vSphere infrastructure because of several reasons. It provides interconnect between clustered components - think about vSphere networks like Management, vMotion, Fault Tolerance, VSAN, etc. It also routes network traffic to the outside world. And sometimes it even provides storage fabrics (iSCSI, FCoE, NFS). That's actually the reason why I'm going to write this series of blog posts about DELL Force10 networking - because of networking importance. However I don't want to write about  legacy networking but modern networking approach for next generation virtualized and software defined datacenters.

Modern physical networking is not only about hardware (burned intelligence in ASICs with high bandwidth, fast and low latency interfaces) but also in software. The main software sits inside DELL Force10 switches. It is switch firmware called FTOS - Force10 Operating System (see. for more general information about FTOS look here).  However, today it is not only about switch embedded firmwares but also about whole software ecosystem - managements, centralized control planes, virtual distributed switches, network overlays, etc.
 
In future articles I would like to deep dive into FTOS features, configuration examples and virtualization related integrations.

Next, actually first technical article in this series will be about about typical initial configuration of Force10 switch. I know it is not rocket science but we have to know basics before taking off. In the future I would like to write about more complex designs, capabilities and configurations like

• Multi-Chassis Link Aggregation (aka MC-LAG). In Force10 terminology we call it VLT - Virtual Link Trunking.
• Virtual Routing and Forwarding (aka VRF). Some S-series Force10 models with FTOS 9.4 support VRF-lite.
• some Software Define Networking (aka SDN) capabilities like python/perl scripting inside the switch, REST API, VXLAN hardware VTEP, Integration with VMware Distributed Virtual Switch, Integration with VMware NSX, OpenFlow, etc.

If you prefer some topics please let me know in comments and I'll try to prioritize it. Otherwise I'll write future posts based on my preferences.   

So let's finish blog post series introduction and start some technical stuff and begin with deep dive into switch initial configuration! Just click next and below ...

---

[ Next | DELL Force10 : Initial switch configuration ]