DELL Force10 : DCB configuration - design decision justification and configuration

Introduction to DCB

Datacenter bridging (DCB) is group of protocols for modern QoS mechanism on Ethernet networks. There are four key DCB protocols described with more details here. In this blog post I'll show you how to configure DCB ETS, PFC and DCBX on Force10 S4810.

ETS (Enhanced Transmission Selection) is bandwidth management allowing reservations of link bandwidth resources when link is congested. DCB QoS is based on 802.1p CoS (Class of Service) which can handle up to 8 class of services (aka priority levels). Any QoS is always done via dedicated queues for different class of services and I/O scheduler which understand configured priorities.

S4810 has 4 queues and 802.1p CoS are by default mapped as outputted bellow …

DCSWCORE-A#show qos dot1p-queue-mapping
Dot1p Priority : 0  1  2  3  4  5  6  7
         Queue : 0  0  0  1  2  3  3  3

Command service-class dot1p-mapping can reconfigure mapping but let's use default mapping for our example. Queue CoS mapping:

• To Queue 0 are mapped CoS'es 0,1,2
• To Queue 1 is mapped CoS 3
• To Queue 2 is mapped CoS 4
• To Queue 3 are mapped CoS'es 5,6,7

PFC (Priority Flow Control) is nothing else then classic Ethernet flow control protocol but just in one specific 802.1p CoS. Force10 S4810 support PFC on two queues.

Now, let's define our design requirements and constraints for our specific design decision.

Design decision justification

R1: 4Gb guarantee for iSCSI traffic on each 10Gb converged link is required.
R2: Lost-less ethernet is required for iSCSI traffic
R3: 1Gb guarantee for Hypervisor Management network on each 10Gb converged link is required.
R4: 2Gb guarantee for Hypervisor Live Migration network on each 10Gb converged link is required.
R5: 3Gb guarantee for production networks on each 10Gb converged link is required.

C1: We have 10Gb links to edge devices (servers and storage)
C2: We have only four switch queues for DCB on DELL Force10 S4810
C3: We have DCB capable iSCSI storage DELL EqualLogic

A1: No other storage protocol then iSCSI is required
A2: No other network traffic type requires QoS
A3: We have iSCSI traffic in 802.1p CoS 4

Let's design best DCB Mapping based on requirements, constraints and assumptions above. Following priority groups reflects all requirements and constraints.

• PG0 - Hypervisor management; 10% reservation; lossy ethernet;  CoS 0,1,2 -> Switch Queue 0
• PG1 - Hypervisor live migrations; 20% reservation; lossy ethernet;  CoS 3 -> Switch Queue 1
• PG2 - iSCSI; 40% reservation; loss-less ethernet;  CoS 4 -> Switch Queue 2
• PG3 - Production; 30% reservation; lossy ethernet;  CoS 5,6,7 -> Switch Queue 3

Below is Force10 configuration snippet of DCB mapping to 802.1p CoS'es.

dcb-map converged
  priority-group 0 bandwidth 10 pfc off
  priority-group 1 bandwidth 20 pfc off
  priority-group 2 bandwidth 40 pfc on
  priority-group 3 bandwidth 30 pfc off
  priority-pgid 0 0 0 1 2 3 3 3

DCB map has to be configured on particular Force10 switch port. One particular switch port configuration snippet is below.

interface TenGigabitEthernet 0/6
 no ip address
 mtu 12000
 switchport
 spanning-tree rstp edge-port
 dcb-map converged
!
 protocol lldp
  dcbx port-role auto-downstream
 no shutdown

Following technologies are configured on switch port Te 0/6 by configuration snippet above.

• DCB ETS and PFC defined in dcb-map converged
• LLDP  protocol streaming down DCB information configured in the network
• MTU 12000 (Force10 maximum) because Jumbo Frames are beneficial for iSCSI. iSCSI Jumbo Frames require payload 9000 bytes plus some Ethernet and TCP/IP protocol overhead. MTU 9216 woudl be enough but why not set maximal MTU in the core network? Performance overhead is negligible and we are ready for everything.
• Edge port configuration for faster port transition to forwarding state

Advertisement

Techno Tip : Get an instant safe access to your favorite windows applications/data from anywhere on any device with innovative cloud solutions by CloudDesktopOnline. For hosted SharePoint and other hosted software products visit Apps4Rent.com today !

Force10 switch port iSCSI configuration

Here is snippet of Force10 switch port configuration of port facing storage front-end port or host NIC port dedicated just for iSCSI. In other words this is non-DCB switch port configuration.

interface TenGigabitEthernet 0/12
  no ip address
  mtu 12000
  switchport
  flowcontrol rx on tx off
  spanning-tree rstp edge-port
  spanning-tree rstp edge-port bpduguard shutdown-on-violation
  storm-control broadcast 100 in
  storm-control unknown-unicast 50 in
  storm-control multicast 50 in
  no shutdown

MTU is set to 12000 which is the supported maximum and allowing to use Jumbo Frames. I know 9216 is enough MTU for iSCSI however modern switches is capable for larger MTU without performance overhead so why not use it?

Flow-control is enabled for receive traffic so switch can sen PAUSE frame to initiator or target to wait a while in case switch port buffers are full. Transience flow-control is not enabled because switch port buffers are not deep enough to help.

Switch port is set to edge-port to eliminate spanning-tree algorithm because connected devices are edge devices - server (iSCSI initiator) or storage (iSCSI target).

[Updated at 2015-01-19 based on Martin's comment]
Bpduguard can be optionally enabled on switch port as prevention against cabling mistake (also known as an human factor) because iSCSI storage should not send any BPDU. 

[/Updated]

Storm control of broadcasts, multicasts and unknown unicasts is enabled to eliminate unwanted storms. Please note that unicast storm may not be enabled because iSCSI generates lot of traffic which can be possible identified as unicast storm.

That's it for iSCSSI network configuration of dedicated ports. When you use shared ports for iSCSI and LAN traffic you should use DCB. DCB configuration is similar to configuration above but replacing flow-control with priority-flow-control (PFC) and leveraging link QoS technology (ETS) based on 802.1p CoS. You can check DCB design justification and sample configuration for iSCSI here, 

And as always, any comment is really appreciated. 

ISCSI Best Practices

General ISCSI Best Practices

• Separate VLAN for iSCSI traffic. 
• Two separate networks or VLANs for multipath iSCSI. 
• Two separate IP subnets for the separate networks or VLANs in multipath iSCSI. 
• Gigabit (or better) Full Duplex connectivity between storage targets (storage front-end ports) and all storage initiators (server ports) 
• Auto-Negotiate for all switches that will correctly negotiate Full Duplex 
• Full Duplex hard set for all iSCSI ports for switches that do not correctly negotiate 
• Bi-Directional Flow Control enabled for all Switch Ports that servers or controllers are using for iSCSI traffic. 
• Bi-Directional Flow Control enabled for all ports that handle iSCSI traffic. This includes all devices between two sites that are used for replication. 
• Unicast storm control disabled on every switch that handles iSCSI traffic. 
• Multicast disabled at the switch level for any iSCSI VLANs. 
• Broadcast disabled at the switch level for any iSCSI VLANs. 
• Routing disabled between the regular network and iSCSI VLANs. 
• Do not use Spanning Tree (STP or RSTP) on ports that connect directly to end nodes (the server or storage iSCSI ports.) If you must use it, enable the Cisco PortFast option or equivalent on these ports so that they are configured as edge ports. 
• Ensure that any switches used for iSCSI are of a non-blocking design. 
• When deciding which switches to use, remember that you are running SCSI traffic over it. Be sure to use a quality managed enterprise-class networking equipment. It is not recommended to use SBHO (small business/home office) class equipment outside of lab/test environments. 

For Jumbo Frame Support

• Some switches have limited buffer sizes and can only support Flow Control or Jumbo Frames, but not both at the same time. It is strongly recommended to choose Flow Control. 
• All devices connected through iSCSI need to support 9k jumbo frames. 
• All devices used to connect iSCSI devices need to support it. 
• This means every switch, router, WAN Accelerator, and any other network device that will handle iSCSI traffic needs to support 9k Jumbo Frames. 
• If it is not 100% positive that every device in the iSCSI network supports 9k Jumbo Frames, then NOT turn on Jumbo Frames. 
• Because devices on both sides (server and SAN) need Jumbo Frames enabled, change disable to enable Jumbo Frames is recommended during a maintenance window. If servers have it enabled first, the Storage System will not understand their packets. If Storage System enables it first, servers will not understand its packets.

VMware ESXi iSCSI tunning

• Disabling "TCP Delayed ACK" (esxcli iscsi adapter param set -A vmhba33 -k DelayedAck -v 0 - command not tested)
• Adjust iSCSI Login Timeout (esxcli iscsi adapter param set -A vmhba33 -k LoginTimeout -v 60)
• Disable large receive offload (LRO) (esxcli system settings advanced set -o /Net/TcpipDefLROEnabled 0 or esxcfg-advcfg -s 0 /Net/TcpipDefLROEnabled)
• Set up Jumbo Frames is configured end to end (esxcli network vswitch standard set -m 9000 -v vSwitch2 and  esxcli network ip interface set -m 9000 -i vmk1)
• Set up appropriate multi pathing based on iSCSI storage system
• FlowControl is enabled on ESXi by default. To display FlowControl settings use ethtool --show-pause vmnic0 or esxcli system module parameters list --module e1000 | grep "FlowControl"

If you know about some other best practice, tuning setting or recommendation don't hesitate to leave a comment below this blog post. 

 

Related documents:

[1] VMware. Best Practices For Running VMware vSphere On iSCSI. In: core.vmware.com, URL: https://core.vmware.com/resource/best-practices-running-vmware-vsphere-iscsi

iSCSI and Ethernet

Each manufacturer of Ethernet switch may implement features unique to their specific model. Below are some general tips to look for when implementing an iSCSI network infrastructure. Each tip may or may not apply to a specific installation. Be aware that this is list is inspired by DELL Compellent iSCSI bets practices and it is not an all-inclusive list.

• Bi-Directional Flow Control enabled for all Switch Ports that carry iSCSI traffic, including any inter switch links.
• Separate networks or VLANs from data.
• Separate iSCSI traffic multi-path traffic also.
• Unicast storm control disabled on every switch that handles iSCSI traffic.
• Multicast disabled at the switch level for any iSCSI VLANs - Multicast storm control enabled (if available) when multicast cannot disabled.
• Broadcast disabled at the switch level for any iSCSI VLANs - Broadcast storm control enabled (if available) when broadcast cannot disabled.
• Routing disabled between regular network and iSCSI VLANs - Use extreme caution if routing any storage traffic, performance of the network can be severely affected. This should only be done under controlled and monitored conditions.
• Disable Spanning Tree (STP or RSTP) on ports which connect directly to end nodes (the server or Dell Compellent controller's iSCSI ports.) You can do it by enabling PortFast or EdgePort option  on these ports so that they are configured as edge ports.
• Ensure that any switches used for iSCSI are of a non-blocking design.
• Hard set for all switch ports and server ports for Gigabit Full Duplex if applicable.
• When deciding which switches to use, remember that you are running SCSI traffic over it. Be sure to use a quality managed enterprise class networking equipment. It is not recommended to use SBHO (small business/home office) class equipment outside of lab/test environments.

Do you want configuration examples for DELL PowerConnect and DELL Force10 switches? Leave a comment with particular switch model and firmware version and I'll try my best to prepare it for you.

Unified Network, DCB and iSCSI challenges

iSCSI SAN is Storage Area Network. Storage need lost less fabric. If, for any reason, unified fabric need to be used then quality of ethernet/IP network  is crucial for problem less storage operation.

For example DELL EqualLogic supports and leverage DCB (PFC, ETS and DCBX).
iSCSI-TLV is a part of DCBX. However the DCB protocol primitives must be supported end to end so if one member of the chain doesn't support it than it is useless.

How DCB makes iSCSI better is deeply explained here. 

So think twice if you really want converged network (aka unified fabric) or dedicated iSCSI network is better option for you.

DELL PCT 6224 (EQL configuration)

1. Enable the RSTP on the switch
2. Enable “spanning-tree portfast” on all ports connected to the SAN
3. Enable Jumbo Frame support on ports connected to the SAN
4. Disable the “storm control” feature on ports connected to the SAN
5. When you have dedicated SAN network enable Flow Control on the switch

Configuration:

Switch(config)# spanning-tree mode rstp
Switch(config)# interface range ethernet 1/g1-1/g12
Switch(config-if)# spanning-tree portfast
Switch(config-if)# mtu 9216
Switch(config-if)# no storm-control unicast
Switch(config-if)# exit
Switch(config)# flowcontrol (only you have dedicated SAN)
Switch# copy running-config startup-config

NUTANIX

http://www.nutanix.com/

A “Multivendor Post” to help our mutual iSCSI customers using VMware

Very nice article explaining iSCSI in ESX environment
http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html

iSCSI on FreeBSD

FreeBSD iSCSI howto
http://www.cyberciti.biz/faq/freebsd-iscsi-initiator-howto/

FreeBSD network link agregation
http://www.freebsd.org/doc/en/books/handbook/network-aggregation.html