DELL Force10 : DCB configuration - design decision justification and configuration

Introduction to DCB

Datacenter bridging (DCB) is group of protocols for modern QoS mechanism on Ethernet networks. There are four key DCB protocols described with more details here. In this blog post I'll show you how to configure DCB ETS, PFC and DCBX on Force10 S4810.

ETS (Enhanced Transmission Selection) is bandwidth management allowing reservations of link bandwidth resources when link is congested. DCB QoS is based on 802.1p CoS (Class of Service) which can handle up to 8 class of services (aka priority levels). Any QoS is always done via dedicated queues for different class of services and I/O scheduler which understand configured priorities.

S4810 has 4 queues and 802.1p CoS are by default mapped as outputted bellow …

DCSWCORE-A#show qos dot1p-queue-mapping
Dot1p Priority : 0  1  2  3  4  5  6  7
         Queue : 0  0  0  1  2  3  3  3

Command service-class dot1p-mapping can reconfigure mapping but let's use default mapping for our example. Queue CoS mapping:

• To Queue 0 are mapped CoS'es 0,1,2
• To Queue 1 is mapped CoS 3
• To Queue 2 is mapped CoS 4
• To Queue 3 are mapped CoS'es 5,6,7

PFC (Priority Flow Control) is nothing else then classic Ethernet flow control protocol but just in one specific 802.1p CoS. Force10 S4810 support PFC on two queues.

Now, let's define our design requirements and constraints for our specific design decision.

Design decision justification

R1: 4Gb guarantee for iSCSI traffic on each 10Gb converged link is required.
R2: Lost-less ethernet is required for iSCSI traffic
R3: 1Gb guarantee for Hypervisor Management network on each 10Gb converged link is required.
R4: 2Gb guarantee for Hypervisor Live Migration network on each 10Gb converged link is required.
R5: 3Gb guarantee for production networks on each 10Gb converged link is required.

C1: We have 10Gb links to edge devices (servers and storage)
C2: We have only four switch queues for DCB on DELL Force10 S4810
C3: We have DCB capable iSCSI storage DELL EqualLogic

A1: No other storage protocol then iSCSI is required
A2: No other network traffic type requires QoS
A3: We have iSCSI traffic in 802.1p CoS 4

Let's design best DCB Mapping based on requirements, constraints and assumptions above. Following priority groups reflects all requirements and constraints.

• PG0 - Hypervisor management; 10% reservation; lossy ethernet;  CoS 0,1,2 -> Switch Queue 0
• PG1 - Hypervisor live migrations; 20% reservation; lossy ethernet;  CoS 3 -> Switch Queue 1
• PG2 - iSCSI; 40% reservation; loss-less ethernet;  CoS 4 -> Switch Queue 2
• PG3 - Production; 30% reservation; lossy ethernet;  CoS 5,6,7 -> Switch Queue 3

Below is Force10 configuration snippet of DCB mapping to 802.1p CoS'es.

dcb-map converged
  priority-group 0 bandwidth 10 pfc off
  priority-group 1 bandwidth 20 pfc off
  priority-group 2 bandwidth 40 pfc on
  priority-group 3 bandwidth 30 pfc off
  priority-pgid 0 0 0 1 2 3 3 3

DCB map has to be configured on particular Force10 switch port. One particular switch port configuration snippet is below.

interface TenGigabitEthernet 0/6
 no ip address
 mtu 12000
 switchport
 spanning-tree rstp edge-port
 dcb-map converged
!
 protocol lldp
  dcbx port-role auto-downstream
 no shutdown

Following technologies are configured on switch port Te 0/6 by configuration snippet above.

• DCB ETS and PFC defined in dcb-map converged
• LLDP  protocol streaming down DCB information configured in the network
• MTU 12000 (Force10 maximum) because Jumbo Frames are beneficial for iSCSI. iSCSI Jumbo Frames require payload 9000 bytes plus some Ethernet and TCP/IP protocol overhead. MTU 9216 woudl be enough but why not set maximal MTU in the core network? Performance overhead is negligible and we are ready for everything.
• Edge port configuration for faster port transition to forwarding state

Advertisement

Techno Tip : Get an instant safe access to your favorite windows applications/data from anywhere on any device with innovative cloud solutions by CloudDesktopOnline. For hosted SharePoint and other hosted software products visit Apps4Rent.com today !

Force10 switch port iSCSI configuration

Here is snippet of Force10 switch port configuration of port facing storage front-end port or host NIC port dedicated just for iSCSI. In other words this is non-DCB switch port configuration.

interface TenGigabitEthernet 0/12
  no ip address
  mtu 12000
  switchport
  flowcontrol rx on tx off
  spanning-tree rstp edge-port
  spanning-tree rstp edge-port bpduguard shutdown-on-violation
  storm-control broadcast 100 in
  storm-control unknown-unicast 50 in
  storm-control multicast 50 in
  no shutdown

MTU is set to 12000 which is the supported maximum and allowing to use Jumbo Frames. I know 9216 is enough MTU for iSCSI however modern switches is capable for larger MTU without performance overhead so why not use it?

Flow-control is enabled for receive traffic so switch can sen PAUSE frame to initiator or target to wait a while in case switch port buffers are full. Transience flow-control is not enabled because switch port buffers are not deep enough to help.

Switch port is set to edge-port to eliminate spanning-tree algorithm because connected devices are edge devices - server (iSCSI initiator) or storage (iSCSI target).

[Updated at 2015-01-19 based on Martin's comment]
Bpduguard can be optionally enabled on switch port as prevention against cabling mistake (also known as an human factor) because iSCSI storage should not send any BPDU. 

[/Updated]

Storm control of broadcasts, multicasts and unknown unicasts is enabled to eliminate unwanted storms. Please note that unicast storm may not be enabled because iSCSI generates lot of traffic which can be possible identified as unicast storm.

That's it for iSCSSI network configuration of dedicated ports. When you use shared ports for iSCSI and LAN traffic you should use DCB. DCB configuration is similar to configuration above but replacing flow-control with priority-flow-control (PFC) and leveraging link QoS technology (ETS) based on 802.1p CoS. You can check DCB design justification and sample configuration for iSCSI here, 

And as always, any comment is really appreciated. 

Data Center Bridging

DCB 4 key protocols:

•  Priority-based Flow Control (PFC): IEEE 802.1Qbb
•  Enhanced Transmission Selection (ETS): IEEE 802.1Qaz
•  Congestion Notification (CN or QCN): IEEE 802.1Qau
•  Data Center Bridging Capabilities Exchange Protocol (DCBx)

PFC - provides a link level flow control mechanism that can be controlled independently for each frame priority. The goal of this mechanism is to ensure zero loss under congestion in DCB networks.PFC is independent traffic priority pausing and enablement of lossless packet buffers/queuing for particular 802.1p CoS.

ETS - provides a common management framework for assignment of bandwidth to frame priorities. Bandwidth can be dynamic based on congestion and relative ratios between defined flows. ETS provides minimum, guaranteed bandwidth allocation per traffic class/priority group during congestion and permits additional bandwidth allocation during non-congestion.

CN - provides end to end congestion management for protocols that are capable of transmission rate limiting to avoid frame loss. It is expected to benefit protocols such as TCP that do have native congestion management as it reacts to congestion in a more timely manner. Excellent blog post about CN is here.

DCBX - a discovery and capability exchange protocol that is used for conveying capabilities and configuration of the above features between neighbors to ensure consistent configuration across the network. Performs discovery, configuration, and mismatch resolution using Link Layer Discovery Protocol (IEEE 802.1AB - LLDP).

DCBX can be leveraged for many applications.
One DCBX application example is iSCSI application priority - Support for the iSCSI protocol in the application priority DCBX Type Length Value (TLV). Advertises the priority value (IEEE 802.1p CoS, PCP field in VLAN tag) for iSCSI protocol. End devices identify and tag Ethernet frames containing iSCSI data with this priority value.

Unified Network, DCB and iSCSI challenges

iSCSI SAN is Storage Area Network. Storage need lost less fabric. If, for any reason, unified fabric need to be used then quality of ethernet/IP network  is crucial for problem less storage operation.

For example DELL EqualLogic supports and leverage DCB (PFC, ETS and DCBX).
iSCSI-TLV is a part of DCBX. However the DCB protocol primitives must be supported end to end so if one member of the chain doesn't support it than it is useless.

How DCB makes iSCSI better is deeply explained here. 

So think twice if you really want converged network (aka unified fabric) or dedicated iSCSI network is better option for you.