The Path to CCIE Data Center

It’s official!


If I can do this, so can you!  Some of you may be just starting out, some of you may be on the homestretch with a lab right around the corner.  Either way, this post may have some interest to you.  I’d like to share my story, how I prepared, what study methods worked best for me, how I picked myself back up after defeat, and what I did to prepare once more for ultimate victory. (more…)


Configuring WWxN Pools in UCS

In UCS we can create these things called WWxN pools.  The “x” in WWxN can stand for either “N” node, or “P” port.  This is essentially a pool that you can use for all of your WWN assignments without having to create separate WWNN and WWPN pools.  For people who don’t necessarily care about a schema to signify the difference between WWNN and WWPN, maybe the streamlined WWxN pool is for you. (more…)

Configuring iSLB for CCIE DC

I’ll be going through iSLB, explaining what it is, and showing how to configure it.  A full template is at the bottom of this post.

Part 1 of the series, “Configuring iSCSI for CCIE DC” can be found here.

What is iSLB?

iSLB is iSCSI Server Load-Balancing. It is iSCSI, so don’t confuse it as some other protocol, think of it as iSCSIv2. iSLB introduces a few new features to iSCSI:

– Load-balancing between MDS’s (or ports on the same MDS)
– Cisco Fabric Services (CFS) Distribution
– iSLB Initiators (with Automatic Zone creation)


iSLB uses VRRP between two MDS switches for high availability and load-balancing. With VRRP you have a master and backup virtual gateway. Typically all traffic is sent to the master active gateway. So how does load-balancing work? A pair of MDSs will run CFS to keep track of an iSLB VRRP table. This table records the current load for each Initiator-to-MDS pair. When an initiator request comes into the VRRP master switch, the table is checked to see the current load on each MDS. The master will take the initiator and create a session if it’s load is lower than the backup switch. If it’s current load is higher, the master switch sends an ICMP redirect back to the initiator and a new session is built to the direct IP of the backup MDS switch. The default weight (load) for each initiator is 1000. This, of course, can be changed to influence path selection.

Although not visible initially, the master of VRRP starts automatically with more load since it has more responsibility. This means that the first session is always going to be redirected and load-balanced to the backup MDS switch. All sessions afterwards will be load-balanced based on load reported in the table.

As an example, say we have 3 initiators. Initiator 1 has a default metric of 1000, Initiator 2 has a configured metric of 900, and Initiator 3 has a default metric of 1000.


Configuring iSCSI for CCIE DC

First thing, credit where due: iSCSI Configuration Guide (Please read first!)

Peter Revill has some great posts on iSCSI (Thanks, Peter!)

Ben King has a good post on setting up iSCSI interfaces for ESXi in UCS (Thanks, Ben!)

For those looking for a quick reference, template is at the bottom of this post.

What is iSCSI?

iSCSI (Internet Small Computer Systems Interface) is a SCSI transport mechanism over IP. You take the SCSI payload (Reads/Writes), encapsulate it in TCP and send it over IP. This is a completely separate stack than FC, and does not rely on FC at all. You can, however, compare it to the same way that FCIP transports SCSI over IP. The differences can be seen in the packet:

FCIP is:

| SCSI | FC | FCIP | TCP | IP | Eth |

iSCSI is:

| SCSI | iSCSI | TCP | IP | Eth |

iSCSI works like any storage protocol, you have initiators and targets. When comparing to Fibre Channel, note these differences: (more…)

Multihop FCoE to Server in vPC + NPV/NPIV

If you’re like me and had a difficult time at first understanding how to present separate VSAN fabrics to a server in a single vPC, I hope this post helps you out. I’ll be covering this exact scenario, as well as multihop FCoE with Nexus 5K switches in two topologies – Without NPV/NPIV and with NPV/NPIV.

You can run ethernet and FCoE from a server in a vPC connected to two switches. The ethernet portion of the CNA will look like a port-channel, and the fibre channel portion will look like separate connection to separate fabrics. In this topology, we have 1 northbound storage device, 2 north switches, 2 south switches, and an server connected with CNAs. Our end goal is to present the left-side A fabric of our storage device to the server on CNA1 via VSAN 101, and present the right-side B fabric of our storage device to the server on CNA2 via VSAN 102. The server should be in a vPC to both south switches, using VLAN 10 for Data, and separate VSANs for each CNA. All switches should participate in the fabric (no NPV or NPIV).

 Our topology looks like this:



Fibre Channel over IP (FCIP) for CCIE DC

Fibre Channel over IP (FCIP) is a tunneling protocol used to connect FC networks across IP networks, such as a WAN. It uses TCP with the DF bit set. Being that this is IP storage, it is only supported on the MDS platform. The basic configuration is straight forward, but be aware that there are lots of configurable tweaks.  In this blog post I’ll be going through the configuration of several FCIP topologies, feel free to follow along.  At the end I’ll post a quick template.

Reference (This document is quite excellent):

Below is the topology we’re looking at.  We have a server in Data Center 1 that needs to attach to JBOD storage in Data Center 2 over the IP network.


To accomplish this, we’ll create an FCIP tunnel between MDS1 in Data Center 1 and MDS2 in Data Center 2. (more…)

FC Security for CCIE DC – Fabric Binding

Fabric binding ensures that switches configured in the fabric binding database are permitted to connect to the switch. If a switch tries to join the fabric, and that switch is not in the fabric binding database, access is denied.  Fabric binding is configured on a per-VSAN basis.

From Cisco, “This feature helps prevent unauthorized switches from joining the fabric or disrupting current fabric operations. It uses the Exchange Fabric Membership Data (EFMD) protocol to ensure that the list of authorized switches is identical in all switches in the fabric.”

This is very similar to Port Security, except Fabric Binding is for switches only (not devices). Switches bind to the fabric instead of interfaces like in Port Security. Additionally, Fabric Binding is manually configured on each switch, it cannot be distributed through CFS.

More information can be found here:

Steps involved:

1. Enable the fabric binding feature
2. Configure a list of sWWNs and their corresponding domain IDs for devices permitted in the fabric
3. Active the fabric binding database
4. Copy the fabric binding database to the fabric binding config database


FC Security for CCIE DC – FC Port Security

Fibre Channel port security prevents unauthorized Fibre Channel devices and switches from logging into the fabric. This protects the fabric from accidents, malicious intent or attacks such as WWN identity spoofing. It’s configured on a per-VSAN basis.  

Everything covered here can be found in this configuration guide:

You have a few options to choose from when configuring Port Security:

1. Configure with auto-learning and CFS distribution
2. Configure with auto-learning without CFS distribution
3. Configure with manual database

The first method is definitely most practical, as you can configure once, learn the current environment, and use Cisco Fabric Services (CFS) to distribute throughout the fabric. I’ll be following this method in this blog post, feel free to follow along.  Also added a quick template at the bottom.


FC Security for CCIE DC – FC-SP / DHCHAP

Fibre Channel Security Protocol (FC-SP) provides the capabilities for Diffie-Hellman Challenge Handshake Authentication Protocol (DHCHAP) to authenticate switches and/or hosts attempting to enter the fabric. The terms FC-SP and DHCHAP are used interchangeably. Unlike most FC feature, DHCHAP is not configured on a per-VSAN basis.

All things in this post can be found in the configuration guide:

Steps involved to configure FC-SP:

2. (Optional) Configure the hash algorithm and Diffie-Hellamn groups
3. Configure the DHCHAP password for the local switch
4. Configure the DHCHAP password for the remote switches/devices in the fabric
5. Configure and enable DHCHAP on interfaces
_a. Modes
_b. Reauthentication
6. Verify