Upcoming Events

Where the Cloud Touches Down: Simplifying Data Center Infrastructure Management

Thursday, July 25, 2013
10:00 AM PT/1:00 PM ET

In most data centers, DCIM rests on a shaky foundation of manual record keeping and scattered documentation. OpManager replaces data center documentation with a single repository for data, QRCodes for asset tracking, accurate 3D mapping of asset locations, and a configuration management database (CMDB). In this webcast, sponsored by ManageEngine, you will see how a real-world datacenter mapping stored in racktables gets imported into OpManager, which then provides a 3D visualization of where assets actually are. You'll also see how the QR Code generator helps you make the link between real assets and the monitoring world, and how the layered CMDB provides a single point of view for all your configuration data.

Register Now!

A Network Computing Webinar:
SDN First Steps

Thursday, August 8, 2013
11:00 AM PT / 2:00 PM ET

This webinar will help attendees understand the overall concept of SDN and its benefits, describe the different conceptual approaches to SDN, and examine the various technologies, both proprietary and open source, that are emerging. It will also help users decide whether SDN makes sense in their environment, and outline the first steps IT can take for testing SDN technologies.

Register Now!

More Events »

Subscribe to Newsletter

  • Keep up with all of the latest news and analysis on the fast-moving IT industry with Network Computing newsletters.
Sign Up

How to Set Up Cisco Nexus Fabric Extender

In a data center that has deployed Cisco Nexus 5000 or 7000 switches, Cisco Nexus 2000 series fabric extenders (FEX) are commonly used for top of rack (ToR) connectivity. FEX units are priced attractively, as they serve a limited purpose and are not feature-rich switches. In fact, a FEX is not a switch, in that a FEX cannot switch traffic locally or be managed independently. A FEX functions only when connected to a Cisco Nexus 5K or 7K series. All traffic flowing into a FEX will be sent down to the parent 5K or 7K for forwarding, even if the destination is on the originating FEX. If you think of a FEX as a remote line card with no local switching capabilities, you've got the idea.

Having established that a FEX is not a switch, let's take a look at the process of installing a new FEX and bringing it online.

More Insights


More >>

White Papers

More >>


More >>

1. Rack the FEX. The official Cisco documentation demonstrates racking a FEX in a four-post rack, using the supplied rack ears and rail slides. I have successfully racked FEXs in open two-post racks using only the rack ears. If you choose to do this, be sure to mount the ears further back along the chassis for better weight balance. However you mount the FEX, provide enough clearance for the fiber uplink cables, as they stick out of the front of the chassis. Tight tolerances could prevent the rack door from shutting.

Do not power on the FEX until all cabling and uplink port provisioning on the uplinked 5K or 7K has been completed.

2. Install & cable the uplinks. When uplinking to Nexus 5Ks, one topology option is to dual-home the physical Fabric Extender. Dual-homing a FEX provides path redundancy, but cuts in half the total number of Fabric Extenders you can deploy. A Nexus 5K supports 24 total connected FEX devices, meaning that two 5Ks could support 48 total single-homed physical FEX. When dual-homing, only 24 total FEX are supported between the two 5Ks.

If you choose to single-home the FEX, you lose uplink path redundancy for single-attached hosts. If single-attached hosts are not a concern, a common scenario is to deploy two FEX to ToR, each single-homed to a Nexus 5K or 7K. Multiattached servers then spread their uplinks across the two ToR FEX devices, and in that way enjoy uplink path redundancy. As of this writing, only Nexus 5Ks support dual-homing of FEX.

If you choose to dual-home the FEX, the uplink ports you select must match on both 5K-1 and 5K-2. For example, if you chose Eth1/1 on 5K-1, you must also use Eth1/1 on 5K-2. Another wise design choice is to spread your uplinks over multiple 5500 ASICs (the silicon inside the switch responsible for forwarding traffic). In the Nexus 5500s, ASICs are mapped to groups of 8 consecutive Ethernet ports on the front to the switch, 1-8, 9-16, etc. Therefore, spreading FEX uplinks over ports 1 and 9 is smarter than 1 and 2. If the ASIC servicing ports 1-8 fails, you won't lose both FEX uplinks.

Although options vary by model, FEX can be uplinked with a variety of media. When ordering, be aware of Cisco SKUs that bundle the FEX with uplink media, such as Fabric Extender Transceivers (FETs) or Twinax. Each option has a different price and distance constraint, so research this carefully to be sure you meet your installation’s requirements.

Another consideration is the number of uplinks to use. For example, a 2248TP FEX has 48 10/100/1000 access ports and four 10-Gbps uplink ports. If uplinking all four 10G ports, the oversubscription ratio of access to uplink ports is 1.2:1--very low. At the same time, you’ve used up four expensive 10G ports on the Nexus switch on the other end. Do your traffic patterns warrant using all four uplink ports, or can you get by with just two? Note that from a technical standpoint, the FEX will function correctly with only a single working uplink, but a sensible design uses at least two in a production environment.

The remainder of this example assumes a dual-homed FEX using all four uplinks, connected to a Nexus 5500 pair configured in a virtual port-channel domain.

3. Configure a virtual port-channel and add physical interfaces. Two of the FEX uplinks will homed to one 5K, and two to the other 5K. Then, all four of the FEX uplinks will be combined into a single virtual port channel. Each FEX is assigned a number from 100-199.

The physical interface requires two specific commands to tell the hosting 5K that the interface is servicing a FEX. The command "switchport mode fex-fabric" lets the Nexus switch know that the device on the other end of the link is fabric extender. Note that if you use FETs as uplink media, the switch can’t use these optical modules until this command is in place.

The command "fex associate " tells the Nexus switch which specific FEX is being uplinked to that port. The number selected must match for all uplink ports.

4. Apply the code below to both 5Ks. Once this step is complete, you can optionally prepare your console session to watch the messages that will scroll as the FEX comes up for the first time by typing "term mon".

interface Po101
 description UPLINK FEX-01
 vpc 101
 switchport mode fex-fabric
 fex associate 101
 no shut

interface Ethernet1/1
 description UPLINK FEX-01
 switchport mode fex-fabric
 fex associate 101
 channel-group 101
 no shut

interface Ethernet1/9
 description UPLINK FEX-01
 switchport mode fex-fabric
 fex associate 101
 channel-group 101
 no shut

Next Page: Powering Up

Page:  1 | 2  | Next Page »

Related Reading

Network Computing encourages readers to engage in spirited, healthy debate, including taking us to task. However, Network Computing moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing/SPAM. Network Computing further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | Please read our commenting policy.
Vendor Comparisons
Network Computing’s Vendor Comparisons provide extensive details on products and services, including downloadable feature matrices. Our categories include:

Research and Reports

Network Computing: April 2013

TechWeb Careers