Tag: fex

Placement of mezzanine adapters in full width blades

During a discussion recently with some other UCS-savvy folks, I realized that there may be some confusion in how to place mezzanine adapters in full width blades when you’re only using one adapter.

First, a quick review of how UCS pins mezzanine ports to uplinks.   I’ll skip the one uplink option, since all ports use the single uplink.

In a two-uplink configuration, all odd numbered slots use uplink #1, while all even numbered slots use uplink #2.  (Easy to remember – odds go to the odd uplink, evens go to the even uplnk).   Since a full-width blade occupies two horizontally adjacent slots, each full width blade will contain both an even and an odd numbered mezzanine slot.  Let’s look at the slot numbering – this diagram shows half width blades.

Consider the case of 4 full width blades, all with a single mezzanine card in the “left” (as viewed from the front) slot.   In a two-uplink configuration, only the first uplink would be used – since all odd numbered slots are pinned to uplink #1.   Uplink #2 would be completely unused.

In a four uplink configuration, the pinning is as follows:

Ports 1,5 -> Uplink 1

Ports 2,6 -> Uplink 2

Ports 3,7 -> Uplink 3

Ports 4,8 -> Uplink 4

Consider the same previous example.   In the four-uplink configuration, only uplinks 1 and 3 are used, as these are the uplinks pinned to the odd-numbered mezzanine slots.   Uplinks 2 and 4 would remain idle.

When using full width blades and single mezzanine cards, it’s important to consider the other blades in the chassis when selecting where to install the mezzanine card.   Individual configurations will vary, depending on the number of half and full width blades in use.

If using all full width blades with single mezzanine cards, two of the blades should have the mezzanine card in the left slot and two blades should have the mezzanine card in the right slot.   By properly selecting the mezzanine placement, such as first blade (slots 1 and 2) in the left (1) position, the second blade (slots 3 and 4) in the left (3) position, third blade (slots 5 and 6) in the right (6) position, and the fourth blade (slots 7 and 8) in the right (8) position, you achieve proper usage of all four uplinks.   The inverse is also functionally identical, populating mezzanine slots 2, 4, 5, 7.

For a two-uplink configuration, any combination of mezzanine layout is acceptable as long as two blades have mezzanine cards in the left slots, and two have mezzanine cards in the right slots.  If beginning with a two-uplink configuration, it would be wise to use the four-uplink layout to minimize future reconfiguration if expanding to four uplinks.

Chassis Discovery Policies in UCS

Cisco UCS provides a configurable “Chassis Discovery Policy” that affects how chassis links (called “Server Ports”) are activated when discovering a new chassis.  See this page on cisco.com.

After a recent discussion on Twitter, I decided to test out a few scenarios.

My configuration is a pair of UCS 6120 Fabric Interconnects, two UCS 5108 Chassis, with 2 links per IO Module.

Additionally, this particular lab is on a reasonably old version of UCSM code, 1.0(1f).    I’m not in a position to upgrade it at the moment – I can re-test at a later date with more current code.  I don’t expect the behavior to change, however.

I started with the default discovery policy of 1-link.   I enabled one link per fabric interconnect to a chassis, and checked the status of the ports.   The ports were in an “Up” Overall Status, which is proper – and means that the link is available for use.   I then enabled a second link per fabric interconnect to the same chassis.   The second activated link went into a “Link Up” state, with “Additional info” of “FEX not configured”.   This means that the physical link has been activated, but is not in use – the IO Module (FEX) is not using the link.   A re-acknowledgement of the chassis activates the second link.

I then changed the discovery policy to 4-links.   I enabled one link per fabric interconnect to a different chassis, and checked the status of the ports.  The ports went into the “Link Up” state, “FEX Not Configured” – in other words, the links are not use.   While we can detect that the chassis is present, no traffic will flow as the FEX has not yet been configured to pass traffic.   Furthermore, the chassis is an error state, in as much as the cabling doesn’t match the chassis discovery policy.

Reacknowledging the chassis activates the links, and removes the error condition, even without changing the discovery policy.

Finally, I tested the scenario of setting the chassis discovery policy to 2 links and activating only one link per Fabric Interconnect.   As expected, the link enters the “link-up”, “FEX not configured” state.   I then activated a second link per Fabric Interconnect.   After a brief period, both ports per Fabric Interconnect enter the “Up” status.

In short, setting the Chassis Discovery Policy determines how many links must be present per Fabric Interconnect to an IO Module before the ports are activated and the chassis is put into service.   If the policy is set at 1 link, and more than 1 link are activated, a simple re-acknowledgement of the chassis will activate the additional links.   If the policy is set to a higher number – and that number of links are not present – the chassis will not be activated unless a manual re-acknowledgement is done.

Frankly, I don’t see a lot of value in this feature – unless you’re adding a large number of chassis that will all have identical numbers of uplinks.  Even then, you’re saving yourself at most a few clicks of the mouse to re-ack the chassis that don’t match the policy.   Why not simply leave the policy at default and re-ack any chassis that has more than 1 link per IOM?   Presumably you’re going to do this before actually deploying any service profiles, so there’s no potential for disruption with the re-ack.

Thoughts or comments?