Chassis Discovery Policies in UCS

Cisco UCS provides a configurable “Chassis Discovery Policy” that affects how chassis links (called “Server Ports”) are activated when discovering a new chassis.  See this page on

After a recent discussion on Twitter, I decided to test out a few scenarios.

My configuration is a pair of UCS 6120 Fabric Interconnects, two UCS 5108 Chassis, with 2 links per IO Module.

Additionally, this particular lab is on a reasonably old version of UCSM code, 1.0(1f).    I’m not in a position to upgrade it at the moment – I can re-test at a later date with more current code.  I don’t expect the behavior to change, however.

I started with the default discovery policy of 1-link.   I enabled one link per fabric interconnect to a chassis, and checked the status of the ports.   The ports were in an “Up” Overall Status, which is proper – and means that the link is available for use.   I then enabled a second link per fabric interconnect to the same chassis.   The second activated link went into a “Link Up” state, with “Additional info” of “FEX not configured”.   This means that the physical link has been activated, but is not in use – the IO Module (FEX) is not using the link.   A re-acknowledgement of the chassis activates the second link.

I then changed the discovery policy to 4-links.   I enabled one link per fabric interconnect to a different chassis, and checked the status of the ports.  The ports went into the “Link Up” state, “FEX Not Configured” – in other words, the links are not use.   While we can detect that the chassis is present, no traffic will flow as the FEX has not yet been configured to pass traffic.   Furthermore, the chassis is an error state, in as much as the cabling doesn’t match the chassis discovery policy.

Reacknowledging the chassis activates the links, and removes the error condition, even without changing the discovery policy.

Finally, I tested the scenario of setting the chassis discovery policy to 2 links and activating only one link per Fabric Interconnect.   As expected, the link enters the “link-up”, “FEX not configured” state.   I then activated a second link per Fabric Interconnect.   After a brief period, both ports per Fabric Interconnect enter the “Up” status.

In short, setting the Chassis Discovery Policy determines how many links must be present per Fabric Interconnect to an IO Module before the ports are activated and the chassis is put into service.   If the policy is set at 1 link, and more than 1 link are activated, a simple re-acknowledgement of the chassis will activate the additional links.   If the policy is set to a higher number – and that number of links are not present – the chassis will not be activated unless a manual re-acknowledgement is done.

Frankly, I don’t see a lot of value in this feature – unless you’re adding a large number of chassis that will all have identical numbers of uplinks.  Even then, you’re saving yourself at most a few clicks of the mouse to re-ack the chassis that don’t match the policy.   Why not simply leave the policy at default and re-ack any chassis that has more than 1 link per IOM?   Presumably you’re going to do this before actually deploying any service profiles, so there’s no potential for disruption with the re-ack.

Thoughts or comments?

The Problem With Vendor Sponsored Testing

I suppose this post has been a long time coming.

It was spurred into reality by an exchange with @bladeguy who pointed out that Cisco, too, sponsors tests of their equipment – just like HP and the Tolly reports.  At first, I’d intended to do a comparison of the Tolly reports and the Principled Technologies reports, looking for obvious (or not so obvious) bias.   Once I started down that path, however, I realized it really wasn’t necessary.   Sponsored tests (from any organization) will always be biased, and therefore unreliable from a technical perspective.   There are always tuning parameters that the “loser” will insist were wrong which skewed the results, there are always different ways to architect the test that would have given the “loser” an edge.   That’s why they’re sponsored tests.

I commented briefly before on Tolly’s first HP vs. Cisco UCS report, so I won’t repeat it again here.  Suffice it to say, the bloggers and such did a pretty good job of chopping it up.

My issue with the Tolly reports I’ve seen thus far, the bandwidth debacle and the airflow test, simply don’t pass the smell test.   Yes, they’re repeatable.   Yes, the testing results are defensible.   But the conclusions and declarations of a “winner”?  Not so much.   I’m not faulting Tolly here.   They’re doing their job, producing the product that their customer has asked them for.  These tests aren’t sponsored by the HP engineering groups (for whom I have a lot of respect) looking to validate their technological prowess – they’re sponsored by the marketing departments to provide ammunition for the sales process.   As such, do you really think they’re going into this looking for a fair fight?   Of course not.   They’re going to stack the deck in their favor as much as they think they can get away with (and knowing marketing departments, more than they can get away with).   That’s what marketing groups (from every vendor) do.

@bladeguy pointed out that Cisco has engaged Principled Technologies to do some testing of UCS equipment versus both legacy and current HP equipment.  At first glance, I didn’t detect any significant bias – especially in the tests comparing legacy equipment to current UCS gear.   I’m not sure how any bias could be construed, since really they’re just showing the advantage and consolidation ratios capable when moving from old gear to new gear.   Obviously you can’t compare against Cisco’s legacy servers (since there aren’t any), and HP servers are the logical choice since they have a huge server market share.   I would suspect that similar results would have been achieved when comparing legacy HP equipment against current HP equipment as well.   HP can (and I’m sure has) perform similar tests if they’d like to demonstrate that.

The more troublesome tests are those comparing current generations of equipment from two manufacturers.   The sponsor of the test will always win, or that report will never see the light of day.  That’s simply how it works.   Companies like Tolly, Principled Technologies, etc aren’t going to bite the hand that feeds them.   As such, they’re very careful to construct the tests such that the sponsor will prevail.   This is no secret in the industry.   It’s been discussed many times before.

Even the Principled Technologies tests that compared current generations of hardware looked like pretty fair fights to me.  If you look closely at the specifications of the tested systems, they really tend to reveal the benefits of more memory, or other such considerations, as opposed to the hardware itself.   @bladeguy pointed out several items in the Principled Technologies tests that, in his opinion, skewed the results towards Cisco.   I’m not in any position to refute his claims – but the items he mentioned really come down to tuning.   So essentially he’s saying that the HP equipment in the test wasn’t tuned properly, and I’m certainly not going to argue that point.   As a sponsored test, the sponsor will be victorious.

And therein lies the problem.   Sponsored tests are meaningless, from any vendor.   I simply don’t believe that sponsored tests provide value to the technical community.  But that’s ok – they’re not targeted at the technical community.   They’re marketing tools, used by sales and marketing teams to sway the opinions of management decision makers with lots of “independent” results.    If I want to know which server platform is better for my environment, I’m going to do my own research, and if necessary invite the vendors in for a bake-off.   Prove it to me, with your tuning as necessary, and I’ll have the other vendors do the same.

My real problem with these tests, understanding that they’re not aimed at the technical community, is the that many in the technical community use them to “prove” that their platform is superior to whoever their competing against at the moment.   Like politics, these kinds of arguments just make me tired.   Anyone coming into the argument already has their side picked – no amount of discussion is going to change their mind.   My point for blogging about UCS is not to sell it – I don’t sell gear.   It’s because I believe in the platform and enjoy educating about it.

I happen to prefer Cisco UCS, yes.   If you’ve ever been in one of my UCS classes, you’ll have also heard me say that HP and IBM – Cisco’s chief rivals in this space – also make excellent equipment with some very compelling technologies.   The eventual “best” solution simply comes down to what’s right for your organization.   I understand why these sponsored tests exist, but to me, they actually lessen the position of sponsor.   They make me wonder, “if your product is so good, why stack the deck in the test?”   The answer to that, of course, is that the engineers aren’t the ones requesting or sponsoring the test.

As came up in my UCS class today, for the vast majority of data center workloads, the small differences in performance that you might be able to get out of Vendor X or Vendor Y’s server is completely meaningless.   When I used to sell/install storage, I used to get asked the question as to which company’s storage to buy if the customer wanted maximum performance.  My answer, every single time was, “HP, or IBM, or HDS, or EMC, or…”   Why?  Because technology companies are always leapfrogging each other with IOPS and MB/s and any other metric you can think of.   What’s the fastest today in a particular set of circumstances will get replaced tomorrow by someone else.

So what’s the solution?  Well, true independent testing, of course.   How do you do true independent testing?   You get a mediator (Tolly, Principled Technologies, etc are fine), representatives from both manufacturers to agree on the testing criteria, and allow each manufacturer to submit their own architecture and tuning to meet the testing criteria.   The mediator then performs the tests.    The results are published with the opportunity for both manufacturers to respond to the results.   Think any marketing organization from any company would ever allow this to happen?   The standard line in the testing industry is “Vendor X was invited to participate, but declined.”  Of course they declined.   They’ve already lost before the first test was run.   I wouldn’t expect Cisco to participate in a HP-sponsored Tolly test any more than I’d expect HP to participate in a Cisco-sponsored Principled Technologies test.

Don’t chase that 1% performance delta.   You’ll just waste time and money.   Find the solution that meets your organizational needs, provides you the best management and support options, and buy it.   Let some other chump chase that magical “fastest” unicorn.   It doesn’t exist.  As in all things IT, “it depends.”

All comments are welcome, including those from testing organizations!

Private Isolated VSANs?

Ok, so this isn’t really UCS related.   Just a random thought I had today while working on a lab project… why don’t we have Private VSANs?   As in, the same type of technology as Private VLANs (PVLANs)?

First, some background.   Standard SAN best practice for access control is to use single-initiator/single-target zoning.   This means that there’s one zone for each combination of host and storage, tape, virtualization platform, etc port.    Some administrators think this is overkill, and create just a few zones of lots of initiators to single targets, but this is overall a bad idea.   The purpose of this post is not to argue for single-initiator zoning, since it’s accepted recommended practice.

Private VLANs provide a method for simplifying access control within a L2 Ethernet domain, restricting access between nodes.   Community PVLANs allow communication only between members of the same community, and the promiscuous port(s).   This is actually fairly close to the idea of a fibre channel zone, with the distinction that fibre channel doesn’t have promiscuous ports.   Isolated PVLANs allow communication only between each individual node and the promiscuous port(s).   In a way, you could compare this to having a lot of nodes (initiators) zoned only to a single target node (target) in fibre channel – but without the administrative overhead of zoning.

So, why not combine these approaches?   Having the concept of an Isolated Private VSAN would simplify some types of fibre channel deployments, by enforcing recommended practices around access control without the administrative overhead.  In a smaller environment, you could simply create an Isolated Private VSAN to contain the ports for a given fabric – set the storage ports as promiscuous, and all node ports would be restricted to connecting only to the storage ports – and prevented from communicating with each other.   In fact, I’d imagine that this would be enforced with standard FC zoning (since that’s the hosts are expecting when they query the name server anyway) – really we’d just be automating the creation of the zones.   Cisco already does something similar by automatically creating zones when doing Inter-VSAN Routing (IVR).

For slightly larger environments, I could even see adding in the idea of Community Private VSANs – whereby you group initiators and specify specific target (promiscuous) ports per community – without having to add additional VSANs.

Now that I’m thinking out-loud, why not have isolated zones instead?   Mark a zone as “isolated”, and tag any necessary WWNs/ports/etc as promiscuous, and enforce the traditional zoning behind the scenes.

True, this approach wouldn’t accomplish anything that traditional VSANs and zoning do not.  The implementation would likely have to use traditional zoning behind the scenes.   Just as PVLANs aren’t used in every situation, nor would PVSANs, but I could definitely see some use cases here.  So what do you think?   Am I completely insane?   Thoughts, comments, rebukes are all welcome.  🙂

UCSM 1.3(1c) Released!

Cisco has released UCS Manager version 1.3(1c).   This is the first public release in the 1.3 line, also known as “Aptos+”.

Release notes are here:

Haven’t gotten a chance to play with the new version yet, but there are some significant enhancements.    Among them…

  • 1 GE support on UCS6120 and UCS6140 Fabric Interconnects
    • On the 6120, you can now use 1GE transceivers in the first 8 physical ports.
    • On the 6140, you can now use 1GE transceivers in the first 16 physical ports.
    • Watch for a post soon on why I think this is a bad idea.  🙂
  • Support for the new, 2nd generation mezzanine cards
    • Both Emulex and Qlogic have produced a 2nd generation mezzanine card, using a single-chip design which should lower power consumption
      • Be warned that these new mezzanine cards won’t support the “Fabric Failover” feature as supported by the first generation CNAs, or by the VIC (Palo) adapter
      • These aren’t shipping quite yet, but will be soon
    • A Broadcom BCM57711 mezzanine adapter
      • This will compete with the Intel based, 10GE mezzanine adapters that UCS has had until now
      • The Broadcom card supports TOE (TCP Offload Engine) and iSCSI Offload, but not iSCSI boot
    • An updated Intel mezzanine adapter, based on the Niantic chipset
  • Support for the B440-M1 blade
    • The B440 blade will be available in a 2 or 4 processor configuration, using the Intel Xeon 7500 processors
    • Up to 4 SFFP hard drives
    • 32 DIMM slots, for up to 256GB of memory
    • 2 Mezzanine slots
    • Full-width form factor
  • SSD hard drive support in B200-M2, B250-M2, and B440-M1 blades
    • First drive available is a Samsung 100GB SSD
  • Improved SNMP support
  • Ability to configure more BIOS options, such as virtualization options, through the service profiles
    • This is a big step towards making UCS blades honestly and truly stateless
    • Previously, I’d recommended that UCS customers configure each blade’s BIOS options to support virtualization when they received them, whether or not they were going to use ESX/etc on all of the blades.  This way they didn’t have to worry about setting them again when moving service profiles
  • Support for heterogeneous mezzanine adapters in full-width blades
  • Increased the supported limit of chassis to 14.
  • Increased the limit of VLANs in UCSM to 512
    • There’s been some discussion around this lately, particular in the service provider space.   Many service providers need many more VLANs than this for their architectures.
    • I’ve seen reference to a workaround using ESX, Nexus 1000V, private VLANs, and a promiscuous VLAN through the Fabric Interconnect into an upstream switch, but I’m still trying to get my head around that one.  🙂
  • Ability to cap power levels per blade
    • Will have to wait until I get a chance to test out the code level to see what kinds of options are available here

Looking forward to seeing customer reaction to the new features.

Correction to L2 Forwarding Rules post

I posted here about the L2 forwarding rules when UCS is in EHV mode.   Several readers have pointed out a flaw in the logic I posted, which was taken from Cisco’s DCUCI course.   In Cisco’s defense, I did write that course.   🙂

At issue is how UCS deals with unknown unicast frames.   The other post incorrectly states than an unknown unicast frame received from a server port would be flooded out all other server ports participating in the same VLAN.   This is not the case.

The logic behind EHV mode is that it is impossible to have an unknown unicast address “behind” or “south” of the Fabric Interconnect.   All adapter MAC addresses are known, either because they were assigned in the service profile or inventoried (if using “derived” values).    For MAC addresses that are generated within a host, say for a virtual machine, the assumption is that at creation (or arrival through vMotion, etc) the MAC address will be announced using gratuitous ARP or other traffic generation techniques and the Fabric Interconnect can learn the address through normal L2 methods.

So to clarify, an unknown unicast frame received from a server port will be flooded out ONLY that interface’s pinned uplink.   Otherwise, all traffic destined for MAC addresses outside of UCS (such as the MAC address of a default gateway, for example) would also get flooded internally – which would not be a good thing.