Surely everyone has heard this saying, as a way of suggesting that one’s sphere of knowledge is perhaps limited. Indulge me for a moment in an example from my own past.

It was the early nineties and I was working on my truck, a 1982 Datsun 4×4 pickup.

Datsun 720 Pickip (not mine – but similar)

Now, for those of you unfamiliar with a pitman arm, here’s where it fits into a typical steering setup:

The pitman arm connects the steering box (manual or power) to the rest of the steering links. In other words, once broken, you have absolutely zero control over the direction of your vehicle. Neat, huh?

A friend with a trailer came to rescue me and dragged my truck back to my Dad’s garage, where he had a large selection of tools that I could use to repair my truck. Now, first step was to get a new pitman arm, which was easily obtained (if expensive) from the local Nissan dealer. After that, it was just a matter of yanking the old arm off, bolting in the new one, and connecting it to the linkage. Right?

Not so fast. Now, the pitman arm is pressed onto a splined shaft coming out of the steering box and then large nut secures it further. Removing the nut took a bit of doing, including a fair bit of penetrating oil, etc. Once that was off, I began trying to pull the arm from the shaft. No amount of pounding, pulling, penetrating oil, or other mechanical motivation could get it to budge. Someone suggested using a propane torch to heat up the arm to loosen in – that didn’t help (and made things worse as I’ll get to later) either. I got the brilliant idea to use a gear puller:

Gear Puller

Desperate (it was now about 18 hours into the removal process), I went to a local mom-and-pop tool store. I asked the grizzled man behind the counter for the biggest, meanest gear puller he had. The conversation went something like this:

Tool Guy: What kind of gear are you pulling?
Me: I just need a really strong gear puller.
Tool Guy: What kind of gear are you pulling?
Me: It’s not a gear. But I need a strong gear puller to do what I’m trying to do.
Tool Guy: What are you pulling?
Me: *rolls eyes* Well, if you MUST know, it’s a pitman arm.
Me (to myself): Pshh, like you even know what a pitman arm is. Sheesh.
Tool Guy: Well, why don’t you use a pitman arm puller like this one?

Pitman Arm Puller

No joke, once I had the right tool, the pitman arm was off in 5 minutes. I installed the new arm, aligned my steering wheel, and drove home.

In the morning, I went out to drive to work, and there’s a puddle of power steering fluid beneath my truck. Uh oh. The heat from the propane torch must have melted the seals inside the steering box. Easiest fix? Get a new steering box from a junkyard and install it. Guess what the new steering box had already installed? A perfectly functional pitman arm that I didn’t have to pull.

So what lessons did I learn here?

Just because you have a large number of tools at your disposal, don’t assume you have them all – or even that you know what tools exist. Simply knowing that there was a tool called a “pitman arm puller” would have saved me days of effort, and ultimately, a lot of money in replacing parts that I damaged by using the wrong tools.

Always be looking for new or better tools. “I’ve always done it this way” is not a valid explanation for why you used the tool you did. “This is the best tool I have found so far, what other ideas do you have?” is a much better approach.

Don’t be so arrogant as to assume you know more than everyone else – even if you happen to know a lot. No one person can contain the whole of human knowledge – reach out to experts in their respective fields (like my Tool Guy) to see if they have ways to make your problems easier to solve. Approach this research with an open mind and humility – it will serve you well.

In technology, it’s very easy to get confortable with the tools/products/vendors that you’ve always used, especially if you can solve the problems you encounter with them. Even if you can solve the problem in front of you with the “way I’ve always done it”, why not be open to new options that might be easier, or solve the problem in a better way?

If you get locked into solving a particular technical problem with a particular product, you may miss opportunities to discover that you’re solving the wrong problem in the first place. Taking a step back from the problem and realizing that you could avoid it in the first place by solving a problem further up the line may ultimately be more efficient. Engage experts. Be humble. Be open to new ideas, technologies, and approaches.

I remind myself of these lessons every day.

In my career, like most of you, I have sat through too many product presentations, marketing pitches, and technical demos to count. I have talked to countless engineers, account managers, architects, gurus, and charlatans. Some folks fit more than one category.

It struck me recently that I tune out almost immediately when I hear, in the context of explaining why I should choose a particular product, that “we have the largest market share and ship more units of this tech than any of our competitors!” Why? Because if that’s your lead story, and not the quality or innovativeness of your tech, you’re riding on past success and inertia. I’m not interested in inertia. I want to see what you’re doing that’s INTERESTING.

A certain large OEM told me that they invented a particular technology class (when in fact it was invented – more or less – by a company they acquired), as a basis for why their technology was superior. Now, don’t get me wrong – “we’ve been doing this longer than anyone else, and therefore have had more time to refine our solution” is a perfectly valid argument – but not as your lead story. Likewise, telling me (this was another OEM) that you invented a particular idea (even though you didn’t) and that everyone else is copying you now should NOT be your marketing pitch. In fact, if you tell me that you came up with an idea first and then everyone else jumped on the bandwagon later, it actually makes me want to look to your competitors. Since they have the advantage of having seen what you did right and wrong, and were able to craft their solutions afterwards – what’s to say that theirs aren’t better? First to market does NOT necessarily mean best.

Just because you were first to market doesn’t mean you’re not the best either – I’m just saying that to me, that fact is irrelevant.

I’ve been accused of being biased to particular technology companies, but that’s not actually the truth. I’m biased to technologies that make sense to me and solve real problems that I have or see in my customers’ environments. If my “A Game” technology (see link for Joe Onisick’s explanation in the context of Cisco UCS) competes with your product, it isn’t because I dislike your technology, it’s because for the problems I’m trying to solve, I prefer this one. Come out with something better, and I’ll look at it.

In my new role, I have the advantage of partnering with many different OEMs and selecting the right products to meet my customers’ needs. These needs are not always purely technical. But as a technical guy, I’m going to start with my “A Game” solution unless a customer requirement dictates something else, or something better technically comes along.

So this is my message to AMs, PMs, and anyone else that wants to convince me (and I’m very open to being convinced, I just have very high standards) to look at their technology: Do not lead with market share, time in the market, or that you invented a particular class of technology. Tell me what you do that’s innovative, solves my customers’ problems, and does it better/faster/cheaper than your competitors. THAT is what I care about.

For the last five years, I have held various roles within Firefly Communications, the premier datacenter education partner within the Cisco ecosystem. I started out as a storage instructor and consulting, teaching primarily Cisco MDS courses, before moving into a little known (for some pretty good reasons) product called VFrame Data Center. It was part of an acquisition Cisco made (TopSpin), and provided an interesting mix of server deployment, automation, network configuration, and policy enforcement across Cisco and other products. As a product, it had massive potential, but a couple of significant flaws – most notably, it’s reliance on the APIs or command lines of third party products that Cisco could neither control nor predict. When that product died, I went back to teaching MDS, and a little bit of a new thing called Fibre Channel over Ethernet on this newfangled line of switches called “Nexus”.

The combination of skill in those three products (MDS, Nexus, and VFrame Data Center) would turn out to be very fortuitous, as I was invited (along with Joe Onisick and Fabricio Grimaldi, both rockstars of the datacenter) to be among the first to learn “Project California”, a mostly-secret project to produce Cisco’s first foray into compute – what eventually would be released as the Cisco Unified Computing System. UCS became my primary technology focus for the rest of my time at Firefly.

Over the subsequent years, I moved through several roles in Firefly, including Product Line Director for the UCS platform, Chief Technology Officer for the Americas, and finally Vice President of Engineering, overseeing all technical instructors, platforms, and internal IT for the company worldwide. It was a challenging but rewarding position, where I was able to use my love of technology and mentoring, and develop my managerial and leadership (not the same things!) skills through interaction with some great mentors.

Eventually, though, it came time to make a change. I had reached the end of what I felt I could accomplish professionally at Firefly, so I decided that it was time to move on.

In deciding what I wanted to move into, I consulted some great peers, associates, and legends for assistance (thanks all of you who provided guidance – especially @drjmetz @bradhedlund @omarsultan @jonisick) and boiled my interests down into a few key areas:

• Technology Evangelism
• I love taking a piece of technology that I’m passionate about, and getting others just as excited about it or helping them see how to solve key operational or business problems with it. I’ve done that with MDS, Nexus, and UCS over the last five years and been very successful and fulfilled by it.
• Staff Mentoring
• One of the more rewarding parts of working for Firefly was the opportunity to work with some of the best and brightest people in the datacenter space. Everyone brings their own unique talents and experiences, and I was able to learn just as much from working with them as any class could ever teach. At the same time, I enjoyed being able to help others develop their skills – whether technical, presentation, instruction, or just general business experiences.
• Independence
• Firefly afforded me a great deal of independence in how I went about mentoring my team and accomplishing the strategy as set forth by the rest of the senior leadership. Not being tied to a desk job with an hour commute each way was very important to me. Being out in the field, in front of my team and customers was always one of the best parts of my job.

After evaluating a number of different roles and companies, I have selected World Wide Technology as my new professional home. I will be working as a Technical Architect in the Federal sales team. I’m excited – and just a little bit nervous – about stepping out of my comfort zone in a familiar company and striking it out on a new adventure. Without challenge there is no growth, and without growth there is only decay. So let’s go see what’s out there.

One of my new year’s resolutions will be to blog more and get more ideas and discussions flowing. In my new role, I will have a much more broad set of technologies to focus on beyond just UCS, so I hope to do the same with my blog here.

Thanks everyone for reading and for your support!

So this is a request for your ideas on what you’d like to see in such a post. I’ll try to answer as many as I can. Please add your suggestions as a comment to this post.

I’m not going to have an opportunity to test this until next week, so I’m hoping to get some feedback about whether or not I’m attempting the impossible. I’m trying to see if I can multiple sub-interfaces on a NetApp VIF all in the same subnet and with the same IP address. This is for a lab environment where each VLAN represents an isolated lab, but I want all of the labs to be able to connect to the same NetApp device using the same IP address so as to simplify addressing, etc.

This diagram describes basically what I’m trying to accomplish. Can anyone confirm if this will or will not work?

I’m proud to announce that the company I work for, Firefly Communications, has been awarded the Global Learning Partner of the Year for 2012! I’m extremely proud of the team I work with and the recognition from Cisco on our dedication to education and the adoption of new technologies.

And now back to our regularly scheduled programming.

This wasn’t intended to be a detailed analysis, just a couple of random musings.   Comments as always are welcome.

—–

While NFS and iSCSI are completely different approaches to accessing
storage, they both “suffer” from the same ailment – TCP.   Remember folks,
TCP was developed in the 70′s for the express purpose of connecting
disparate networks over long, latent, and likely unreliable links.  The
overheads placed onto communication solely to address these criteria
simply aren’t appropriate in the datacenter.  We’re talking about a
protocol written to support links slower than your Bluetooth headset. 

iSCSI is a hack, plain and simple.  It solves a cost problem, not a
technology one.  Even its name is misleading – iSCSI.   It isn’t SCSI over
IP – it’s SCSI over TCP over IP. So call it tSCSI or tiSCSI.

I’m not saying they’re not “good enough”, but why do “good enough” now
that “better” is getting much closer in price?  On the array side, I
expect more and more vendors to go the NetApp route – all protocols in one
box – just turn on which ones you want to use (via appropriate licensing,
of course).  10G DCB makes this even easier and more attractive – one
port, you pick the protocol you’re comfortable with.

As one of my coworkers points out, FCoE is a bit of a cannon – and for many customers,
their storage challenges are more in mosquito scale.

Fibre Channel was developed with storage in mind as a datacenter protocol,
I haven’t seen one yet I like better for moving SCSI commands around *in
the datacenter*.   I’m sure someone will develop a new protocol at some
point that utilizes DCB-specific architectures to replace iSCSI and
FCoE… but why?   If you want a high performance, low latency,
made-for-storage protocol, run FC over whatever wire you feel like.   If
you want a low-cost solution utilizing commodity
hardware/switching/routing, use iSCSI and/or NFS.   I don’t know that
there’s a new problem to solve here.

For customers that already have and know FC, FCoE is a no-brainer.
Nothing new to learn about how to control access, you’re just replacing
the wires.  iSCSI and NFS introduce whole new mechanisms and mindsets into
accessing storage if you’re not used to them.

I saw a quote the other day that said that Fibre Channel is like smoking -
if you’re not already doing it, there’s no reason to start now.   I get
the sentiment, but I don’t agree.   FC as a protocol is the right tool for
a lot of jobs – but it’s not the right tool for every job.

When I connected my NAS to use the appliance port, in order for the vnic of the blade server to communicate with the NAS, i found that there should be a connected uplink port in the 6100 even though I created a private VLAN for the appliance port and the vnic. Why?

I thought the topic would make for a nice brief post explaining the Network Control policy and its effect on Service Profile vNIC objects.

By default, all vNIC configurations use a Network Control Policy called “default” which is created automatically by UCS Manager.

The policy specifies that CDP frames are not delivered to any vNIC using this policy, and that if no uplinks are available, that the vNIC should be brought down.

In Peter’s case, since there were no uplinks available, his vNIC is kept down keeping him from using the appliance connected to the Fabric Interconnect.

If we change the policy to instead to Warning, the vNIC will be kept up (though a system warning will be generated) even when there are no available uplinks.   Note that this effectively disables the Fabric Failover feature on any vNIC using this policy.

If you have some interfaces that you want to stay up even when there are no available uplinks, create a policy with this setting and then specify it in the vNIC configuration.  Alternatively, if you want the default behavior of all vNICs (unless specifically configured) to be that they stay up even when no uplinks are available, you can modify the “default” policy as shown here.

One of the mistakes I made in my original post was to start throwing around numbers without setting any sort of baseline for comparison.   Technology vendors have played slight-of-hand games with units of measure and data rates for years – think of how hard drive manufacturers prefer to define a megabyte (1 million bytes) versus how the rest of the world define[d] a megabyte (2^20 bytes or 1,048,576 bytes).

It’s important that if we’re going to compare the speed of two different network technologies, we establish where we’re taking the measurement.   Is it, as with 10GE, measured as bandwidth available at the MAC layer (in other words, after encoding overhead), or as I perhaps erroneously did with FC, measuring it at the physical layer (in other words, before encoding overhead).   I also incorrectly stated, unequivocally, that 10GE used 64/66b encoding, when in fact 10GE can use 8b/10b, 64b/66b, or other encoding mechanisms – what’s important is not what is used at the physical layer, but rather what is available at the MAC layer.

In the case of 10GE, 10Gb/s is available at the MAC layer, regardless of the encoding mechanism, transceivers, etc used at the physical layer.

The Fibre Channel physical layer, on the other hand, sets its targets in terms of MB/s available to the Fibre Channel protocol (FC-2 and above).  This is the logical equivalent of Ethernet’s MAC layer – after any encoding overhead.  1Gb Fibre Channel (hereafter FC), as the story goes, was designed to provide a usable data rate of 100 MB/s.

If we’re truly going to take an objective look at the two protocols and how much bandwidth they provide at MAC (or equivalent) and above, we have to pick one method and stick with it.   Since the subject is storage-focus (and frankly, most of the objections come from storage folks), let’s agree to use the storage method – measuring in MB/s available to the protocol.   As long as we use that measurement, any differences in encoding mechanism becomes moot.

So back to 1Gb/s FC, with it’s usable data rate of 100 MB/s.   The underlying physical layer of 1Gb/s FC uses a 1.0625 Gb/s data rate, along with 8b/10b encoding.

Now, this is where most of the confusion and debate seems to have crept into the conversation.   I’ve been attacked by a number of folks (not on this site) for suggesting that 1Gb FC has a 20% encoding overhead, dismissing it as long-standing FUD – created by whom and for what purpose, I’ve yet to discover.   No matter how you slice it, a 1.0625 Gb/s physical layer using 8b/10b encoding results in 0.85 Gb/s available to the next layer – in this case, FC-2.   Conveniently enough, as there are 8 bits in a byte, 100MB/s can be achieved over a link providing approximately 800Mb/s, or 0.8Gb/s.

Now, who doesn’t like nice round numbers?   Who cares what the underlying physical layer is doing, as long as it meets your needs/requirements/targets at the next layer up?

If the goal is 100MB/s, 1Gb/s FC absolutely meets it.   Does 1Gb/s FC have a 20% encoding overhead?   Yes.   Is that FUD?  No.   Do we care?   Not really.

As each generation of FC was released, the same physical layer was multiplied, without changing the encoding mechanism.  So 8Gb/s FC is eight times as fast as 1Gb/s FC.  The math is pretty simple : ( 1.0625 * 8 ) * 0.8 = 6.8 Gb/s available to the next layer.   Before my storage folks (by the way – my background is storage, not Ethernet) cry foul, let’s look at what 6.8 Gb/s provides in terms of MB/s.   A quick check of Google Calculator tells me that 6.8 Gb/s is 870 MB/s – well over the 800 MB/s we’d need if we were looking to maintain the same target of 100MB/s per 1 Gb/s of link.   So again, who cares that there’s a 20% encoding overhead?  If you’re meeting your target, it doesn’t matter.   Normalized per Gb/s, that’s about 108 MB/s for every Gb/s of link speed.

At this point, you’re probably thinking – if we don’t care, why are you writing this?   Well, in a converged network, I don’t really care what the historical target was for a particular protocol or link speed.   I care about what I can use.

Given my newly discovered understanding of 10Gb Ethernet, and how it provides 10 Gb/s to the MAC layer, you can already see the difference.   At the MAC layer or equivalent, 10GE provides 10Gb/s, or 1,280MB/s.   8G FC provides 6.8Gb/s, or 870MB/s.   For the Fibre Channel protocol, native FC requires no additional overhead, while FCoE does require that the native FC frame (2148 bytes, maximum) be encapsulated to traverse an Ethernet MAC layer.   This creates a total frame size of 2188 bytes maximum, which is about a 2% overhead incurred by FCoE as compared to native FC.  Assuming that the full bandwidth of a 10Gb Ethernet link was being used to carry Fibre Channel protocol, we’re looking at an effective bandwidth of (1280MB/s * .98) = 1254Mb/s.     Normalized per Gb/s, that’s about 125 MB/s for every Gb/s of link speed.

The whole idea of FCoE was not to replace traditional FC.   It was to provide a single network that can carry any kind of traffic – storage, application, etc, without needing to have protocol-specific adapters, cabling, switching, etc.

Given that VERY few servers will ever utilize 8Gb/s of Fibre Channel bandwidth (regardless of how or where you measure it), why on earth would you invest in that much bandwidth and the cables, HBAs, and switches to support it?   Why wouldn’t you look for a solution where you have burst capabilities that meet (or in this case, exceed) any possible expectation you have, while providing flexibility to handle other protocols?

I don’t see traditional FC disappearing any time soon – but I do think its days are numbered at the access layer.   Sure, there are niche server cases that will need lots of dedicated storage bandwidth, but the vast majority of servers will be better served by a flexible topology that provides better efficiencies in moving data around the data center.   Even at the storage arrays themselves, why wouldn’t I use 10GE FCoE (1254 MB/s usable) instead of 8Gb FC (870 MB/s usable)?

Now, when 16Gb FC hits the market, it will be using 64/66b encoding.   The odd thing, however, is that based on the data I’ve turned up from FCIA, it’s actually only going to be using a line-rate of 14.025 Gb/s, and after encoding overheads, etc, supplying 1600 MB/s usable (though my math shows it to be more like 1700 MB/s) – in keeping with the 1Gb/s = 100MB/s target that FC has maintained since inception.

Sometime after 16Gb FC is released, will come 40GE, followed by 32Gb FC, and again followed by 100GE.   It’s clear that these technologies will continue to leapfrog each other for some time.   My only question is, why would you continue to invest in a protocol-specific architecture, when you can instead have a flexible one?   Even if you want the isolation of physically separate networks (and there’s still justification for that), why not use the one that’s demonstrably more efficient?   FCoE hasn’t yet reached feature parity with FC – there’s no dispute there.  It will, and when it does, I just can’t fathom keeping legacy FC around as a physical layer.   The protocol is rock solid – I can’t see it disappearing the foreseeable future.   The biggest benefits to FCoE come at the access layer, and we have all the features we need there today.

If you’d like to post a comment, all I ask is that you keep it professional.   If you want to challenge my numbers, please, by all means do so – but please provide your math, references for your numbers, and make sure you compare both sides.   Simply stating that one side or the other has characteristic X doesn’t help the discussion, nor does it help me or my readers learn if I’m in error.

Finally, for those who have asked (or wondered in silence) – I don’t work for Cisco, or any hardware manufacturer for that matter.   My company is a consulting and educational organization focused on data center technologies.    I don’t have any particular axe to grind with regards to protocols, vendors, or specific technologies.   I blog about the things I find interesting, for the benefit of my colleagues, customers, and ultimately myself.   Have a better mousetrap?  Excellent.   That’s not going to hurt my feelings one bit. 

To understand the impact, first look at the way disruptive changes were handled prior to this release.   When changing a configuration setting on a service profile, updating service profile template, or many policies, if the change would cause a disruption to running service profiles (i.e. requiring a reboot), you had two options : yes, or no.  When modifying a single profile, this wasn’t a big issue.  You could simply change the configuration when you were also ready to accommodate a reboot of that particular profile.   Where it became troublesome was when you wanted to modify an updating service profile or policy that affected many service profiles – your choice was really only to reboot them all simultaneously, or modify each individually.   Obviously for large deployments using templates and policies (the real strength of UCS), this wasn’t ideal.

With UCSM 1.4, we now have the concept of a Maintenance Policy.   The screenshot below is taken from the Servers tab:

Creating a Maintenance Policy allows the administrator to define the manner in which a service profile (or template) should behave when disruptive changes are applied.   First, there’s the old way:

A policy of “Immediate” means that when a disruptive change is made, the affected service profiles are immediately rebooted without confirmation.   A normal “soft” reboot occurs, whereby a standard ACPI power-button press is sent to the physical compute node – assuming that the operating system traps for this, the OS should gracefully shut down and the node will reboot.

A much safer option is to use the “user-ack” policy option:

When this option is selected, disruptive changes are staged to each affected service profile, but the profile is not immediately rebooted.   Instead, each profile will show the pending changes in its status field, and will wait for the administrator to manually acknowledge the changes when it is acceptable to reboot the node.

The most interesting new option is the is the “timer-automatic” setting.   This setting allows the maintenance policy to reference another new object, the Schedule.

Schedules allow you to define one-time or reoccurring time periods where one or more of the affected nodes may be rebooted without administrator intervention.  Note that the Schedules top-level object is located within the Servers tab:

The only schedule created automatically by UCSM is the “default” schedule, which has one recurring entry to reboot all service profiles that reference a “timed-automatic” maintenance policy associated with the “default” schedule each day at midnight.   This “default” schedule can be modified, of course.

Creating a user-defined schedule provides the ability to control when – and how many – profiles are rebooted to apply disruptive changes.

The One Time Occurrence option sets a single date and time when this schedule will be in effect.  For example, if you wanted all affected profiles to be rebooted on January 18th at midnight, you could create an entry such as the following.

Once the date and time have been selected, the other options for the occurrence can be selected.

Max Duration specifies how long this occurrence can run.   Based on the other options selected below it, it is possible that all service profiles may not be able to be rebooted in the time allotted.   If this is the case, changes to those profiles will not take effect.

Max Number Of Tasks specifies how many total profiles could be rebooted by this occurrence.

Max Number Of Concurrent Tasks controls how many profiles can be rebooted simultaneously.   If, for example, this schedule will be used on a large cluster of service profiles where workload can be sustained even while 5 nodes are unavailable, set this value to 5 and the reboots will occur in groups of that size.

Minimum Internal Between Tasks allows you to set a delay between each reboot.  This can be set to ensure that each node rebooted is given time to fully boot before the next node is taken down.

The Recurring Occurrence option provides for the creation of a schedule that will run every day, or on a specific day, to apply disruptive changes.

This option has the same per-task options as the previous example.

Once you have created your maintenance policy and schedule (if necessary), the service profile or service profile template must reference the maintenance policy in order for it to have any effect.  After selecting your service profile or template, the Actions window has an option to Change Maintenance Policy.

You may then select the Maintenance Policy you wish to use, or create a new one.

The service profile properties will now show that a maintenance policy has been selected.

In this example, a policy requiring user acknowledgement has been chosen.   Now if any disruptive changes are made, the service profile will not reset until manually acknowledged by an administrator.   Any time profiles are awaiting acknowledgement, a warning “Pending Activities” will be shown on the UCSM status bar.

Within the profile properties, a description of the pending changes will be displayed along with the “Reboot Now” option.

I hope this description of the new maintenance policies and schedules options was helpful.  I’m very excited by all the new features rolling into UCS – it was a great system before, and it’s only getting better!