By far, the most popular post on this blog has been my discussion on the various protocol efficiencies between native 8 Gb/s Fibre Channel and 10 Gb/s Ethernet using Fibre Channel over Ethernet encapsulation. I wrote the original post as much as an exercise in the logic as it was an attempt to educate. I find that if I can’t explain a subject well, I don’t yet understand it. Well, as it’s been pointed out in the comments of that post, there were some things that I missed or just had simply wrong. That’s cool – so let’s take another stab at this. While I may have been wrong on a few points, the original premise still stands – on a per Gb/s basis, 10Gb FCoE is still more efficient than 8Gb FC. In fact, it’s even better than I’d originally contended.
One of the mistakes I made in my original post was to start throwing around numbers without setting any sort of baseline for comparison. Technology vendors have played slight-of-hand games with units of measure and data rates for years – think of how hard drive manufacturers prefer to define a megabyte (1 million bytes) versus how the rest of the world define[d] a megabyte (2^20 bytes or 1,048,576 bytes).
It’s important that if we’re going to compare the speed of two different network technologies, we establish where we’re taking the measurement. Is it, as with 10GE, measured as bandwidth available at the MAC layer (in other words, after encoding overhead), or as I perhaps erroneously did with FC, measuring it at the physical layer (in other words, before encoding overhead). I also incorrectly stated, unequivocally, that 10GE used 64/66b encoding, when in fact 10GE can use 8b/10b, 64b/66b, or other encoding mechanisms – what’s important is not what is used at the physical layer, but rather what is available at the MAC layer.
In the case of 10GE, 10Gb/s is available at the MAC layer, regardless of the encoding mechanism, transceivers, etc used at the physical layer.
The Fibre Channel physical layer, on the other hand, sets its targets in terms of MB/s available to the Fibre Channel protocol (FC-2 and above). This is the logical equivalent of Ethernet’s MAC layer – after any encoding overhead. 1Gb Fibre Channel (hereafter FC), as the story goes, was designed to provide a usable data rate of 100 MB/s.
If we’re truly going to take an objective look at the two protocols and how much bandwidth they provide at MAC (or equivalent) and above, we have to pick one method and stick with it. Since the subject is storage-focus (and frankly, most of the objections come from storage folks), let’s agree to use the storage method – measuring in MB/s available to the protocol. As long as we use that measurement, any differences in encoding mechanism becomes moot.
So back to 1Gb/s FC, with it’s usable data rate of 100 MB/s. The underlying physical layer of 1Gb/s FC uses a 1.0625 Gb/s data rate, along with 8b/10b encoding.
Now, this is where most of the confusion and debate seems to have crept into the conversation. I’ve been attacked by a number of folks (not on this site) for suggesting that 1Gb FC has a 20% encoding overhead, dismissing it as long-standing FUD – created by whom and for what purpose, I’ve yet to discover. No matter how you slice it, a 1.0625 Gb/s physical layer using 8b/10b encoding results in 0.85 Gb/s available to the next layer – in this case, FC-2. Conveniently enough, as there are 8 bits in a byte, 100MB/s can be achieved over a link providing approximately 800Mb/s, or 0.8Gb/s.
Now, who doesn’t like nice round numbers? Who cares what the underlying physical layer is doing, as long as it meets your needs/requirements/targets at the next layer up?
If the goal is 100MB/s, 1Gb/s FC absolutely meets it. Does 1Gb/s FC have a 20% encoding overhead? Yes. Is that FUD? No. Do we care? Not really.
As each generation of FC was released, the same physical layer was multiplied, without changing the encoding mechanism. So 8Gb/s FC is eight times as fast as 1Gb/s FC. The math is pretty simple : ( 1.0625 * 8 ) * 0.8 = 6.8 Gb/s available to the next layer. Before my storage folks (by the way – my background is storage, not Ethernet) cry foul, let’s look at what 6.8 Gb/s provides in terms of MB/s. A quick check of Google Calculator tells me that 6.8 Gb/s is 870 MB/s – well over the 800 MB/s we’d need if we were looking to maintain the same target of 100MB/s per 1 Gb/s of link. So again, who cares that there’s a 20% encoding overhead? If you’re meeting your target, it doesn’t matter. Normalized per Gb/s, that’s about 108 MB/s for every Gb/s of link speed.
At this point, you’re probably thinking – if we don’t care, why are you writing this? Well, in a converged network, I don’t really care what the historical target was for a particular protocol or link speed. I care about what I can use.
Given my newly discovered understanding of 10Gb Ethernet, and how it provides 10 Gb/s to the MAC layer, you can already see the difference. At the MAC layer or equivalent, 10GE provides 10Gb/s, or 1,280MB/s. 8G FC provides 6.8Gb/s, or 870MB/s. For the Fibre Channel protocol, native FC requires no additional overhead, while FCoE does require that the native FC frame (2148 bytes, maximum) be encapsulated to traverse an Ethernet MAC layer. This creates a total frame size of 2188 bytes maximum, which is about a 2% overhead incurred by FCoE as compared to native FC. Assuming that the full bandwidth of a 10Gb Ethernet link was being used to carry Fibre Channel protocol, we’re looking at an effective bandwidth of (1280MB/s * .98) = 1254Mb/s. Normalized per Gb/s, that’s about 125 MB/s for every Gb/s of link speed.
The whole idea of FCoE was not to replace traditional FC. It was to provide a single network that can carry any kind of traffic – storage, application, etc, without needing to have protocol-specific adapters, cabling, switching, etc.
Given that VERY few servers will ever utilize 8Gb/s of Fibre Channel bandwidth (regardless of how or where you measure it), why on earth would you invest in that much bandwidth and the cables, HBAs, and switches to support it? Why wouldn’t you look for a solution where you have burst capabilities that meet (or in this case, exceed) any possible expectation you have, while providing flexibility to handle other protocols?
I don’t see traditional FC disappearing any time soon – but I do think its days are numbered at the access layer. Sure, there are niche server cases that will need lots of dedicated storage bandwidth, but the vast majority of servers will be better served by a flexible topology that provides better efficiencies in moving data around the data center. Even at the storage arrays themselves, why wouldn’t I use 10GE FCoE (1254 MB/s usable) instead of 8Gb FC (870 MB/s usable)?
Now, when 16Gb FC hits the market, it will be using 64/66b encoding. The odd thing, however, is that based on the data I’ve turned up from FCIA, it’s actually only going to be using a line-rate of 14.025 Gb/s, and after encoding overheads, etc, supplying 1600 MB/s usable (though my math shows it to be more like 1700 MB/s) – in keeping with the 1Gb/s = 100MB/s target that FC has maintained since inception.
Sometime after 16Gb FC is released, will come 40GE, followed by 32Gb FC, and again followed by 100GE. It’s clear that these technologies will continue to leapfrog each other for some time. My only question is, why would you continue to invest in a protocol-specific architecture, when you can instead have a flexible one? Even if you want the isolation of physically separate networks (and there’s still justification for that), why not use the one that’s demonstrably more efficient? FCoE hasn’t yet reached feature parity with FC – there’s no dispute there. It will, and when it does, I just can’t fathom keeping legacy FC around as a physical layer. The protocol is rock solid – I can’t see it disappearing the foreseeable future. The biggest benefits to FCoE come at the access layer, and we have all the features we need there today.
If you’d like to post a comment, all I ask is that you keep it professional. If you want to challenge my numbers, please, by all means do so – but please provide your math, references for your numbers, and make sure you compare both sides. Simply stating that one side or the other has characteristic X doesn’t help the discussion, nor does it help me or my readers learn if I’m in error.
Finally, for those who have asked (or wondered in silence) – I don’t work for Cisco, or any hardware manufacturer for that matter. My company is a consulting and educational organization focused on data center technologies. I don’t have any particular axe to grind with regards to protocols, vendors, or specific technologies. I blog about the things I find interesting, for the benefit of my colleagues, customers, and ultimately myself. Have a better mousetrap? Excellent. That’s not going to hurt my feelings one bit. 🙂
