Update 2011/01/31 – I’ve added new thoughts and comments on the subject here: http://www.unifiedcomputingblog.com/?p=234
Which is better? Which is faster?
I’ve been stuck on this one for a while. I’m traditionally a pure fibre channel kind of guy, so I’ve been pretty convinced that traditional FC was here to stay for a while, and that FCoE – as much as I believe in the technology – would probably be limited to the access and aggregation layers for the near term. That is, until it was pointed out to me the encoding mechanisms used by these two technologies and the effective data rates they allowed. I’m not sure why it never occurred to me before, but it hit me like the proverbial ton of bricks this week.
First off, a quick review of encoding mechanisms. Any time we’re transmitting or storing data, we encode it in some form or another. Generally, this is to include some type of a checksum to ensure that we can detect errors in reading or receiving the data. I remember the good old days when I discovered that RLL hard drive encoding was 50% more efficient than MFM encoding, and with just a new controller and a low level format, my old 10MB (yes, that’s ten whopping megabytes, kids. Ask your parents what a megabyte was.) suddenly became 15MB! Well, we’re about to embark on a similar discovery.
1, 2, 4, and 8 Gb Fibre Channel all use 8b/10b encoding. Meaning, 8 bits of data gets encoded into 10 bits of transmitted information – the two bits are used for data integrity. Well, if the link is 8Gb, how much do we actually get to use for data – given that 2 out of every 10 bits aren’t “user” data? FC link speeds are somewhat of an anomaly, given that they’re actually faster than the stated link speed would suggest. Original 1Gb FC is actually 1.0625Gb/s, and each generation has kept this standard and multiplied it. 8Gb FC would be 8×1.0625, or actual bandwidth of 8.5Gb/s. 8.5*.80 = 6.8. 6.8Gb of usable bandwidth on an 8Gb FC link.
10GE (and 10G FC, for that matter) uses 64b/66b encoding. For every 64 bits of data, only 2 bits are used for integrity checks. While theoretically this lowers the overall protection of the data, and increases the amount of data discarded in case of failure, that actual number of data units that are discarded due to failing serialization/deserialization is minuscule. For a 10Gb link using 64b/66b encoding, that leaves 96.96% of the bandwidth for user data, or 9.7Gb/s.
So 8Gb FC = 6.8Gb usable, while 10Gb Ethernet = 9.7Gb usable. Even if I was able to use all of the bandwidth available on an 8Gb FC port (which is very unlikely at the server access layer), with 10GE running FCoE, I’d still have room for 3 gigabit Ethernet-class “lanes”. How’s that for consolidation?
10Gb FC has the same usable bandwidth, and without the overhead (albeit a small 2% or so) of FCoE, but you don’t get the consolidation benefits of using the same physical link for your storage and traditional Ethernet traffic.
I’m sold.