Demartek Storage Networking Interface Comparison
Updated 30 December 2011
By Dennis Martin, Demartek President
Because of the number of storage interface types and related technologies that are used for storage devices, we have compiled this summary document providing some basic information for each of the interfaces. This document will be updated periodically. This document may become larger over time. Contact us if you’d like to see additional information in this document.
The interface types listed here are known as “block” interfaces, meaning that they provide an interface for “block” reads and writes. They simply provide a conduit for blocks of data to be read and written, without regard to file systems, file names or any other knowledge of the data in the blocks. The host requesting the block access provides a starting address and number of blocks to read or write.
We are producing deployment guides for some of the interfaces described in this document. The Demartek iSCSI Deployment Guide 2011 is now available.
More information
To be notified when we post new deployment guides, additional tips and reports, sign-up for our free monthly newsletter, Demartek Lab Notes. We do not give out, rent or sell our email list.
Contents
- Acronyms
- Storage Networking Interface Comparison Table
- Transfer Rate, Bits vs. Bytes, and Encoding Schemes
- History
- Roadmaps
- Cables: Fiber Optics and Copper
- Connector Types
- PCI Express (PCIe)
- FC — Fibre Channel
- FCoE — Fibre Channel over Ethernet (also see Demartek FCoE Zone)
- IB — Infiniband
- iSCSI — Internet Small Computer System Interface (also see Demartek iSCSI Zone)
- PCIe — PCI Express
- SAS — Serial Attached SCSI
- SATA — Serial ATA
- USB — Universal Serial Bus
- 10GbE — 10 Gigabit Ethernet
- CNA — Converged Network Adapter (used with FCoE)
- HBA — Host Bus Adapter (used with FC, iSCSI, SAS, SATA)
- HCA — Host Channel Adapter (used with IB)
- NIC — Network Interface Controller or Network Interface Card (used with FCoE, iSCSI)
- ISL — Inter-Switch Link
- SAN — Storage Area Network
- Gb — Gigabit
- GB — Gigabyte
- Mb — Megabit
- MB — Megabyte
- Gb/s — Gigabits per second
- Gbit/s — Gigabits per second
- Gbps — Gigabits per second
- GB/s — Gigabytes per second
- GBps — Gigabytes per second
- Mb/s — Megabits per second
- Mbit/s — Megabits per second
- Mbps — Megabits per second
- MB/s — Megabytes per second
- MBps — Megabytes per second
 |
Number of Devices | Maximum Distance (m) | Cable Type | Interface Device | Transfer Rate (MB/sec) | Interface Attributes |
|---|---|---|---|---|---|---|
| FC | 16M | 10 (copper) 10KM+ (optical) |
Copper Optical |
HBA | 100, 200, 400, 800, 1600 | Dual Port |
| FCoE | 16M | 10 (copper) very long (optical) |
Copper Optical |
CNA 10GbE NIC |
1150 | Dual Port |
| IB | 16M | 15 (copper) very long (optical) |
Copper Optical |
HCA | 1000, 2000, 4000 | Full Duplex, Dual Port |
| iSCSI | Many | Ethernet cable distance | Copper Optical |
NIC, HBA | 100, 1000 | |
| SAS (passive) |
16K | 10 | Copper | Onboard, HBA | 300, 600 | Full Duplex, Dual Port |
| SAS (active) |
16K | 20 | Copper | Onboard, HBA | 300, 600 | Full Duplex, Dual Port |
| SAS (active) |
16K | 100 | Optical | Onboard, HBA | 300, 600 | Full Duplex, Dual Port |
| SATA | 1 | 1 | Copper | Onboard, HBA | 150, 300, 600 | Half Duplex, Single Port |
| USB | 127 | 5 | Copper, Wireless | Onboard, Adapter card | 0.15, 1.5, 48, 500 | Single Port |
PCIe data rates are provided in the PCI Express section below.
Transfer rate, sometimes known as transfer speed, is the maximum rate at which data can be transferred across the interface. This is not to be confused with the transfer rate of individual devices that may be connected to this interface. Some interfaces may not be able to transfer data at the maximum possible transfer rate due to processing overhead inherent with that interface. Some interface adapters provide hardware offload to improve performance, manageability and/or reliability of the data transmission across the respective interface. The transfer rates listed are across a single port at half duplex.
Bits vs. Bytes and Encoding Schemes
Transfer rates for storage interfaces and devices are generally listed as MB/sec or MBps (MegaBytes per second),
which is generally calculated as Megabits per second (Mbps) divided by 10. Many of these interfaces use
“8b/10b” encoding which maps 8 bit bytes into 10 bit symbols for transmission on the wire, with the extra
bits used for command and control purposes. When converting from bits to bytes on the interface, dividing
by ten (10) is exactly correct. 8b/10b encoding results in a 20 percent overhead (10-8)/10 on the raw bit rate.
Beginning with 10GbE and 10GbFC (for ISL’s), some of the newer speeds emerging in 2010 and beyond, a newer “64b/66b” encoding scheme is being used to improve data transfer efficiency. 64b/66b is the encoding scheme for 16Gb FC and is planned for higher data rates for IB. 64b/66b encoding is not directly compatible with 8b/10b, but the technologies that implement it will be built so that they can work with the older encoding scheme. 16Gb Fibre Channel uses a line rate of 14.025 Gbps, but with the 64b/66b encoding scheme results in a doubling of the throughput of 8Gb Fibre Channel, which uses a line rate of 8.5 Gbps with the 8b/10b encoding scheme. 64b/66b encoding results in a 3 percent overhead (66-64)/66 on the raw bit rate.
PCIe versions 1.x and 2.x use 8b/10b encoding. PCIe version 3 uses 128b/130b encoding, resulting in a 1.5 percent overhead on the raw bit rate. Additional PCIe information is provided in the PCI Express section below.
History
Products became available with the interface speeds listed during these years. Newer interface speeds are often
available in switches and adapters long before they are available in storage devices and storage systems.
- FC — 1Gb/s in 1997, 2Gb/s in 2001, 4Gb/s in 2005, 8Gb/s in 2008, 10Gb/s (ISL only) 2004, 16Gb/s in 2011
- FCoE — FC:4Gb/s and Ethernet:10Gb/s in 2008, 10Gb/s in 2009.
(FC-BB-5 was approved in June 2009, INCITS 462-2010 was approved in Spring 2010) - IB — 10Gb/s in 2002, 20Gb/s in 2005, 40Gb/s in 2008, 56Gb/s in 2011
- iSCSI — 1Gb/s in 2003, 10Gb/s in 2007 (basic 10GbE first appeared in 2002)
- SAS — 3Gb/s in 2005, 6Gb/s in 2009
- SATA — 1.5Gb/s in 2003, 3Gb/s in 2005, 6Gb/s in 2010
- USB — 1.5Mb/s in 1997?, 12Mb/s in 1999?, 480Mb/s in 2001?, 5Gb/s in 2009
PCIe history is provided in the PCI Express section below.
Roadmaps
These roadmaps include the estimated calendar years that higher speeds may become available and are based on our industry
research, which are subject to change. Looking at past history shows that several of these interfaces are on a
three-to-four year development cycle for the next improvement in speed. It is reasonable to expect that pace to continue.
It should be noted that it typically takes several months after the specification is complete before products are
generally available in the marketplace. Widespread adoption of those new products takes additional time, sometimes years.
Some of the standards groups are now working on “Energy Efficient” versions of these interfaces to indicate additions to
their respective standards to reduce power consumption.
See the Connector Types section below for additional roadmap information.
- FC — 16Gb/s in late 2011. The 16Gb Fibre Channel specification, FC-PI-5, was completed on September
28, 2010. 16Gb FC (“16GFC”) is backward compatible with 8Gb and 4Gb FC. Also note the encoding schemes
described above and the cable length estimates below. Work on the 32Gb FC standard,
FC-PI-6, began in early 2010 and 32Gb products are expected to become available by 2014 or 2015. The 32Gb FC standard
is expected to include Energy Efficient features. Each FC revision is expected to be backwards compatible with at
least the two previous generations.
- SAN interface — FC has a future as a SAN interface for the foreseeable future. There has been a huge investment in FC infrastructure over the years, primarily in enterprise datacenters, which is likely to remain deployed for many years.
- Disk drive interface — FC is approaching end-of-life as a disk drive interface, as the disk drive manufacturers are moving to 6Gb/s SAS as the interface for enterprise drives. We expect to see the FC interface on 3.5-inch disk drives to live a while to maintain spare parts, due to the relatively large number of 3.5-inch FC disk drives in enterprise disk subsystems. We expect that relatively few 2.5-inch enterprise disk drives will have an FC interface.
- FCoE — 40Gb/s a few years away, possibly in the same time period as 32Gb FC. The IEEE 802.3ba 40Gb/s and 100Gb/s Ethernet standards were ratified in June 2010. Products are expected to follow over time. It is expected that 40Gb FCoE and 100Gb FCoE based on the 2010 standards will be used exclusively for Inter-Switch Link (ISL) cores, thereby maintaining 10Gb FCoE as the predominant FCoE edge connection probably through 2012 or 2013. See the connector types section below for discussion on 40Gb and 100Gb connectors.
- IB — 56Gb/s in late 2011, 100Gb/s in 2013.
- iSCSI — follows Ethernet roadmap (see FCoE roadmap above).
- SAS — SAS Advanced Connectivity (active copper and optical) in 2010 or 2011, 12Gb/s in 2012-2013. We are beginning to see SAS used as a SAN fabric in some server and storage solutions. Currently ongoing in the T10 (www.t10.org) committee is the development of SCSI over PCIe (SOP), an effort to standardize the SCSI protocol across a PCIe physical interface. SOP will support two queuing interfaces - NVMe and PQI (PCIe Queuing Interface).
- SATA — SATA Express is a new specification under development that combines SATA software infrastructure with the PCI Express® (PCIe®) interface to deliver high-speed storage solutions. SATA express enables the development of new devices that utilize the PCIe interface and maintain compatibility with existing SATA applications. The technology will provide a cost-effective means to increase device interface speeds to 8Gb/s and 16Gb/s. The SATA Express specification is expected to be completed by the end of 2011 or early 2012.
- USB — Some industry estimates are that SuperSpeed USB (USB 3.0) provides good headroom until at least 2014. The SuperSpeed USB and xHCI controller were designed to scale to reach data rates of 25Gb/s and beyond in the future.
Cables: Fiber Optics and Copper
As interface speeds increase, expect increased usage of fiber optic cables and connectors for most interfaces. At higher Gigabit speeds (10Gb+), copper cables and interconnects generally have too much amplitude loss except for short distances, such as within a rack or to a nearby rack. This amplitude loss is sometimes called a poor signal-to-noise ratio or simply “too noisy”.
Single-mode fiber vs. Multi-mode fiber
There are two general types of fiber optic cables available: single-mode fiber and multi-mode fiber.
- Single-mode fiber (SMF), typically with an optical core of approximately 9 µm (microns), has lower modal dispersion than multi-mode fiber and can support distances up to 80-100 Km (Kilometers) or more, depending on transmission speed, transceivers and the buffer credits allocated in the switches.
- Multi-mode fiber (MMF), with optical core of either 50 µm or 62.5 µm, supports distances up to 600 meters, depending on transmission speeds and transceivers.
There are different designations for fiber optic cables depending on the bandwidth supported.
- Multi-mode: OM1, OM2, OM3, OM4
- Single-mode: OS1 (there is a proposed OS2 standard)
When planning datacenter cabling requirements, be sure to consider that a service life of 15 to 20 years can be expected for fiber optic cabling, so the choices made today need to support legacy, current and emerging data rates. Also note that deploying large amounts of new cable in a datacenter can be labor- intensive, especially in existing environments.
OM3 and OM4 are newer multi-mode cables that are “laser optimized” (LOMMF) and support 10 Gigabit Ethernet applications. OM3 and OM4 cables are also the only multi-mode fibers included in the IEEE 802.3ba 40G/100G Ethernet standard that was ratified in June 2010. The 40G and 100G speeds are currently achieved by bundling multiple channels together in parallel with special multi-channel (or multi-lane) connector types. This standard defines an expected operating range of up to 100m for OM3 and up to 150m for OM4 for 40 Gigabit Ethernet and 100 Gigabit Ethernet. These are estimates of distance only and supported distances may differ when 40GbE and 100GbE products become available in the coming years. See the Connector Types section below for additional detail.
Newer multi-mode OM2, OM3 and OM4 (50 µm) and single-mode OS1 (9 µm) fiber optic cables have been introduced that can handle tight corners and turns. These are known as “bend optimized,” “bend insensitive,” or have “enhanced bend performance.” These fiber optic cables can have a very small turn or bend radius with minimal signal loss or “bending loss.” The term “bend optimized” multi-mode fiber (BOMMF) is sometimes used.
The Commercial Building Telecommunications Cabling Subcommittee (TR-42.1) of the Telecommunications Cabling Systems Committee (TR-42), is making progress on the first revision of the Telecommunications Infrastructure Standard for Data Centers (TIA-942-A). In this new specification, OM1 and OM2 cables are no longer recognized for use as backbone and horizontal cabling in data centers, but OM3 and OM4 cables are recognized for this purpose. In a June 2011 meeting, OM4 cabling has been designated as “recommended” and is preferred over OM3 cabling for backbone and horizontal cabling in data centers.
OS1 and OS2 single-mode fiber optics are used for long distances, up to 10,000m (6.2 miles) with the standard transceivers and have been known to work at much longer distances with special transceivers and switching infrastructure.
Each of the multi-mode and single-mode fiber optic cable types includes two wavelengths. The higher wavelengths are used for longer-distance connections.
Indoor vs. Outdoor cabling
Indoor fiber-optic cables are suitable for indoor building applications. Outdoor cables, also known as outside plant
or OSP, are suitable for outdoor applications and are water (liquid and frozen) and ultra-violet resistant.
Indoor/outdoor cables provide the protections of outdoor cables with a fire-retardant jacket that allows deployment
of these cables inside the building entrance beyond the OSP maximum distance, which can reduce the number of transition
splices and connections needed.
|
Mode | Core Diameter | Wavelength | Modal Bandwidth | Cable jacket color |
|---|---|---|---|---|---|
| OM1 | multi-mode | 62.5 µm | 850 nm 1300 nm |
200 MHz | Orange |
| OM2 | multi-mode | 50 µm | 850 nm 1300 nm |
500 MHz | Orange |
| OM3 | multi-mode | 50 µm | 850 nm 1300 nm |
2000 MHz | Aqua |
| OM4 | multi-mode | 50 µm | 850 nm 1300 nm |
4700 MHz | Aqua |
| OS1 | single-mode | 9 µm | 1310 nm 1550 nm |
— | Yellow |
|
OM1 | OM2 | OM3 | OM4 |
|---|---|---|---|---|
| 1 Gb/s | 300m | 500m | 860m | |
| 2 Gb/s | 150m | 300m | 500m | |
| 4 Gb/s | 70m | 150m | 380m | 400m |
| 8 Gb/s | 21m | 50m | 150m | 190m |
| 10 Gb/s | 33m | 82m | 150m ¹ | 190m ¹ |
| 16 Gb/s | 15m ² | 35m ² | 100m ² | 125m ² |
The distances shown above are for 850 nm wavelength multi-mode cables. The 1300 nm wavelength cables can support longer distances.
¹ These are conservative estimates of distances supported for OM3 and OM4 cables at
10Gb/s. Specific implementations may support up to 300m for OM3 at 10Gb/s. In early 2011, some prototype OM4 cable and CFP
connector combinations were shown to support more than 300m at 100Gb/s (10x10).
² OM1 cable is not recommended for 16Gb/s FC, but is expected to operate up to 15m.
These distance estimates are listed in the 16Gb/s FC specification completed in September 2010. When 16Gb FC products become
available, specific implementations may support different distances.
Distances supported in actual configurations are generally less than the distance supported by the raw fiber optic cable.
Active Copper vs. Passive Copper
Passive copper connections are common with many interfaces. The industry is finding that as the transfer rates
increase, passive copper does not provide the distance needed and takes up too much physical space. The industry
is moving towards an active copper type of interface for higher speed connections, such as 6Gb/s SAS. Active
copper connections include components that boost the signal, reduce the noise and work with smaller-gauge
cables, improving signal distance, cable flexibility and airflow. These active copper components are expected
to be less expensive and consume less electric power than the equivalent components used with fiber optic cables.
Copper: 10GBASE-T and 1000BASE-T
1000BASE-T cabling is commonly used for 1Gb Ethernet traffic in general, and 1Gb iSCSI for storage connections.
This is the familiar four pair copper cable with the RJ45 connectors. Cables used for 1000BASE-T are known as
Cat5e (Category 5 enhanced) or Cat6 (Category 6) cables.
10GBASE-T cabling supports 10Gb Ethernet traffic, including 10Gb iSCSI storage traffic. The cables and connectors are similar to, but not the same as the cables used for 1000BASE-T. 10GBASE-T cables are Cat6a (Category 6 augmented), also known as Class EA cables. These support the higher frequencies required for 10Gb transmission up to 100 meters (330 feet). Cables must be certified to at least 500MHz to ensure 10GBASE-T compliance. Cat7 (Category 7, Class F) cable is also certified for 10GBASE-T compliance, and is typically deployed in Europe. Cat6 cables may work in 10GBASE-T deployments up to 55m, but should be tested first. 10GBASE-T cabling is not expected to be deployed for FCoE applications in the near future.
Several types of connectors are available with cables used for storage interfaces. This is not an exhaustive list but is intended to show the more common types. Each of the connector types includes the number of lanes (or channels) and the rated speed.
As of early 2011, the fastest generally available connector speeds supported were 10 Gbps per lane. Significantly higher speeds are currently achieved by bundling multiple lanes in parallel, such as 4x10 (40Gbps), 10x10 (100Gbps), 12x10 (120Gbps), etc. Most of the current implementations of 40GbE and 100GbE use multiple lanes of 10GbE and are considered “channel bonded” solutions.
14 Gbps per lane connectors have begun to appear in the last half of 2011. These connectors support 16Gb Fibre Channel (single-lane) and 56Gb (FDR) Infiniband (multi-lane).
25 Gbps per lane connectors are expected to become available in 2012 or 2013. When 25 Gbps per lane connectors are available, then higher speeds, such as 100 Gbps can be achieved by bundling four of these lanes together. Other variations of bundling multiple lanes of 25 Gbps may be possible, such as 10x25 (250 Gbps), 12x25 (300 Gbps) or 16x25 (400 Gbps). It is expected that the 25Gbps (actually 28Gbps) connectors will support 32Gb Fibre Channel in single-lane configurations and higher speeds for Ethernet and Infiniband in multi-lane configurations.
Note the encoding schemes described above for additional detail on speeds available for various connector and cable combinations.
|
Type | Lanes | Max. Speed per lane (Gbps) | Max. Speed total (Gbps) | Cable type | Usage |
|---|---|---|---|---|---|---|
| mini SAS | SAS | 4 | 6 | 24 | Copper | 3Gb, 6Gb SAS |
| Copper CX4 | CX4 | 4 | 5 | 20 | Copper | 10Gb Ethernet, SDR and DDR Infiniband |
| Small Form-factor Pluggable | SFP | 1 | 4 | 4 | Copper, Optical | 1Gb Ethernet, Fibre Channel: 1, 2, 4Gb |
| Small Form-factor Pluggable enhanced | SFP+ | 1 | 16 | 16 | Copper, Optical | 10Gb Ethernet, 8Gb & 16Gb Fibre Channel, 10Gb FCoE |
| Quad Small Form-factor Pluggable | QSFP | 4 | 5 | 20 | Copper, Optical | Various |
| Quad Small Form-factor Pluggable enhanced | QSFP+ | 4 | 16 | 64 | Copper, Optical | 40Gb Ethernet, DDR, QDR & FDR Infiniband, 64Gb Fibre Channel |
| CXP | CXP | 10, 12 | 10 | 100, 120 | Copper | 100Gb Ethernet, 120Gb other |
| CFP | CFP | 10 | 10 | 100 | Optical | 100Gb Ethernet |
PCIe data rates and connector types are provided in the PCI Express section.
Infiniband Data Rates
SDR: Single Data Rate, DDR: Double Data Rate, QDR: Quad Data Rate, FDR: Fourteen Data Rate
|
Type | Diagram |
|---|---|---|
| mini SAS | SAS |  |
| Copper CX4 | CX4 |  |
| Small Form-factor Pluggable | SFP, SFP+ |  |
| Quad Small Form-factor Pluggable | QSFP, QSFP+ |  |
PCIe connector types are provided in the PCI Express section.
Mini SFP
In the second half of 2010, a new variant of the SFP/SFP+ connector was introduced to accommodate the
Fibre Channel backbone with 64-port blades and the planned increased density Ethernet core switches. This new
connector, known as mSFP, mini-SFP or mini-LC SFP, narrows the optical centerline of a conventional SFP/SFP+
connector from 6.25 mm to 5.25 mm. Although this connector looks very much like a standard SFP style connector,
it is narrower and is required for the higher-density devices. The photo at the right shows the difference
between mini-SFP and the standard size.
CXP and CFP
The CXP (copper) and CFP (optical) connectors are expected to be used initially for switch-to-switch connections.
These are expected for Ethernet and may also be used for Infiniband.
PCI Express, also known as PCIe, stands for Peripheral Component Interconnect Express and is the computer industry standard for the I/O bus for computers introduced in the last few years. The first version of the PCIe specification, 1.0a, was introduced in 2003. Version 2.0 was introduced in 2007 and version 3.0 was introduced in 2010. These versions are often identified by their generation (“gen 1”, “gen 2”, etc.). It can take a year or two between the time the specification is introduced and general availability of computer systems using those specification versions. The PCIe specifications are developed and maintained by the PCI-SIG (PCI Special Interest Group). PCI Express and PCIe are registered trademarks of the PCI-SIG.
Data rates for different versions of PCIe are shown in the table below. PCIe data rates are expressed in Gigatransfers per second (GT/s) and are a function of the number of lanes in the connection. The number of lanes are expressed with an “x” before the number of lanes, and is often spoken as “by 1”, “by 4”, etc. PCIe supports full-duplex (traffic in both directions). The data rates shown below are in each direction. Note the explanation of encoding schemes described above.
|
GT/s | Encoding | x1 | x2 | x4 | x8 | x16 |
|---|---|---|---|---|---|---|---|
| PCIe 1.x | 2.5 | 8b/10b | 250 MB/s | 500 MB/s | 1 GB/s | 2 GB/s | 4 GB/s |
| PCIe 2.x | 5 | 8b/10b | 500 MB/s | 1 GB/s | 2 GB/s | 4 GB/s | 8 GB/s |
| PCIe 3.x | 8 | 128b/130b | 1 GB/s | 2 GB/s | 4 GB/s | 8 GB/s | 16 GB/s |
Efforts are underway to enable SATA and SAS to be carried over PCIe connections. See the roadmaps section above.
In 2008, the PCI-SIG announced the completion of its I/O Virtualization (IOV) suite of specifications including single-root IOV (SR-IOV) and multi-root IOV (MR-IOV). These technologies can work with system virtualization technologies and can allow multiple operating systems to natively share PCIe devices.
The concept of sharing PCIe devices or providing access to PCIe devices that may be physically larger than some smaller form-factor systems can accomodate has led to the development of external connections to some PCIe devices. Cables have been developed for extending the PCIe bus outside of the chassis holding the PCIe slots. These cables are specified by indicating the number of PCIe lanes (x4, x8, etc.) supported. Cables are typically available for x4, x8 and x16 lane configurations. Common cable lengths are 1m and 3m. The photo below shows some PCIe cables and connectors. PCIe can also be carried over fiber-optic cables for longer distances.
