Protocol Layers
The Infiniband protocol architecture is a typical 5-layer protocol (i.e. like Ethernet TCP/IP).
Physical Layer
The physical layer specifies both mechanical and electrical designs, like connectors, cables, fiber or copper receptacles, backplane connectors, hot swap characteristics, etc.
Three link possibilities are defined: 1x 4x and 12x.
These roughly* means that there are 3 different possible cables, the 1x has only 1 pair of TX and 1 pair of RX cables. 4x has 4 pairs of TX cables and 4 pairs of RX cables. 12x has 12 pairs of TX cables and 12 pairs of RX cables.
*I say roughly because an Infiniband cable can be made of copper wires or fiber wires. On fiber wires the assumption above is ok, but it isn’t on electrical ones, where double of lanes are needed. See Infiniband Bandwith Architecture Specification from http://www.infinibandta.org.
On each pair of lanes it is possible to have different data rate signalling that can be categorized as SDR (single data rate), DDR (double data rate), QDR (quadruple data rate), FDR (fourteen data rate), EDR (enhanced data rate) and so on.
- On SDR data is transmitted only on the rising edge of the clock signal.
- On DDR data is transmitted on both of the rising edge and falling edge of the clock signal, doubling the SDR rate.
- On QDR data is transmitted on the rising and falling edge and also on the middle of these cycles, using a secondary clock that is 90º out of phase from the main clock.
- On FDR and EDR the same principle using additional clocks is applied.
| SDR | DDR | QDR | FDR | EDR | |
|---|---|---|---|---|---|
| 1X | 2.5 Gbit/s | 5 Gbit/s | 10 Gbit/s | 14,0625 Gbit/s | 25.78125 Gbit/s |
| 4X | 10 Gbit/s | 20 Gbit/s | 40 Gbit/s | 56,25 Gbit/s | 103,125 Gbit/s |
| 12X | 30 Gbit/s | 60 Gbit/s | 120 Gbit/s | 168,75 Gbit/s | 309,375 Gbit/s |
But this is not the effective data rate because as you know the codification mechanisms add some bits that are not part of the payload of the message. In this case, Infiniband uses, for SDR, DDR and QDR a 8b/10b encoding, while FDR (and FDR-10, a lower data-rate one), and EDR uses a 64/66b encoding.
For this reason, the effective data rate signals, not the signalling data rates, are the following:
| SDR | DDR | QDR | FDR-10 | FDR | EDR | |
|---|---|---|---|---|---|---|
| 1X | 2 Gbit/s | 4 Gbit/s | 8 Gbit/s | 10.3125 Gbit/s | 13.64 Gbit/s | 25 Gbit/s |
| 4X | 8 Gbit/s | 16 Gbit/s | 32 Gbit/s | 41.25 Gbit/s | 54.54 Gbit/s | 100 Gbit/s |
| 12X | 24 Gbit/s | 48 Gbit/s | 96 Gbit/s | 123.75 Gbit/s | 163.64 Gbit/s | 300 Gbit/s |
References:
http://www.cse.ohio-state.edu/~panda/788/papers/1d_IB_Intro_WP_190.pdf
http://en.wikipedia.org/wiki/InfiniBand