I did this in CentOS 5. So some steps can differ if you work with another flavor of Linux.
Major implementation of InfiniBand stack is called OFED. It’s a collaborative development of several vendors to standardize APIs. Allience HQ site is http://www.openfabrics.org/. You can take vanilla OFED version there. Vendors also release their own tweaked versions of original OFED, like Mellanox OFED.
OFED stack provides several interfaces for underlying hardware:
- First and simplest way of working with InfiniBand is IPoIB. In this case IP stack is put above IB. You don’t need to rewrite your applications while you can utilize high throughput. On the other hand you will kill IB low latencies and won’t be able to utilize whole IB throughput capabilities.
- Second way is Sockets Direct Protocol (SDP) which is designed to utilize IB RDMA capabilities and bypass TCP/IP stack. SDP can be used transparently w/o recompiling your application. It’s not that fast as native IB API but os better than IPoIB.
- Third and hardest way is using IB Verbs which is the lowest API, User Direct Access Programming Library (uDAPL) which is based on IB Verbs, Message Passing Interface (MPI) or Unified Parallel C. Different versions of MPI and UPC can be based on either IB Verbs or uDAPL. I personally work with MPI and UPC so I will describe their installation over InfiniBand.
To bring up your InfiniBand hardware you need to do the following things:
1. Install kernel level and user level HCA drivers. Interesting thing about IB is that driver is split into two parts. Kernel part is usually already compiled as modules. You just need an openibd package which will run as a service upon startup and will load all necessary kernel modules. User level part you should install by your self. Since I’m working with Mellanox ConnectX MT25418 I had to install libmlx4 package.
2. Next thing is Subnet Manager. The InfiniBand subnet manager (OpenSM) assigns Local IDentifiers (LIDs) to each port connected to the InfiniBand fabric, and develops a routing table based off of the assigned LIDs. opensmd package performs Subnet Manager role. You need to set this service up on any one IB node and it will initiate IB fabric upon system startup.
3. Install IB API. Depending on what software you going to use install appropriate API. I use MPI and UPC parallel applications compilers which use IB Verbs. libibverbs package is responsible for that.
Additionally you would probably like to use some diagnostic tools to check IB connectivity. Found them in ibutils and infiniband-diags RPMs:
- ibv_devinfo, ibstat, ibstatus show device information;
- sminfo, ibnodes, ibhosts, ibswitches, ibnetdiscover, ibchecknet show fabric information.
Use perftest package for IB performance testing. Simply run ib_write_bw in server mode on one node by just calling # ib_write_bw and run client on the other node # ib_write_bw node_name. Use ib_write_lat for latency testing.
You will probably also need libibcommon, libibumad and libibmad libraries which is needed for opensm, infinband-diags and ibutils to operate.
Links I find useful:
Niktips's Blog 