Instead of benchmarking IBoIP as I did in my previous post it’s much more interesting to test pure InfiniBand. There is a so-called OSU Micro-Benchmarks suite from Ohio State. It has several tests: unidirectional bandwidth (bw), bidirectional bandwidth (bibw), unidirectional one-sided get (get_bw), unidirectional one-sided put (put_bw), bidirectional one-sided put (put_bibw) and same latency tests. Here is what I got on my Mellanox ConnectX cards:
As I’ve already mentioned in my previous post called “Activating InfiniBand stack in Linux” there is a perftest package which has simple tests for benchmarking IB bandwidth and latency. Here go my results for default ib_write_bw and ib_write_lat tests. Write, read and send tests results don’t differ much, that’s why I’m posting only write results.
------------------------------------------------------------------
RDMA_Write BW Test
Number of qp's running 1
Connection type : RC
Each Qp will post up to 100 messages each time
Inline data is used up to 0 bytes message
local address: LID 0x04, QPN 0x18004a, PSN 0xcf8a2e
RKey 0x2c042529 VAddr 0x002af439bf2000
remote address: LID 0x01, QPN 0x12004a, PSN 0xb446fe,
RKey 0x440428db VAddr 0x002b46ea9b5000
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
65536 5000 1350.34 1350.27
------------------------------------------------------------------
------------------------------------------------------------------
RDMA_Write Latency Test
Inline data is used up to 400 bytes message
Connection type : RC
local address: LID 0x04 QPN 0x16004a PSN 0x5d05e8
RKey 0x2a042529 VAddr 0x00000017f88002
remote address: LID 0x01 QPN 0x10004a PSN 0xb8cade
RKey 0x420428db VAddr 0x00000000ae2002
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec]
2 1000 1.16 6.93 1.22
------------------------------------------------------------------
Niktips's Blog 