I have two hosts that are connected through RDMA (one is a SmartNIC, the other is the server). How can I know which pair of device/port/gid to use, if for example I want to run ib_send_bw -d <device> -i <port> -x <gid> between them?
For example, here is the output of show_gids for my two hosts:
Host A:
DEV PORT INDEX GID IPv4 VER DEV--- ---- ----- --- ------------ --- ---mlx5_0 1 0 fe80:0000:0000:0000:4ab0:2dff:fe8f:add0 v1 ens2f0np0mlx5_0 1 1 fe80:0000:0000:0000:4ab0:2dff:fe8f:add0 v2 ens2f0np0mlx5_0 1 2 0000:0000:0000:0000:0000:ffff:c0a8:0001 192.168.0.1 v1 ens2f0np0mlx5_0 1 3 0000:0000:0000:0000:0000:ffff:c0a8:0001 192.168.0.1 v2 ens2f0np0mlx5_1 1 0 fe80:0000:0000:0000:4ab0:2dff:fe8f:add1 v1 ens2f1np1mlx5_1 1 1 fe80:0000:0000:0000:4ab0:2dff:fe8f:add1 v2 ens2f1np1mlx5_1 1 2 0000:0000:0000:0000:0000:ffff:c0a8:0101 192.168.1.1 v1 ens2f1np1mlx5_1 1 3 0000:0000:0000:0000:0000:ffff:c0a8:0101 192.168.1.1 v2 ens2f1np1mlx5_2 1 0 fe80:0000:0000:0000:966d:aeff:fe9b:95e0 v1 ens4f0np0mlx5_2 1 1 fe80:0000:0000:0000:966d:aeff:fe9b:95e0 v2 ens4f0np0mlx5_3 1 0 fe80:0000:0000:0000:966d:aeff:fe9b:95e1 v1 ens4f1np1mlx5_3 1 1 fe80:0000:0000:0000:966d:aeff:fe9b:95e1 v2 ens4f1np1mlx5_4 1 0 fe80:0000:0000:0000:1270:fdff:fe86:5c7e v1 ens7f0np0mlx5_4 1 1 fe80:0000:0000:0000:1270:fdff:fe86:5c7e v2 ens7f0np0mlx5_5 1 0 fe80:0000:0000:0000:1270:fdff:fe86:5c7f v1 ens7f1np1mlx5_5 1 1 fe80:0000:0000:0000:1270:fdff:fe86:5c7f v2 ens7f1np1n_gids_found=16Host B:
DEV PORT INDEX GID IPv4 VER DEV--- ---- ----- --- ------------ --- ---mlx5_2 1 0 fe80:0000:0000:0000:0040:abff:febb:0745 v2 enp3s0f0s0mlx5_3 1 0 fe80:0000:0000:0000:0017:2eff:fef9:f1f6 v2 enp3s0f1s0n_gids_found=2