Tomo: Thanks for the update. [...snip...] > I made the tgt patch for 2.6.17. Would it help? > > http://svn.berlios.de/svnroot/repos/stgt/trunk/patchset/tgt-2.6.17.patch > > Sorry. There is no instruction about it yet. I will write something > later on. Great, thanks. I'm not sure we really need instructions. I'll give it a whirl and see how it goes. > > As you said, it's complicated to run tgt right now. I've been working > on it because some people might be interested in tgt after OLS. I want > to make sure that everyone can run tgt without effort. I'm in the > middle of large user-space rewrite (I've commited some last night). > > Sorry for delay about iSER tgt stuff. I will work on it after OLS. The biggest thing I'd like some guidance on is how we want to plug into the transport switch. If connection management (not LOGIN, just the listen/accept paths), is pushed into the kernel, then the user-mode code is insulated from the transport switch. This requires pushing the current TCP connection management into the kernel though and adding netlink messages for listen/accept. Otherwise, we need to replicate the switch in user-mode. The TCP tgt can still use sockets in user mode, but he RDMA side will need to have the netlink messages anyway. The issue is that unlike a socket, the RDMA user-mode CM doesn't have the ability to have user-mode code and kernel-mode code both doing I/O on the transport like a socket. I've coded up the kernel-only approach, but haven't gotten much feedback on it, so I'm assuming that it's not the preferred method ;-) > > If > you guys teach me how to setup Chelsio T3 at OLS, I can work on it > right after OLS. :) Here's a How-To on bringing up the Chelsio code for RDMA ============================================ Setup kernel tree. ============================================ Pull your favorite post 2.6.16.18 kernel tree from kernel.org. NOTE: The openib iwarp branch stays up to date with the latest kernel.org release. ============================================ Checkout iWARP branch: ============================================ Pull the iwarp branch using svn. You will need access to this code on your victim machine. # svn co https://openib.org/svn/gen2/branches/iwarp ============================================ Apply the notifier patch. ============================================ To support MAC address changes and next hop routing changes, you need to install the notifier patch. # cd path_to_kernel # patch -p1 < ${path_to_iwarp_branch}/src/linux-kernel/patches/cxgb3/notifier.patch Depending on the kernel you're using you may need to merge this patch. ============================================ Link the iWARP branch into your kernel tree ============================================ # cd ${path_to_kernel}/drivers # mv infiniband infinband.org # ln -s ${path_to_iwarp_branch}/src/linux-kernel/infiniband . # cd ${path_to_kernel}/include # mv rdma rdma.org # ln -s path_to_kernel/drivers/infiniband/include/rdma . Link in the cxgb3 LLD Driver: # cd ${path_to_kernel}/drivers/net If there is already a cxgb3 driver there, move it. # mv cxgb3 cxgb3.org # ln -sf path_to_iwarp_branch/src/linux-kernel/net/cxgb3 . Apply the kernel patches to build the cxgb3 LLD Driver: # cd ${path_to_kernel} # patch -p0 < ${path_to_iwarp_branch}/src/linux-kernel/patches/cxgb3/cxgb3-Kconfig.patch # patch -p0 < ${path_to_iwarp_branch}/src/linux-kernel/patches/cxgb3/cxgb3-Makefile.patch ============================================ Configure, build, install, and boot kernel and modules. ============================================ # cd ${path_to_kernel} # make menuconfig Device Drivers->Network device support->Ethernet (10000 Mbit)-> <M> Chelsio T3 10Gb Ethernet support Make sure you turn on the "Chelsio T3" driver and turn OFF any other Chelsio net driver that may be present. Device Drivers->Infiniband support-> <M> Infiniband support <M> Infiniband userspace MAD support <M> Infiniband userspace access (verbs & cm) <M> Chelsio OpenIB module <M> Kernel RDMA Ping Module # make # make modules_install # make install # <edit grub file if needed> # reboot ============================================ Configure, build, and install userspace libraries ============================================ Here's a link to the OpenFabrics wiki on building userspace support. You can ignore the stuff in this wiki up to the point where you build the userspace libraries. BTW, I have seen issues with autogen.sh on some 64b distros. The symptom is a failing configure script with an error like "unable to determine sizeof (long)". The work-around is to move the AC_CHECK_SIZEOF(long) in the configure.in file up to just below AC_PROG_CC. https://openib.org/tiki/tiki-index.php?page=Installation+Cheat+Sheet In addition to the libraries in the wiki, you need to build and install the following: # cd ${path_to_iwarp_branch}/src/userspace # (cd librdmacm && ./autogen.sh && ./configure && make && make install) # (cd libcxgb3 && ./autogen.sh && ./configure && make && make install) You need device files for kernel modules we just built. We could add them later, manually, but udev is more convenient. Create a new file /etc/udev/rules.d/90-ib.rules: KERNEL="umad*", NAME="infiniband/%k" KERNEL="issm*", NAME="infiniband/%k" KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666" KERNEL="ucm*", NAME="infiniband/%k", MODE="0666" KERNEL="rdma_cm", NAME="infiniband/%k", MODE="0666" Load all the OpenIB Modules: # modprobe cxgb3 # modprobe iw_cxgb3 # modprobe ib_mad # modprobe rdma_cm # modprobe ib_umad # modprobe ib_uverbs # modprobe ib_ucm # modprobe rdma_ucm # modprobe rdma_krping # udevstart 2>&1 >/dev/null & ============================================ Test everything with rdma_krping: ============================================ Configure the Ethernet interfaces for your cxgb3 device. Then on the server machine: echo "verbose,server,addr=$server_ip_addr,port=9999" > /proc/krping And on the client machine: # echo "verbose,client,addr=$server_ip_addr,port=9999" > /proc/krping If you cat /proc/krping, you'll see data indicating how many messages, bytes, etc...have been sent for the various RDMA message types. You can also tail -f /var/log/messages to see messages output there as well. Hope this helps, and don't hesitate to e-mail or call me (512-343-9196 x 108) if you need help. Tom |