[Stgt-devel] Linux Kernel Version

[Tom Tucker]

Tomo:

Thanks for the update. 

[...snip...]

> I made the tgt patch for 2.6.17. Would it help?
> 
> http://svn.berlios.de/svnroot/repos/stgt/trunk/patchset/tgt-2.6.17.patch
> 
> Sorry. There is no instruction about it yet. I will write something
> later on.

Great, thanks. I'm not sure we really need instructions. I'll give it a
whirl and see how it goes.

> 
> As you said, it's complicated to run tgt right now. I've been working
> on it because some people might be interested in tgt after OLS. I want
> to make sure that everyone can run tgt without effort. I'm in the
> middle of large user-space rewrite (I've commited some last night).
> 
> Sorry for delay about iSER tgt stuff. I will work on it after OLS. 

The biggest thing I'd like some guidance on is how we want to plug into
the transport switch. If connection management (not LOGIN, just the 
listen/accept paths), is pushed into the kernel, then the user-mode
code is insulated from the transport switch. This requires pushing the 
current TCP connection management into the kernel though and adding netlink 
messages for listen/accept. 

Otherwise, we need to replicate the switch in user-mode. The TCP tgt can
still use sockets in user mode, but he RDMA side will need to have the 
netlink messages anyway. The issue is that unlike a socket, the RDMA 
user-mode CM doesn't have the ability to have user-mode code and 
kernel-mode code both doing I/O on the transport like a socket. 

I've coded up the kernel-only approach, but haven't gotten much feedback 
on it, so I'm assuming that it's not the preferred method ;-)
>
> If
> you guys teach me how to setup Chelsio T3 at OLS, I can work on it
> right after OLS. :)

Here's a How-To on bringing up the Chelsio code for RDMA

============================================
Setup kernel tree.
============================================

Pull your favorite post 2.6.16.18 kernel tree from kernel.org. 

NOTE: The openib iwarp branch stays up to date with the latest 
kernel.org release.

============================================
Checkout iWARP branch:
============================================
Pull the iwarp branch using svn.  You will need access to this code
on your victim machine.

# svn co https://openib.org/svn/gen2/branches/iwarp


============================================
Apply the notifier patch.
============================================

To support MAC address changes and next hop routing changes, you need
to install the notifier patch.

# cd path_to_kernel
# patch -p1 < ${path_to_iwarp_branch}/src/linux-kernel/patches/cxgb3/notifier.patch

Depending on the kernel you're using you may need to merge this patch.

============================================
Link the iWARP branch into your kernel tree
============================================

# cd ${path_to_kernel}/drivers
# mv infiniband infinband.org
# ln -s ${path_to_iwarp_branch}/src/linux-kernel/infiniband .

# cd ${path_to_kernel}/include
# mv rdma rdma.org
# ln -s path_to_kernel/drivers/infiniband/include/rdma .

Link in the cxgb3 LLD Driver:

# cd ${path_to_kernel}/drivers/net

If there is already a cxgb3 driver there, move it.
# mv cxgb3 cxgb3.org

# ln -sf path_to_iwarp_branch/src/linux-kernel/net/cxgb3 .

Apply the kernel patches to build the cxgb3 LLD Driver:

# cd ${path_to_kernel}
# patch -p0 < ${path_to_iwarp_branch}/src/linux-kernel/patches/cxgb3/cxgb3-Kconfig.patch
# patch -p0 < ${path_to_iwarp_branch}/src/linux-kernel/patches/cxgb3/cxgb3-Makefile.patch

============================================
Configure, build, install, and boot kernel and modules.
============================================

# cd ${path_to_kernel}
# make menuconfig

Device Drivers->Network device support->Ethernet (10000 Mbit)->
<M> Chelsio T3 10Gb Ethernet support

Make sure you turn on the "Chelsio T3" driver and turn OFF 
any other Chelsio net driver that may be present. 

Device Drivers->Infiniband support->
<M> Infiniband support
<M> Infiniband userspace MAD support
<M> Infiniband userspace access (verbs & cm)
<M> Chelsio OpenIB module
<M> Kernel RDMA Ping Module

# make
# make modules_install
# make install
# <edit grub file if needed>
# reboot

============================================
Configure, build, and install userspace libraries
============================================

Here's a link to the OpenFabrics wiki on building userspace
support. You can ignore the stuff in this wiki up to the point
where you build the userspace libraries.

BTW, I have seen issues with autogen.sh on some 64b distros. The symptom 
is a failing configure script with an error like "unable to determine 
sizeof (long)". The work-around is to move the 
AC_CHECK_SIZEOF(long) in the configure.in file up to just below
AC_PROG_CC. 

https://openib.org/tiki/tiki-index.php?page=Installation+Cheat+Sheet

In addition to the libraries in the wiki, you need to
build and install the following:

# cd ${path_to_iwarp_branch}/src/userspace
# (cd librdmacm && ./autogen.sh && ./configure && make && make install)
# (cd libcxgb3 && ./autogen.sh && ./configure && make && make install)

You need device files for kernel modules we just built.  We could add
them later, manually, but udev is more convenient.

Create a new file /etc/udev/rules.d/90-ib.rules:

KERNEL="umad*", NAME="infiniband/%k"
KERNEL="issm*", NAME="infiniband/%k"
KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666"
KERNEL="ucm*", NAME="infiniband/%k", MODE="0666"
KERNEL="rdma_cm", NAME="infiniband/%k", MODE="0666"

Load all the OpenIB Modules:

# modprobe cxgb3
# modprobe iw_cxgb3
# modprobe ib_mad
# modprobe rdma_cm
# modprobe ib_umad
# modprobe ib_uverbs
# modprobe ib_ucm
# modprobe rdma_ucm
# modprobe rdma_krping
# udevstart 2>&1 >/dev/null &

============================================
Test everything with rdma_krping:
============================================

Configure the Ethernet interfaces for your cxgb3 device. 

Then on the server machine:

echo "verbose,server,addr=$server_ip_addr,port=9999" > /proc/krping

And on the client machine:

# echo "verbose,client,addr=$server_ip_addr,port=9999" > /proc/krping

If you cat /proc/krping, you'll see data indicating how many messages, 
bytes, etc...have been sent for the various RDMA message types. 

You can also tail -f /var/log/messages to see messages output there as well.

Hope this helps, and don't hesitate to e-mail or call me (512-343-9196 x 108) 
if you need help.

Tom