Chris Worley wrote: > I've got disks and IB that should be getting many GB/s performance, and I only see a > fraction of that. Any hints on how to configure this to get decent performance? Lets take this slowly to narrow down where your problems are, I would suggest the following path: 1. make sure you have good IB performance (e.g 1.9 GB/s in case you have DDR HW) on your setup using some network/IB benchmark --> for native verbs BW use qperf(8) rc_bw test and for IPoIB/TCP BW use netperf TCP_STREAM 2. make sure you have good disk performance (e.g 800 GB/s as you reported) - using IO benchmark run from the target side --> use whatever benchmark / IO scheduler that works well for you, just make note what you were using 3. make sure you have good iscsi performance over tcp and iser using stgt NULL device --> use something like the below script and in the initiator side use iscsiadm(8) e.g iscsiadm -m node --op show to see what iscsi params are being used by the session/connection On the initiator side, you can use either the distro provided iscsi/iser modules (better) or the ofed ones. On the target side best if you use stgt latest or latest release (0.9.5) from git://git.kernel.org/pub/scm/linux/kernel/git/tomo/tgt.git note you have to $ make ISCSI=1 ISCSI_RDMA=1 for having iscsi and iser support Please note that the open-iscsi initiator mailing list is open-iscsi at googlegroups.com so initiator related questions/issues should be posted there. 4. see what does it take to have see good performance with non NULL device, where assuming you are talking on bandwidth with large IOs I assume the rdwr bs (backing store) should be fine, and in case you are interested in IOPS - you should do better with the sg and/or the aio bs --> we'll see, lets first get 1-3 to work Or. TGTADM=./tgtadm # with iser patch lld should be iser #TRANS=iser #without patch lld must be iscsi TRANS=iscsi TID=1 TARGET_NAME=my.target $TGTADM --lld $TRANS --mode target --op new --tid $TID --targetname $TARGET_NAME # NULL backing store $TGTADM --lld $TRANS --mode logicalunit --op new --tid=$TID --lun=1 --backing-store=/tmp --bstype=null # SG backing-store #$TGTADM --lld $TRANS --mode logicalunit --op new --tid=$TID --lun=1 --backing-store=/dev/sg7 --bstype=sg # synchronous rdwr with block device backing store #$TGTADM --lld $TRANS --mode logicalunit --op new --tid=$TID --lun=1 --backing-store=/dev/sdh --bstype=rdwr # AIO with block device backing store #$TGTADM --lld $TRANS --mode logicalunit --op new --tid=$TID --lun=1 --backing-store=/dev/sdXXX --bstype=aio $TGTADM --lld $TRANS --mode target --op bind --tid=$TID --initiator-address=ALL $TGTADM --lld $TRANS --mode target --op show -- To unsubscribe from this list: send the line "unsubscribe stgt" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html |