On Tue, Nov 8, 2011 at 9:44 PM, MORITA Kazutaka <morita.kazutaka at lab.ntt.co.jp> wrote: > At Tue, 8 Nov 2011 10:56:51 -0500, > Shawn Moore wrote: >> >> > Probably, we need to add support for using different NICs for data >> > I/Os and monitoring. >> >> We currently have 4 1Gb nics bonded together using mode 4 >> (LACP/802.3ad). On another note, I am still doing more testing, but >> it almost looks like the TOTAL cluster speed might be limited to 1Gb >> instead of up to 4Gb (I know I can't get that much in truth though, >> but should be higher than 1Gb). Does anyone have any insight into >> this? We are using enterprise grade switches with two cards (18Gb/s >> fabric). > > How did you measure the total cluster speed? Could your disks be a > bottleneck? What I have been doing is using pssh (parallel ssh) to kick off a script on all nodes at the same time to create a pre-allocated vdi of the same size. Then obtain the time it takes for the operation to complete on each node and then do some math. Below is the script and an example of the data obtained. I did go ahead a re-run my tests with tmpfs instead of true ext4 disks. This did yield quite a performance boost, but seems it actually should have been faster given the usage of ram for the disks. T_START="$(date +%s)" collie vdi create test_$(uname -n) ${1}G -P T_STOP="$(date +%s)" expr ${T_STOP} - ${T_START} This is a run using the disks. This created 9 3GB vdi's. With copies 3, that created 27GB worth of vdi data and 81GB worth of total cluster data. Total cluster speed of 1.46Gb/s. 152 416 sec 153 462 sec 154 394 sec 155 435 sec 156 451 sec 157 483 sec 159 459 sec 160 406 sec 161 486 sec VDI SIZE 3 COPIES 3 TOT sec 3992 AVG sec 443.5555556 27 GB 27648 MB 62.33266533 MB/s 81 GB 82944 MB 186.997996 MB/s 1495.983968 Mb/s 1.460921844 Gb/s This is a run using tmpfs. This created 9 3GB vdi's. With copies 3, that created 27GB worth of vdi data and 81GB worth of total cluster data. Total cluster speed of 5.73Gb/s. 152 80 sec 153 119 sec 154 127 sec 155 118 sec 156 116 sec 157 142 sec 159 80 sec 160 125 sec 161 110 sec VDI SIZE 3 COPIES 3 TOT sec 1017 AVG sec 113 27 GB 27648 MB 244.6725664 MB/s 81 GB 82944 MB 734.0176991 MB/s 5872.141593 Mb/s 5.734513274 Gb/s |