<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Liu, <div><br></div><div>This is quite reproducible, I have an identical test bed adjacent to this one that I can use for exclusively for testing (perhaps with some performance tracking to see where we are taking the reduced speed from). We are looking to use Sheepdog on our production system in about 6 weeks for root volume ephemeral storage. While initially we were using corosync and received good numbers, the community really seemed to stress moving to zookeeper due to reliability issues with corosync. </div><div><br></div><div>On that note, what kinds of instability were you experiencing with corosync that you don’t see with zookeeper? Personally I haven’t seen catastrophic failure with corosync, but that could just be due to the low latency IB network we are running it on.</div><div><br></div><div>If you have some ideas on where this poor zookeeper performance might be coming from, I would certainly be interested in hearing them. My thought is that perhaps the speed delta between corosync/zookeeper isn’t as noticeable on 1G (or even 10G) ethernet which could be why this issue hasn’t been raised before. Also, is there some type of testing tool that would allow us to test sheepdog volumes without invoking qemu/kvm to help further isolate these performance issues?</div><div><br></div><div><br></div><div>-ryan</div><div><br></div><div>PS: Here is some more data below, let me know if there is anything specific you would want to see:</div><div><br></div><div> /usr/bin/java -cp /etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/zookeeper.jar:/usr/share/java/slf4j-api-1.7.5.jar:/usr/share/java/slf4j-log4j12-1.7.5.jar -Xms8G -Xmx8G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,ROLLINGFILE org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg</div><div><br></div><div><div>root@cs161-p:~# cat /etc/zookeeper/conf/zoo.cfg </div><div># <a href="http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html">http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html</a></div><div><br></div><div># The number of milliseconds of each tick</div><div>tickTime=2000</div><div># The number of ticks that the initial </div><div># synchronization phase can take</div><div>initLimit=10</div><div># The number of ticks that can pass between </div><div># sending a request and getting an acknowledgement</div><div>syncLimit=5</div><div># the directory where the snapshot is stored.</div><div>dataDir=/var/lib/zookeeper</div><div># Place the dataLogDir to a separate physical disc for better performance</div><div>dataLogDir=/zookeeper</div><div><br></div><div># the port at which the clients will connect</div><div>clientPort=2181</div><div><br></div><div># specify all zookeeper servers</div><div># The fist port is used by followers to connect to the leader</div><div># The second one is used for leader election</div><div>server.1=cs161-i:2888:3888</div><div>server.2=cs162-i:2888:3888</div><div>server.3=cs163-i:2888:3888</div><div>server.4=cs164-i:2888:3888</div><div>server.5=cs165-i:2888:3888</div><div><br></div><div># To avoid seeks ZooKeeper allocates space in the transaction log file in</div><div># blocks of preAllocSize kilobytes. The default block size is 64M. One reason</div><div># for changing the size of the blocks is to reduce the block size if snapshots</div><div># are taken more often. (Also, see snapCount).</div><div>#preAllocSize=65536</div><div><br></div><div># Clients can submit requests faster than ZooKeeper can process them,</div><div># especially if there are a lot of clients. To prevent ZooKeeper from running</div><div># out of memory due to queued requests, ZooKeeper will throttle clients so that</div><div># there is no more than globalOutstandingLimit outstanding requests in the</div><div># system. The default limit is 1,000.ZooKeeper logs transactions to a</div><div># transaction log. After snapCount transactions are written to a log file a</div><div># snapshot is started and a new transaction log file is started. The default</div><div># snapCount is 10,000.</div><div>#snapCount=1000</div><div><br></div><div># If this option is defined, requests will be will logged to a trace file named</div><div># traceFile.year.month.day. </div><div>#traceFile=</div><div><br></div><div># Leader accepts client connections. Default value is "yes". The leader machine</div><div># coordinates updates. For higher update throughput at thes slight expense of</div><div># read throughput the leader can be configured to not accept clients and focus</div><div># on coordination.</div><div>#leaderServes=yes</div></div><div><br></div><div>-----</div><div><br></div><div><div>root@cs141-p:~# strace sheep -n -c zookeeper:172.21.5.161:2181,172.21.5.162:2181,172.21.5.163:2181,172.21.5.164:2181,172.21.5.165:2181 /meta /var/lib/sheepdog/disc0,/var/lib/sheepdog/disc1,/var/lib/sheepdog/disc2,/var/lib/sheepdog/disc3,/var/lib/sheepdog/disc4,/var/lib/sheepdog/disc5,/var/lib/sheepdog/disc6,/var/lib/sheepdog/disc7,/var/lib/sheepdog/disc8,/var/lib/sheepdog/disc9,/var/lib/sheepdog/disc10,/var/lib/sheepdog/disc11,/var/lib/sheepdog/disc12,/var/lib/sheepdog/disc13</div><div>execve("/usr/sbin/sheep", ["sheep", "-n", "-c", "zookeeper:172.21.5.161:2181,172."..., "/meta", "/var/lib/sheepdog/disc0,/var/lib"...], [/* 20 vars */]) = 0</div><div>brk(0) = 0x139f000</div><div>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)</div><div>mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0db56c000</div><div>access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)</div><div>open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3</div><div>fstat(3, {st_mode=S_IFREG|0644, st_size=20859, ...}) = 0</div><div>mmap(NULL, 20859, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc0db566000</div><div>close(3) = 0</div><div>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)</div><div>open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3</div><div>read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360l\0\0\0\0\0\0"..., 832) = 832</div><div>fstat(3, {st_mode=S_IFREG|0755, st_size=135757, ...}) = 0</div><div>mmap(NULL, 2212936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0db12f000</div><div>mprotect(0x7fc0db146000, 2097152, PROT_NONE) = 0</div><div>mmap(0x7fc0db346000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fc0db346000</div><div>mmap(0x7fc0db348000, 13384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fc0db348000</div><div>close(3) = 0</div><div>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)</div><div>open("/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3</div><div>read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0T\0\0\0\0\0\0"..., 832) = 832</div><div>fstat(3, {st_mode=S_IFREG|0644, st_size=1063328, ...}) = 0</div><div>mmap(NULL, 3158344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0dae2b000</div><div>mprotect(0x7fc0daf2e000, 2093056, PROT_NONE) = 0</div><div>mmap(0x7fc0db12d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x102000) = 0x7fc0db12d000</div><div>close(3) = 0</div><div>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)</div><div>open("/usr/lib/libcpg.so.4", O_RDONLY|O_CLOEXEC) = 3</div><div>read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\25\0\0\0\0\0\0"..., 832) = 832</div><div>fstat(3, {st_mode=S_IFREG|0644, st_size=18448, ...}) = 0</div><div>mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0db565000</div><div>mmap(NULL, 2113680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0dac26000</div><div>mprotect(0x7fc0dac2a000, 2093056, PROT_NONE) = 0</div><div>mmap(0x7fc0dae29000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7fc0dae29000</div><div>close(3) = 0</div><div>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)</div><div>open("/usr/lib/libcfg.so.6", O_RDONLY|O_CLOEXEC) = 3</div><div>read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\17\0\0\0\0\0\0"..., 832) = 832</div><div>fstat(3, {st_mode=S_IFREG|0644, st_size=14288, ...}) = 0</div><div>mmap(NULL, 2109520, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0daa22000</div><div>mprotect(0x7fc0daa25000, 2093056, PROT_NONE) = 0</div><div>mmap(0x7fc0dac24000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fc0dac24000</div><div>close(3) = 0</div><div>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)</div><div>open("/usr/lib/x86_64-linux-gnu/libzookeeper_mt.so.2", O_RDONLY|O_CLOEXEC) = 3</div><div>read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000?\0\0\0\0\0\0"..., 832) = 832</div><div>fstat(3, {st_mode=S_IFREG|0644, st_size=105288, ...}) = 0</div><div>mmap(NULL, 2200608, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0da808000</div><div>mprotect(0x7fc0da821000, 2093056, PROT_NONE) = 0</div><div>mmap(0x7fc0daa20000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7fc0daa20000</div><div>close(3) = 0</div><div>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)</div><div>open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3</div><div>read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\36\2\0\0\0\0\0"..., 832) = 832</div><div>fstat(3, {st_mode=S_IFREG|0755, st_size=1853400, ...}) = 0</div><div>mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0db564000</div><div>mmap(NULL, 3961912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0da440000</div><div>mprotect(0x7fc0da5fd000, 2097152, PROT_NONE) = 0</div><div>mmap(0x7fc0da7fd000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bd000) = 0x7fc0da7fd000</div><div>mmap(0x7fc0da803000, 17464, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fc0da803000</div><div>close(3) = 0</div><div>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)</div><div>open("/usr/lib/libcorosync_common.so.4", O_RDONLY|O_CLOEXEC) = 3</div><div>read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\6\0\0\0\0\0\0"..., 832) = 832</div><div>fstat(3, {st_mode=S_IFREG|0644, st_size=10056, ...}) = 0</div><div>mmap(NULL, 2105360, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0da23d000</div><div>mprotect(0x7fc0da23e000, 2097152, PROT_NONE) = 0</div><div>mmap(0x7fc0da43e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7fc0da43e000</div><div>close(3) = 0</div><div>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)</div><div>open("/usr/lib/libqb.so.0", O_RDONLY|O_CLOEXEC) = 3</div><div>read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\224\0\0\0\0\0\0"..., 832) = 832</div><div>fstat(3, {st_mode=S_IFREG|0644, st_size=146616, ...}) = 0</div><div>mmap(NULL, 2507680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0d9fd8000</div><div>mprotect(0x7fc0d9ff9000, 2097152, PROT_NONE) = 0</div><div>mmap(0x7fc0da1f9000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7fc0da1f9000</div><div>mmap(0x7fc0da1fc000, 263072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fc0da1fc000</div><div>close(3) = 0</div><div>mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0db563000</div><div>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)</div><div>open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3</div><div>read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., 832) = 832</div><div>fstat(3, {st_mode=S_IFREG|0644, st_size=14664, ...}) = 0</div><div>mmap(NULL, 2109736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0d9dd4000</div><div>mprotect(0x7fc0d9dd7000, 2093056, PROT_NONE) = 0</div><div>mmap(0x7fc0d9fd6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fc0d9fd6000</div><div>close(3) = 0</div><div>access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)</div><div>open("/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3</div><div>read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0#\0\0\0\0\0\0"..., 832) = 832</div><div>fstat(3, {st_mode=S_IFREG|0644, st_size=31760, ...}) = 0</div><div>mmap(NULL, 2128984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0d9bcc000</div><div>mprotect(0x7fc0d9bd3000, 2093056, PROT_NONE) = 0</div><div>mmap(0x7fc0d9dd2000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7fc0d9dd2000</div><div>close(3) = 0</div><div>mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0db562000</div><div>mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0db55f000</div><div>arch_prctl(ARCH_SET_FS, 0x7fc0db560ec0) = 0</div><div>mprotect(0x7fc0da7fd000, 16384, PROT_READ) = 0</div><div>mprotect(0x7fc0db346000, 4096, PROT_READ) = 0</div><div>mprotect(0x7fc0d9dd2000, 4096, PROT_READ) = 0</div><div>mprotect(0x7fc0d9fd6000, 4096, PROT_READ) = 0</div><div>mprotect(0x7fc0da1f9000, 4096, PROT_READ) = 0</div><div>mprotect(0x7fc0da43e000, 4096, PROT_READ) = 0</div><div>mprotect(0x7fc0db12d000, 4096, PROT_READ) = 0</div><div>mprotect(0x7fc0daa20000, 4096, PROT_READ) = 0</div><div>mprotect(0x7fc0dac24000, 4096, PROT_READ) = 0</div><div>mprotect(0x7fc0dae29000, 4096, PROT_READ) = 0</div><div>mprotect(0x648000, 4096, PROT_READ) = 0</div><div>mprotect(0x7fc0db56e000, 4096, PROT_READ) = 0</div><div>munmap(0x7fc0db566000, 20859) = 0</div><div>set_tid_address(0x7fc0db561190) = 13362</div><div>set_robust_list(0x7fc0db5611a0, 24) = 0</div><div>futex(0x7fff129e212c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7fc0db560ec0) = -1 EAGAIN (Resource temporarily unavailable)</div><div>rt_sigaction(SIGRTMIN, {0x7fc0db1357e0, [], SA_RESTORER|SA_SIGINFO, 0x7fc0db13ebb0}, NULL, 8) = 0</div><div>rt_sigaction(SIGRT_1, {0x7fc0db135860, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fc0db13ebb0}, NULL, 8) = 0</div><div>rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0</div><div>getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0</div><div>brk(0) = 0x139f000</div><div>brk(0x13c0000) = 0x13c0000</div><div>rt_sigaction(SIGSEGV, {0x405e80, [], SA_RESTORER|SA_NODEFER|SA_RESETHAND, 0x7fc0db13ebb0}, NULL, 8) = 0</div><div>rt_sigaction(SIGABRT, {0x405e80, [], SA_RESTORER|SA_NODEFER|SA_RESETHAND, 0x7fc0db13ebb0}, NULL, 8) = 0</div><div>rt_sigaction(SIGBUS, {0x405e80, [], SA_RESTORER|SA_NODEFER|SA_RESETHAND, 0x7fc0db13ebb0}, NULL, 8) = 0</div><div>rt_sigaction(SIGILL, {0x405e80, [], SA_RESTORER|SA_NODEFER|SA_RESETHAND, 0x7fc0db13ebb0}, NULL, 8) = 0</div><div>rt_sigaction(SIGFPE, {0x405e80, [], SA_RESTORER|SA_NODEFER|SA_RESETHAND, 0x7fc0db13ebb0}, NULL, 8) = 0</div><div>rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7fc0da476ff0}, {SIG_DFL, [], 0}, 8) = 0</div><div>rt_sigaction(SIGHUP, {0x405e60, [], SA_RESTORER, 0x7fc0db13ebb0}, NULL, 8) = 0</div><div>mkdir("/meta", 0770) = -1 EEXIST (File exists)</div><div>stat("/meta", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0</div><div>lstat("/meta", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0</div><div>pipe([3, 4]) = 0</div><div>clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fc0db561190) = 13363</div><div>read(3, "\0\0\0\0", 4) = 4</div><div>--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=13363, si_status=0, si_utime=0, si_stime=0} ---</div><div>exit_group(0) = ?</div><div>+++ exited with 0 +++</div></div><div><br></div><div><div>-rw-r--r-- 1 root root 450K Feb 18 15:55 corosync_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 1.7M Feb 18 15:55 corosync-dbg_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 178K Feb 18 15:55 corosync-dev_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 184K Feb 18 15:55 libcfg6_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 184K Feb 18 15:55 libcfg-dev_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 186K Feb 18 15:55 libcmap4_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 228K Feb 18 15:55 libcmap-dev_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 178K Feb 18 15:55 libcorosync4_2.3.3-0ubuntu4_all.deb</div><div>-rw-r--r-- 1 root root 181K Feb 18 15:55 libcorosync-common4_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 183K Feb 18 15:55 libcorosync-common-dev_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 178K Feb 18 15:55 libcorosync-dev_2.3.3-0ubuntu4_all.deb</div><div>-rw-r--r-- 1 root root 187K Feb 18 15:55 libcpg4_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 231K Feb 18 15:55 libcpg-dev_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 182K Feb 18 15:55 libquorum5_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 201K Feb 18 15:55 libquorum-dev_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 188K Feb 18 15:55 libsam4_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 220K Feb 18 15:55 libsam-dev_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 243K Feb 18 15:55 libtotem-pg5_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 257K Feb 18 15:55 libtotem-pg-dev_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 184K Feb 18 15:55 libvotequorum6_2.3.3-0ubuntu4_amd64.deb</div><div>-rw-r--r-- 1 root root 212K Feb 18 15:55 libvotequorum-dev_2.3.3-0ubuntu4_amd64.deb</div></div><div><br></div><div><br></div><div><br></div><div><div><div>On Mar 14, 2014, at 9:58 AM, Liu Yuan <<a href="mailto:namei.unix@gmail.com">namei.unix@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On Fri, Mar 14, 2014 at 02:43:59PM +0000, Aydelott, Ryan M. wrote:<br><blockquote type="cite">Interesting note on the data management, I have not dug much into the internals with Sheepdog yet - but the only possible explanation would be if any metadata on file object positioning was being retrieved through the backend.<br></blockquote><br>Each volume has only 4MB meta object that track the allocatin bitmap for objects<br>mainly for thin provision. We don't record any placement meta data at all,<br>instead we 100% rely on consistent hash for placement of all the objects, resulting<br>in astonishing load and space balance. We have run 150 nodes with 12 disks on<br>each node and confirmed the evenly distributed load and space. As you noticed,<br>we don't have meta servers.<br><br><blockquote type="cite"><br>I agree with your statements as the IO on the zookeeper nodes is very small/nowhere near the data we are pushing throughout the sheepd members.<br><br>Sheepdog is being started as follows for each test iteration:<br><br>Corosync: sheep -n -c corosync:172.21.5.0 /meta,/var/lib/sheepdog/disc0,/var/lib/sheepdog/disc1,/var/lib/sheepdog/disc2,/var/lib/sheepdog/disc3,/var/lib/sheepdog/disc4,/var/lib/sheepdog/disc5,/var/lib/sheepdog/disc6,/var/lib/sheepdog/disc7,/var/lib/sheepdog/disc8,/var/lib/sheepdog/disc9,/var/lib/sheepdog/disc10,/var/lib/sheepdog/disc11,/var/lib/sheepdog/disc12,/var/lib/sheepdog/disc13<br><br>Zookeeper: sheep -n -c zookeeper:172.21.5.161:2181,172.21.5.162:2181,172.21.5.163:2181,172.21.5.164:2181,172.21.5.165:2181 /meta /var/lib/sheepdog/disc0,/var/lib/sheepdog/disc1,/var/lib/sheepdog/disc2,/var/lib/sheepdog/disc3,/var/lib/sheepdog/disc4,/var/lib/sheepdog/disc5,/var/lib/sheepdog/disc6,/var/lib/sheepdog/disc7,/var/lib/sheepdog/disc8,/var/lib/sheepdog/disc9,/var/lib/sheepdog/disc10,/var/lib/sheepdog/disc11,/var/lib/sheepdog/disc12,/var/lib/sheepdog/disc13<br><br>The driver we wrote/use is: <a href="https://github.com/devoid/nova/tree/sheepdog-nova-support-havana">https://github.com/devoid/nova/tree/sheepdog-nova-support-havana</a><br><br>Which builds out libvirt.xml as follows:<span class="Apple-converted-space"> </span><br><br> <disk type="network" device="disk"><br> <driver name="qemu" cache="writethrough"/><br> <source protocol="sheepdog" name="//172.21.5.141:7000/instance_f9dc065b-d05d-47cb-a3e6-b02049f049df_disk"/><br> <target bus="virtio" dev="vda"/><br> </disk><br><br></blockquote><br>Hmm, since you don't use object cache, you might try set 'cache=none' to save<br>one extra internal flush operation in QEMU. But this won't make big difference.<br><br>I have no idea why you see such a huge difference with zookeeper. Wierd. Is this<br>result reproduciable? I guess something wrong with other confiuration or system.<br><br>P.S. I have long wanted someone who can access IB to develop native IB network<br>transport code for sheepdog instead of ip-over-IB.<br><br>Thanks<br>Yuan</div></blockquote></div><br></div></body></html>