[sheepdog-users] Corosync Vs Zookeeper Backends
Aydelott, Ryan M.
ryade at mcs.anl.gov
Fri Mar 14 21:16:55 CET 2014
Liu,
This is quite reproducible, I have an identical test bed adjacent to this one that I can use for exclusively for testing (perhaps with some performance tracking to see where we are taking the reduced speed from). We are looking to use Sheepdog on our production system in about 6 weeks for root volume ephemeral storage. While initially we were using corosync and received good numbers, the community really seemed to stress moving to zookeeper due to reliability issues with corosync.
On that note, what kinds of instability were you experiencing with corosync that you don’t see with zookeeper? Personally I haven’t seen catastrophic failure with corosync, but that could just be due to the low latency IB network we are running it on.
If you have some ideas on where this poor zookeeper performance might be coming from, I would certainly be interested in hearing them. My thought is that perhaps the speed delta between corosync/zookeeper isn’t as noticeable on 1G (or even 10G) ethernet which could be why this issue hasn’t been raised before. Also, is there some type of testing tool that would allow us to test sheepdog volumes without invoking qemu/kvm to help further isolate these performance issues?
-ryan
PS: Here is some more data below, let me know if there is anything specific you would want to see:
/usr/bin/java -cp /etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/zookeeper.jar:/usr/share/java/slf4j-api-1.7.5.jar:/usr/share/java/slf4j-log4j12-1.7.5.jar -Xms8G -Xmx8G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,ROLLINGFILE org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg
root at cs161-p:~# cat /etc/zookeeper/conf/zoo.cfg
# http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/var/lib/zookeeper
# Place the dataLogDir to a separate physical disc for better performance
dataLogDir=/zookeeper
# the port at which the clients will connect
clientPort=2181
# specify all zookeeper servers
# The fist port is used by followers to connect to the leader
# The second one is used for leader election
server.1=cs161-i:2888:3888
server.2=cs162-i:2888:3888
server.3=cs163-i:2888:3888
server.4=cs164-i:2888:3888
server.5=cs165-i:2888:3888
# To avoid seeks ZooKeeper allocates space in the transaction log file in
# blocks of preAllocSize kilobytes. The default block size is 64M. One reason
# for changing the size of the blocks is to reduce the block size if snapshots
# are taken more often. (Also, see snapCount).
#preAllocSize=65536
# Clients can submit requests faster than ZooKeeper can process them,
# especially if there are a lot of clients. To prevent ZooKeeper from running
# out of memory due to queued requests, ZooKeeper will throttle clients so that
# there is no more than globalOutstandingLimit outstanding requests in the
# system. The default limit is 1,000.ZooKeeper logs transactions to a
# transaction log. After snapCount transactions are written to a log file a
# snapshot is started and a new transaction log file is started. The default
# snapCount is 10,000.
#snapCount=1000
# If this option is defined, requests will be will logged to a trace file named
# traceFile.year.month.day.
#traceFile=
# Leader accepts client connections. Default value is "yes". The leader machine
# coordinates updates. For higher update throughput at thes slight expense of
# read throughput the leader can be configured to not accept clients and focus
# on coordination.
#leaderServes=yes
-----
root at cs141-p:~# strace sheep -n -c zookeeper:172.21.5.161:2181,172.21.5.162:2181,172.21.5.163:2181,172.21.5.164:2181,172.21.5.165:2181 /meta /var/lib/sheepdog/disc0,/var/lib/sheepdog/disc1,/var/lib/sheepdog/disc2,/var/lib/sheepdog/disc3,/var/lib/sheepdog/disc4,/var/lib/sheepdog/disc5,/var/lib/sheepdog/disc6,/var/lib/sheepdog/disc7,/var/lib/sheepdog/disc8,/var/lib/sheepdog/disc9,/var/lib/sheepdog/disc10,/var/lib/sheepdog/disc11,/var/lib/sheepdog/disc12,/var/lib/sheepdog/disc13
execve("/usr/sbin/sheep", ["sheep", "-n", "-c", "zookeeper:172.21.5.161:2181,172."..., "/meta", "/var/lib/sheepdog/disc0,/var/lib"...], [/* 20 vars */]) = 0
brk(0) = 0x139f000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0db56c000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=20859, ...}) = 0
mmap(NULL, 20859, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc0db566000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360l\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=135757, ...}) = 0
mmap(NULL, 2212936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0db12f000
mprotect(0x7fc0db146000, 2097152, PROT_NONE) = 0
mmap(0x7fc0db346000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fc0db346000
mmap(0x7fc0db348000, 13384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fc0db348000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0T\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=1063328, ...}) = 0
mmap(NULL, 3158344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0dae2b000
mprotect(0x7fc0daf2e000, 2093056, PROT_NONE) = 0
mmap(0x7fc0db12d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x102000) = 0x7fc0db12d000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libcpg.so.4", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\25\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=18448, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0db565000
mmap(NULL, 2113680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0dac26000
mprotect(0x7fc0dac2a000, 2093056, PROT_NONE) = 0
mmap(0x7fc0dae29000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7fc0dae29000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libcfg.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\17\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14288, ...}) = 0
mmap(NULL, 2109520, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0daa22000
mprotect(0x7fc0daa25000, 2093056, PROT_NONE) = 0
mmap(0x7fc0dac24000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fc0dac24000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libzookeeper_mt.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000?\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=105288, ...}) = 0
mmap(NULL, 2200608, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0da808000
mprotect(0x7fc0da821000, 2093056, PROT_NONE) = 0
mmap(0x7fc0daa20000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7fc0daa20000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\36\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1853400, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0db564000
mmap(NULL, 3961912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0da440000
mprotect(0x7fc0da5fd000, 2097152, PROT_NONE) = 0
mmap(0x7fc0da7fd000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bd000) = 0x7fc0da7fd000
mmap(0x7fc0da803000, 17464, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fc0da803000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libcorosync_common.so.4", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\6\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=10056, ...}) = 0
mmap(NULL, 2105360, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0da23d000
mprotect(0x7fc0da23e000, 2097152, PROT_NONE) = 0
mmap(0x7fc0da43e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7fc0da43e000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/libqb.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\224\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=146616, ...}) = 0
mmap(NULL, 2507680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0d9fd8000
mprotect(0x7fc0d9ff9000, 2097152, PROT_NONE) = 0
mmap(0x7fc0da1f9000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7fc0da1f9000
mmap(0x7fc0da1fc000, 263072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fc0da1fc000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0db563000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14664, ...}) = 0
mmap(NULL, 2109736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0d9dd4000
mprotect(0x7fc0d9dd7000, 2093056, PROT_NONE) = 0
mmap(0x7fc0d9fd6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fc0d9fd6000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0#\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=31760, ...}) = 0
mmap(NULL, 2128984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc0d9bcc000
mprotect(0x7fc0d9bd3000, 2093056, PROT_NONE) = 0
mmap(0x7fc0d9dd2000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7fc0d9dd2000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0db562000
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0db55f000
arch_prctl(ARCH_SET_FS, 0x7fc0db560ec0) = 0
mprotect(0x7fc0da7fd000, 16384, PROT_READ) = 0
mprotect(0x7fc0db346000, 4096, PROT_READ) = 0
mprotect(0x7fc0d9dd2000, 4096, PROT_READ) = 0
mprotect(0x7fc0d9fd6000, 4096, PROT_READ) = 0
mprotect(0x7fc0da1f9000, 4096, PROT_READ) = 0
mprotect(0x7fc0da43e000, 4096, PROT_READ) = 0
mprotect(0x7fc0db12d000, 4096, PROT_READ) = 0
mprotect(0x7fc0daa20000, 4096, PROT_READ) = 0
mprotect(0x7fc0dac24000, 4096, PROT_READ) = 0
mprotect(0x7fc0dae29000, 4096, PROT_READ) = 0
mprotect(0x648000, 4096, PROT_READ) = 0
mprotect(0x7fc0db56e000, 4096, PROT_READ) = 0
munmap(0x7fc0db566000, 20859) = 0
set_tid_address(0x7fc0db561190) = 13362
set_robust_list(0x7fc0db5611a0, 24) = 0
futex(0x7fff129e212c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7fc0db560ec0) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigaction(SIGRTMIN, {0x7fc0db1357e0, [], SA_RESTORER|SA_SIGINFO, 0x7fc0db13ebb0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7fc0db135860, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fc0db13ebb0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(0) = 0x139f000
brk(0x13c0000) = 0x13c0000
rt_sigaction(SIGSEGV, {0x405e80, [], SA_RESTORER|SA_NODEFER|SA_RESETHAND, 0x7fc0db13ebb0}, NULL, 8) = 0
rt_sigaction(SIGABRT, {0x405e80, [], SA_RESTORER|SA_NODEFER|SA_RESETHAND, 0x7fc0db13ebb0}, NULL, 8) = 0
rt_sigaction(SIGBUS, {0x405e80, [], SA_RESTORER|SA_NODEFER|SA_RESETHAND, 0x7fc0db13ebb0}, NULL, 8) = 0
rt_sigaction(SIGILL, {0x405e80, [], SA_RESTORER|SA_NODEFER|SA_RESETHAND, 0x7fc0db13ebb0}, NULL, 8) = 0
rt_sigaction(SIGFPE, {0x405e80, [], SA_RESTORER|SA_NODEFER|SA_RESETHAND, 0x7fc0db13ebb0}, NULL, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7fc0da476ff0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGHUP, {0x405e60, [], SA_RESTORER, 0x7fc0db13ebb0}, NULL, 8) = 0
mkdir("/meta", 0770) = -1 EEXIST (File exists)
stat("/meta", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/meta", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fc0db561190) = 13363
read(3, "\0\0\0\0", 4) = 4
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=13363, si_status=0, si_utime=0, si_stime=0} ---
exit_group(0) = ?
+++ exited with 0 +++
-rw-r--r-- 1 root root 450K Feb 18 15:55 corosync_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 1.7M Feb 18 15:55 corosync-dbg_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 178K Feb 18 15:55 corosync-dev_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 184K Feb 18 15:55 libcfg6_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 184K Feb 18 15:55 libcfg-dev_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 186K Feb 18 15:55 libcmap4_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 228K Feb 18 15:55 libcmap-dev_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 178K Feb 18 15:55 libcorosync4_2.3.3-0ubuntu4_all.deb
-rw-r--r-- 1 root root 181K Feb 18 15:55 libcorosync-common4_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 183K Feb 18 15:55 libcorosync-common-dev_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 178K Feb 18 15:55 libcorosync-dev_2.3.3-0ubuntu4_all.deb
-rw-r--r-- 1 root root 187K Feb 18 15:55 libcpg4_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 231K Feb 18 15:55 libcpg-dev_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 182K Feb 18 15:55 libquorum5_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 201K Feb 18 15:55 libquorum-dev_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 188K Feb 18 15:55 libsam4_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 220K Feb 18 15:55 libsam-dev_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 243K Feb 18 15:55 libtotem-pg5_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 257K Feb 18 15:55 libtotem-pg-dev_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 184K Feb 18 15:55 libvotequorum6_2.3.3-0ubuntu4_amd64.deb
-rw-r--r-- 1 root root 212K Feb 18 15:55 libvotequorum-dev_2.3.3-0ubuntu4_amd64.deb
On Mar 14, 2014, at 9:58 AM, Liu Yuan <namei.unix at gmail.com> wrote:
> On Fri, Mar 14, 2014 at 02:43:59PM +0000, Aydelott, Ryan M. wrote:
>> Interesting note on the data management, I have not dug much into the internals with Sheepdog yet - but the only possible explanation would be if any metadata on file object positioning was being retrieved through the backend.
>
> Each volume has only 4MB meta object that track the allocatin bitmap for objects
> mainly for thin provision. We don't record any placement meta data at all,
> instead we 100% rely on consistent hash for placement of all the objects, resulting
> in astonishing load and space balance. We have run 150 nodes with 12 disks on
> each node and confirmed the evenly distributed load and space. As you noticed,
> we don't have meta servers.
>
>>
>> I agree with your statements as the IO on the zookeeper nodes is very small/nowhere near the data we are pushing throughout the sheepd members.
>>
>> Sheepdog is being started as follows for each test iteration:
>>
>> Corosync: sheep -n -c corosync:172.21.5.0 /meta,/var/lib/sheepdog/disc0,/var/lib/sheepdog/disc1,/var/lib/sheepdog/disc2,/var/lib/sheepdog/disc3,/var/lib/sheepdog/disc4,/var/lib/sheepdog/disc5,/var/lib/sheepdog/disc6,/var/lib/sheepdog/disc7,/var/lib/sheepdog/disc8,/var/lib/sheepdog/disc9,/var/lib/sheepdog/disc10,/var/lib/sheepdog/disc11,/var/lib/sheepdog/disc12,/var/lib/sheepdog/disc13
>>
>> Zookeeper: sheep -n -c zookeeper:172.21.5.161:2181,172.21.5.162:2181,172.21.5.163:2181,172.21.5.164:2181,172.21.5.165:2181 /meta /var/lib/sheepdog/disc0,/var/lib/sheepdog/disc1,/var/lib/sheepdog/disc2,/var/lib/sheepdog/disc3,/var/lib/sheepdog/disc4,/var/lib/sheepdog/disc5,/var/lib/sheepdog/disc6,/var/lib/sheepdog/disc7,/var/lib/sheepdog/disc8,/var/lib/sheepdog/disc9,/var/lib/sheepdog/disc10,/var/lib/sheepdog/disc11,/var/lib/sheepdog/disc12,/var/lib/sheepdog/disc13
>>
>> The driver we wrote/use is: https://github.com/devoid/nova/tree/sheepdog-nova-support-havana
>>
>> Which builds out libvirt.xml as follows:
>>
>> <disk type="network" device="disk">
>> <driver name="qemu" cache="writethrough"/>
>> <source protocol="sheepdog" name="//172.21.5.141:7000/instance_f9dc065b-d05d-47cb-a3e6-b02049f049df_disk"/>
>> <target bus="virtio" dev="vda"/>
>> </disk>
>>
>
> Hmm, since you don't use object cache, you might try set 'cache=none' to save
> one extra internal flush operation in QEMU. But this won't make big difference.
>
> I have no idea why you see such a huge difference with zookeeper. Wierd. Is this
> result reproduciable? I guess something wrong with other confiuration or system.
>
> P.S. I have long wanted someone who can access IB to develop native IB network
> transport code for sheepdog instead of ip-over-IB.
>
> Thanks
> Yuan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wpkg.org/pipermail/sheepdog-users/attachments/20140314/4bef8491/attachment-0005.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.wpkg.org/pipermail/sheepdog-users/attachments/20140314/4bef8491/attachment-0003.sig>
More information about the sheepdog-users
mailing list