[sheepdog-users] calculate network capacity
Andrew J. Hobbs
ajhobbs at desu.edu
Mon Sep 29 16:36:18 CEST 2014
Nodes communicate with each other directly, and the overall throughput
of the cluster, as an aggregate, increases as you add nodes. So some
Raw wire speed would give you a throughput of 125MB/s, after protocol,
other issues, probably closer to 115-120MB. You have two links,
depending on your purpose, you could either a) bond them to double your
data transfer or b) virtual machines on one, sheepdog on the other, thus
removing sheepdog from contending with nodes trying to communicate.
Computationally a makes sense, b makes more sense for service providing
Connections should be full duplex, your writes will be divided by your
copy count. You don't say what your write to read ratio is for the
a) Roughly 230MB/s per node, 10 nodes, each with 10 VMs
read - 23MB/s
write - 7.6MB/s assuming copy=3
b) 115MB/s per node, 10 nodes, each with 10 VMs
read = 11.5MB/s
write = 3.8MB/s, assuming copy=3
This is assuming each VM is literally reading and writing as fast as it
can. Write can be increased by switching to erasure encoding and
increasing the number of data blocks compared to parity blocks. For
example, using -c 4:2 should change write to something more like
15.3MB/s for a or 7.6MB/s for b.
Increasing nodes to 30 you get:
a) Roughly 230MB/s per node, 30 nodes, each with 4 VMs
read = 57.5MB/s
write = 19.1MB/s (copy=3) 38.3MB/s (copy=4:2)
b) 115MB/s per node, 30 nodes, each with 10VMs
read = 28.8MB/s
write = 9.6MB/s (copy=3) 19.2MB/s (copy=4:2) 25.6MB/s (copy=16:2)
Again, all assuming each VM is sucking in data and writing it out as
fast as it can. All of this can be shifted by using a different network
interconnect for the sheepdog side (10G Ethernet, Infiniband, etc).
This is also assuming that each node has suffucient drive capacity and
speed to saturate the network connection. And of course, there's no
discussion as to IO operations per second, but pure throughput which is
what you'd see with dd.
From one of the presentations given recently, it should be noted that
benchmarking with other distributed file systems such as GlusterFS and
Ceph have shown sheepdog to compare favorably or excel in different
categories. The link should be in the archives for this list.
On 09/29/2014 09:05 AM, Vasiliy Tolstov wrote:
> 2014-09-26 23:19 GMT+04:00 Valerio Pachera <sirio81 at gmail.com>:
>> I'm sorry, I'm not understanding much.
>> What is a knot? Is it http://en.wikipedia.org/wiki/Knot_%28unit%29 ?
> Sorry =), translator failed =). I need to know does it possible to
> provide 3-10 Mb/s for each of 100 vps if the server node have only 2
> links with 1Gb/s
>> How many servers/nodes do you have?
> Now 10, but want is about 20-30.
>> How many GB do you need for each vdi?
> From 5 to 200Gb
>> Are the servers connected by a Gb netwrok?
> Yes, 2 links
>> What is the 3-10Mbps speed you are talking about?
> i'm try to run dd inside vps.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 353 bytes
More information about the sheepdog-users