[sheepdog-users] Some questions about the big file in the http simple storage

[Liu Yuan]

On Tue, Jul 29, 2014 at 04:25:13PM +0800, 王力锋 wrote:
> Hi, All
> 
>            I have two questions about the http simple storage feature.
>            Q1:For big file storage. I use tengine as the we bserver and then put one 200G file from client to the cluster. Because the file is too large and it will been written to the disk. Then the store speed has only 100M/S or a little more although my cluster has a 10G NIC.
>             In another word, hard disk IO is slower than the network speed. In this case, how can I raise the store speed?

I remember Tengine support to directly forward request data without storing it
locally. You can consult Tengine team.

> 
> 
>             Q2:When use the http simple storage. when does sheepdog divide the file into fixed-sized objects and do the erasure code operation if I enabled the EC feature?
>             I'm very confused and look forward to your response.

Yes. You can use both EC or replication for http backend storage. Http storage
use the redundancy scheme as specified in 'dog cluster format -c x:y' .E.g, if
you 

dog cluster format -c 4:2

Then http storage will use EC with 4:2 scheme.

if you

dog cluster format -c 2

it will use 2 copies replication.

We do generate both data strips and parity strips when data are uploaded into
cluster. In other words, erasure coding of data is done at real time, NOT a
background convert as in hadoop hdfs.

Thanks
Yuan