[sheepdog] [PATCH v4 0/6] create onode before uploading object completed

[Liu Yuan]

On Wed, Feb 26, 2014 at 11:00:15AM +0800, Robin Dong wrote:
> From: Robin Dong <sanbai at taobao.com>
> 
> At present, the implemention of swift interface for creating object in sheepdog
> is:
> 
> 1. lock container
> 2. check whether the onode with same object name is exists.
> 3. unlock container
> 4. upload object
> 5. create onode
> 
> this sequence have a problem: if two clients uploading same objects
> concurrently, it will create two objects with same names in container.
> To avoid duplicated names, we must put "create onode" operation in container
> lock regions.
> 
> Therefore we need to change the processes of creating object to:
> 
> 1. lock container
> 2. check whether the onode is exists.
> 3. allocate data space for object, and create onode, then write it done
> 4. unlock container
> 5. upload object
> 
> this routine will avoid uploading duplicated objects.
> 
> There is an exception on the new routine: if the client halt the uploading
> progress, we will have a "uploading incompleted" onode.
> 
> The solution is: we can add code for onode to identify its status.
> A new onode will be set to "INIT", and after uploading completed, the onode will
> be set to  "COMPLETED". So, when users try to use swift interface to GET a
> "incompleted" object, sheep will find out the onode is "INIT" which means
> "not completed", so sheep will return "partial content"for http request, and
> user could DELETE the object and upload it again.
> 
> Signed-off-by: Robin Dong <sanbai at taobao.com>
> ---
> v3-->v4:
>   1. change 'Forced: true' to "Force: true'
>   2. fix compile warning for gcc-4.6
> 
> v2-->v3:
>   1. add explain of design on commit log and source file
>   2. use sd_mutex_xxx() instead of pthread_mutex_xxx()
>   3. add HTTP Header 'Forced: true' to delete forcely,
>      instead of using FORCE_DELETE
> 
> v1-->v2:
>   1. remove some redundant function like xxx_nolock()
>   2. use the mmap-memory to store phtread_mutex_t
>   3. add flag SD_RES_INCOMPLETE
> 
> Robin Dong (6):
>   sheep/http: add nolock version for onode_lookup()
>   sheep/http: change routine of kv_create_object()
>   sheep/cluster: let local_lock() support nested
>   sheep/http: add flag for onode
>   sheep/http: allow 'Force: true' http header to delete 'incompleted'
>     object forcely
>   tests: add test case for new kv_create_object() implemention
> 
>  include/sheepdog_proto.h    |   1 +
>  include/util.h              |  14 +++
>  sheep/cluster/local.c       | 109 ++++++++++++++---
>  sheep/http/http.c           |   5 +
>  sheep/http/http.h           |   4 +-
>  sheep/http/kv.c             | 286 +++++++++++++++++++++++++++++++++++---------
>  sheep/http/s3.c             |   2 +-
>  sheep/http/swift.c          |  12 +-
>  tests/functional/087        | 102 ++++++++++++++++
>  tests/functional/087.out    |  32 +++++
>  tests/functional/group      |   1 +
>  tests/functional/nginx.conf |   6 +
>  12 files changed, 495 insertions(+), 79 deletions(-)
>  create mode 100755 tests/functional/087
>  create mode 100644 tests/functional/087.out
> 
> -- 
> 1.7.12.4
> 
> -- 
> sheepdog mailing list
> sheepdog at lists.wpkg.org
> http://lists.wpkg.org/mailman/listinfo/sheepdog

Applied thanks

Yuan