[sheepdog] [PATCH v2 0/6] create onode before uploading object completed

[Robin Dong]

From: Robin Dong <sanbai at taobao.com>

At present, the implemention of swift interface for creating object in sheepdog
is:

1. lock container
2. check whether the onode with same object name is exists.
3. unlock container
4. upload object
5. create onode

this sequence have a problem: if two clients uploading same objects
concurrently, it will create two objects with same names in container.
To avoid duplicated names, we must put "create onode" operation in container
lock regions.

Therefore we need to change the processes of creating object to:

1. lock container
2. check whether the onode is exists.
3. allocate data space for object, and create onode, then write it done
4. unlock container
5. upload object

this routine will avoid uploading duplicated objects.

There is an exception on the new routine: if the client halt the uploading
progress, we will have a "uploading incompleted" onode.

The solution is: we can add code for onode to identify its status.
A new onode will be set to "INIT", and after uploading completed, the onode will
be set to  "COMPLETED". So, when users try to use swift interface to GET a
"incompleted" object, sheep will find out the onode is "INIT" which means
"not completed", so sheep will return "partial content"for http request, and
user could DELETE the object and upload it again.


Imaging a scenario:

1. Client A is uploading a large object which names 'elephant', it
   only allocate meta-data in oalloc and is creating SD_OBJ_DATA files.
2. Client B send a DELETE request to remove object 'elephant', it
   will remove all the backend files for 'elephant'.
   At the same time, Client A dosen't know what happend because uploading
   progress don't need to lock any vdi.
3. Client A return Create-object-success, but the real data have all been
   removed.

To avoid this scenario, we let DELETE operation do nothing but only return
'CONFLICT' when the object is 'incompleted'. And, add a new operation
'FORCE_DELETE' for users to remove 'incompleted' object forcely when they
make sure that there isn't any uploading progress for this object.

Signed-off-by: Robin Dong <sanbai at taobao.com>
---
Robin Dong (6):
  sheep/http: add nolock version for onode_lookup()
  sheep/http: change routine of kv_create_object()
  sheep/cluster: let local_lock() support nested
  sheep/http: add flag for onode
  sheep/http: add FORCE_DELETE to delete 'incompleted' object
  tests: add test case for new kv_create_object() implemention

 include/sheepdog_proto.h |   1 +
 sheep/cluster/local.c    | 108 ++++++++++++++++++----
 sheep/http/http.c        |   5 +
 sheep/http/http.h        |   5 +-
 sheep/http/kv.c          | 231 +++++++++++++++++++++++++++++++++++------------
 sheep/http/s3.c          |   2 +-
 sheep/http/swift.c       |  54 ++++++++++-
 tests/functional/087     | 102 +++++++++++++++++++++
 tests/functional/087.out |  32 +++++++
 tests/functional/group   |   1 +
 10 files changed, 464 insertions(+), 77 deletions(-)
 create mode 100755 tests/functional/087
 create mode 100644 tests/functional/087.out

-- 
1.7.12.4