From: Robin Dong <sanbai at taobao.com> Sheepdog failed to recover object when we running it on 5 servers cluster with about 20G data by erasure-code mode. The reason is in default_create_and_write(): it rename() obj to data-directory and then set xattr of ec-index for it, this will leave a time-window for another process to read the data-object but can't get xattr of ec-index. Then the process will report get-xattr fail and remove the disk as it think it's an io-error event. Signed-off-by: Robin Dong <sanbai at taobao.com> --- sheep/plain_store.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/sheep/plain_store.c b/sheep/plain_store.c index 90ef0a6..754a25a 100644 --- a/sheep/plain_store.c +++ b/sheep/plain_store.c @@ -431,16 +431,18 @@ int default_create_and_write(uint64_t oid, const struct siocb *iocb) goto out; } + if (ec && set_erasure_index(tmp_path, iocb->ec_index) < 0) { + ret = err_to_sderr(tmp_path, oid, errno); + goto out; + } + ret = rename(tmp_path, path); if (ret < 0) { sd_err("failed to rename %s to %s: %m", tmp_path, path); ret = err_to_sderr(path, oid, errno); goto out; } - if (ec && set_erasure_index(path, iocb->ec_index) < 0) { - ret = err_to_sderr(path, oid, errno); - goto out; - } + ret = SD_RES_SUCCESS; objlist_cache_insert(oid); out: -- 1.7.12.4 |