On 2012年07月26日 08:22, MORITA Kazutaka wrote: > At Wed, 25 Jul 2012 20:15:23 +0800, > levin li wrote: >> >> From: levin li <xingke.lwp at taobao.com> >> >> This patch do reclaiming work when the total size of cached objects >> reaches the max size specified by user, I did it in the following way: >> >> 1. check the object tree for the object entry to determine whether the >> cache entry is exist and whether it's reclaiming, if it's reclaiming >> we make sheep ingore the cache. >> 2. In object_cache_rw() we search the cache entry, after passed the sanity >> check, we increment its refcnt to tell the reclaiming worker that this >> entry is being referenced, we should not reclaim it now. >> 3. In add_to_object_cache(), when the cached size reaches the max size, >> we start a reclaiming thread, only one such thread can be running at >> one time. >> 4. In reclaim_work(), we reclaim cached objects until the cache size reduced >> to 80% of the max size. >> 5. In reclaim_object(), we start to reclaim an object, before this, we check >> that if the cache is flushing, we don't reclaim it, and if the refcnt of >> the object is not zero, we also don't reclaim it. >> If the cached object is dirty, we flush it by push_cache_object(), and >> then try to remove the object. >> >> Signed-off-by: levin li <xingke.lwp at taobao.com> >> --- >> include/sheepdog_proto.h | 1 + >> sheep/object_cache.c | 463 +++++++++++++++++++++++++++++++++++++++------- >> sheep/sheep.c | 3 +- >> sheep/sheep_priv.h | 1 + >> sheep/store.c | 8 + >> 5 files changed, 407 insertions(+), 69 deletions(-) >> >> diff --git a/include/sheepdog_proto.h b/include/sheepdog_proto.h >> index 45a4b81..05597fb 100644 >> --- a/include/sheepdog_proto.h >> +++ b/include/sheepdog_proto.h >> @@ -68,6 +68,7 @@ >> #define SD_RES_CLUSTER_RECOVERING 0x22 /* Cluster is recovering. */ >> #define SD_RES_OBJ_RECOVERING 0x23 /* Object is recovering */ >> #define SD_RES_KILLED 0x24 /* Node is killed */ >> +#define SD_RES_NO_CACHE 0x25 /* No cache object found */ > > This should be a sheepdog-internal error code, no? > Yes, it's an internal error code. > >> -static inline void >> -del_from_dirty_tree_and_list(struct object_cache_entry *entry, >> - struct rb_root *dirty_tree) >> -{ >> - rb_erase(&entry->dirty_node, dirty_tree); >> - list_del(&entry->list); >> -} >> - >> /* Caller should hold the oc->lock */ >> static inline void >> add_to_dirty_tree_and_list(struct object_cache *oc, uint32_t idx, >> @@ -289,6 +488,9 @@ add_to_dirty_tree_and_list(struct object_cache *oc, uint32_t idx, >> if (!entry) >> panic("Can not find object entry %" PRIx32 "\n", idx); >> >> + if (cache_is_reclaiming()) >> + return; >> + >> /* If cache isn't in reclaiming, move it >> * to the head of lru list */ >> cds_list_del_rcu(&entry->lru_list); >> @@ -321,6 +523,9 @@ static void add_to_object_cache(struct object_cache *oc, uint32_t idx) >> entry->idx = idx; >> CDS_INIT_LIST_HEAD(&entry->lru_list); >> >> + dprintf("cache object for vdi %" PRIx32 ", idx %08" PRIx32 "added\n", >> + oc->vid, idx); >> + >> pthread_rwlock_wrlock(&oc->lock); >> old = object_cache_insert(&oc->object_tree, entry); >> if (!old) { >> @@ -331,20 +536,55 @@ static void add_to_object_cache(struct object_cache *oc, uint32_t idx) >> entry = old; >> } >> pthread_rwlock_unlock(&oc->lock); >> + >> + dprintf("sys_cache.cache_size %" PRIx64 ", sys->cache_size %" PRIx64 "\n", >> + uatomic_read(&sys_cache.cache_size), sys->cache_size); >> + if (sys->cache_size && >> + uatomic_read(&sys_cache.cache_size) > sys->cache_size && >> + !cache_is_reclaiming()) { >> + struct work *work = xzalloc(sizeof(struct work)); >> + uatomic_set(&sys_cache.reclaiming, 1); > > IIUC, cache_is_reclaiming() and uatomic_set(&sys_cache.reclaiming, 1) > must be done atomically. I think you need to use uatomic_cmpxchg. > You're right, I'd correct it. thanks, levin > > Thanks, > > Kazutaka > |