At Wed, 25 Jul 2012 20:15:23 +0800, levin li wrote: > > From: levin li <xingke.lwp at taobao.com> > > This patch do reclaiming work when the total size of cached objects > reaches the max size specified by user, I did it in the following way: > > 1. check the object tree for the object entry to determine whether the > cache entry is exist and whether it's reclaiming, if it's reclaiming > we make sheep ingore the cache. > 2. In object_cache_rw() we search the cache entry, after passed the sanity > check, we increment its refcnt to tell the reclaiming worker that this > entry is being referenced, we should not reclaim it now. > 3. In add_to_object_cache(), when the cached size reaches the max size, > we start a reclaiming thread, only one such thread can be running at > one time. > 4. In reclaim_work(), we reclaim cached objects until the cache size reduced > to 80% of the max size. > 5. In reclaim_object(), we start to reclaim an object, before this, we check > that if the cache is flushing, we don't reclaim it, and if the refcnt of > the object is not zero, we also don't reclaim it. > If the cached object is dirty, we flush it by push_cache_object(), and > then try to remove the object. > > Signed-off-by: levin li <xingke.lwp at taobao.com> > --- > include/sheepdog_proto.h | 1 + > sheep/object_cache.c | 463 +++++++++++++++++++++++++++++++++++++++------- > sheep/sheep.c | 3 +- > sheep/sheep_priv.h | 1 + > sheep/store.c | 8 + > 5 files changed, 407 insertions(+), 69 deletions(-) > > diff --git a/include/sheepdog_proto.h b/include/sheepdog_proto.h > index 45a4b81..05597fb 100644 > --- a/include/sheepdog_proto.h > +++ b/include/sheepdog_proto.h > @@ -68,6 +68,7 @@ > #define SD_RES_CLUSTER_RECOVERING 0x22 /* Cluster is recovering. */ > #define SD_RES_OBJ_RECOVERING 0x23 /* Object is recovering */ > #define SD_RES_KILLED 0x24 /* Node is killed */ > +#define SD_RES_NO_CACHE 0x25 /* No cache object found */ This should be a sheepdog-internal error code, no? > -static inline void > -del_from_dirty_tree_and_list(struct object_cache_entry *entry, > - struct rb_root *dirty_tree) > -{ > - rb_erase(&entry->dirty_node, dirty_tree); > - list_del(&entry->list); > -} > - > /* Caller should hold the oc->lock */ > static inline void > add_to_dirty_tree_and_list(struct object_cache *oc, uint32_t idx, > @@ -289,6 +488,9 @@ add_to_dirty_tree_and_list(struct object_cache *oc, uint32_t idx, > if (!entry) > panic("Can not find object entry %" PRIx32 "\n", idx); > > + if (cache_is_reclaiming()) > + return; > + > /* If cache isn't in reclaiming, move it > * to the head of lru list */ > cds_list_del_rcu(&entry->lru_list); > @@ -321,6 +523,9 @@ static void add_to_object_cache(struct object_cache *oc, uint32_t idx) > entry->idx = idx; > CDS_INIT_LIST_HEAD(&entry->lru_list); > > + dprintf("cache object for vdi %" PRIx32 ", idx %08" PRIx32 "added\n", > + oc->vid, idx); > + > pthread_rwlock_wrlock(&oc->lock); > old = object_cache_insert(&oc->object_tree, entry); > if (!old) { > @@ -331,20 +536,55 @@ static void add_to_object_cache(struct object_cache *oc, uint32_t idx) > entry = old; > } > pthread_rwlock_unlock(&oc->lock); > + > + dprintf("sys_cache.cache_size %" PRIx64 ", sys->cache_size %" PRIx64 "\n", > + uatomic_read(&sys_cache.cache_size), sys->cache_size); > + if (sys->cache_size && > + uatomic_read(&sys_cache.cache_size) > sys->cache_size && > + !cache_is_reclaiming()) { > + struct work *work = xzalloc(sizeof(struct work)); > + uatomic_set(&sys_cache.reclaiming, 1); IIUC, cache_is_reclaiming() and uatomic_set(&sys_cache.reclaiming, 1) must be done atomically. I think you need to use uatomic_cmpxchg. Thanks, Kazutaka |