[sheepdog] [PATCH v2 7/9] object cache: reclaim cached objects when cache reaches the max size

[levin li]

On 2012年07月26日 08:22, MORITA Kazutaka wrote:
> At Wed, 25 Jul 2012 20:15:23 +0800,
> levin li wrote:
>>
>> From: levin li <xingke.lwp at taobao.com>
>>
>> This patch do reclaiming work when the total size of cached objects
>> reaches the max size specified by user, I did it in the following way:
>>
>> 1. check the object tree for the object entry to determine whether the
>>    cache entry is exist and whether it's reclaiming, if it's reclaiming
>>    we make sheep ingore the cache.
>> 2. In object_cache_rw() we search the cache entry, after passed the sanity
>>    check, we increment its refcnt to tell the reclaiming worker that this
>>    entry is being referenced, we should not reclaim it now.
>> 3. In add_to_object_cache(), when the cached size reaches the max size,
>>    we start a reclaiming thread, only one such thread can be running at
>>    one time.
>> 4. In reclaim_work(), we reclaim cached objects until the cache size reduced
>>    to 80% of the max size.
>> 5. In reclaim_object(), we start to reclaim an object, before this, we check
>>    that if the cache is flushing, we don't reclaim it, and if the refcnt of
>>    the object is not zero, we also don't reclaim it.
>>    If the cached object is dirty, we flush it by push_cache_object(), and
>>    then try to remove the object.
>>
>> Signed-off-by: levin li <xingke.lwp at taobao.com>
>> ---
>>  include/sheepdog_proto.h |    1 +
>>  sheep/object_cache.c     |  463 +++++++++++++++++++++++++++++++++++++++-------
>>  sheep/sheep.c            |    3 +-
>>  sheep/sheep_priv.h       |    1 +
>>  sheep/store.c            |    8 +
>>  5 files changed, 407 insertions(+), 69 deletions(-)
>>
>> diff --git a/include/sheepdog_proto.h b/include/sheepdog_proto.h
>> index 45a4b81..05597fb 100644
>> --- a/include/sheepdog_proto.h
>> +++ b/include/sheepdog_proto.h
>> @@ -68,6 +68,7 @@
>>  #define SD_RES_CLUSTER_RECOVERING 0x22 /* Cluster is recovering. */
>>  #define SD_RES_OBJ_RECOVERING     0x23 /* Object is recovering */
>>  #define SD_RES_KILLED           0x24 /* Node is killed */
>> +#define SD_RES_NO_CACHE      0x25 /* No cache object found */
> 
> This should be a sheepdog-internal error code, no?
> 

Yes, it's an internal error code.

> 
>> -static inline void
>> -del_from_dirty_tree_and_list(struct object_cache_entry *entry,
>> -			     struct rb_root *dirty_tree)
>> -{
>> -	rb_erase(&entry->dirty_node, dirty_tree);
>> -	list_del(&entry->list);
>> -}
>> -
>>  /* Caller should hold the oc->lock */
>>  static inline void
>>  add_to_dirty_tree_and_list(struct object_cache *oc, uint32_t idx,
>> @@ -289,6 +488,9 @@ add_to_dirty_tree_and_list(struct object_cache *oc, uint32_t idx,
>>  	if (!entry)
>>  		panic("Can not find object entry %" PRIx32 "\n", idx);
>>  
>> +	if (cache_is_reclaiming())
>> +		return;
>> +
>>  	/* If cache isn't in reclaiming, move it
>>  	 * to the head of lru list */
>>  	cds_list_del_rcu(&entry->lru_list);
>> @@ -321,6 +523,9 @@ static void add_to_object_cache(struct object_cache *oc, uint32_t idx)
>>  	entry->idx = idx;
>>  	CDS_INIT_LIST_HEAD(&entry->lru_list);
>>  
>> +	dprintf("cache object for vdi %" PRIx32 ", idx %08" PRIx32 "added\n",
>> +		oc->vid, idx);
>> +
>>  	pthread_rwlock_wrlock(&oc->lock);
>>  	old = object_cache_insert(&oc->object_tree, entry);
>>  	if (!old) {
>> @@ -331,20 +536,55 @@ static void add_to_object_cache(struct object_cache *oc, uint32_t idx)
>>  		entry = old;
>>  	}
>>  	pthread_rwlock_unlock(&oc->lock);
>> +
>> +	dprintf("sys_cache.cache_size %" PRIx64 ", sys->cache_size %" PRIx64 "\n",
>> +		uatomic_read(&sys_cache.cache_size), sys->cache_size);
>> +	if (sys->cache_size &&
>> +	    uatomic_read(&sys_cache.cache_size) > sys->cache_size &&
>> +	    !cache_is_reclaiming()) {
>> +		struct work *work = xzalloc(sizeof(struct work));
>> +		uatomic_set(&sys_cache.reclaiming, 1);
> 
> IIUC, cache_is_reclaiming() and uatomic_set(&sys_cache.reclaiming, 1)
> must be done atomically.  I think you need to use uatomic_cmpxchg.
> 

You're right, I'd correct it.

thanks,

levin

> 
> Thanks,
> 
> Kazutaka
>