[sheepdog-users] create_short_thread ! PANIC: Cannot allocate memory since last git (3 jul)

[Alexandre DERUMIER]

Hi,
I'm having sheepdog daemon crash since today (just updated from git,last commit 5b710225269334ff7b2cc632e39d5e3b81604f17)

Jul 04 07:07:21 [main] post_cluster_new_vdi(131) done 0 1143552
Jul 04 07:07:27 [main] queue_cluster_request(347) 0x1f15f20 12
Jul 04 07:07:27 [block] do_lookup_vdi(209) looking for vm-115-disk-1 (117300)
Jul 04 07:07:27 [main] queue_cluster_request(347) 0x1f15f20 13
Jul 04 07:07:27 [main] queue_cluster_request(347) 0x1f15f20 12
Jul 04 07:07:27 [block] do_lookup_vdi(209) looking for vm-115-disk-1 (117300)
Jul 04 07:08:17 [main] create_short_thread(88) PANIC: Cannot allocate memory
Jul 04 07:08:18 [main] crash_handler(408) sheep pid 19014 exited unexpectedly.


system has a lot of free memory.

I was testing fio benchmark, I can reproduce it 100%.

----- Mail original ----- 

De: "Christoph Hellwig" <hch at infradead.org> 
À: "Stefan Priebe" <s.priebe at profihost.ag> 
Cc: sheepdog-users at lists.wpkg.org 
Envoyé: Mercredi 4 Juillet 2012 07:07:19 
Objet: Re: [sheepdog-users] 50.000 iops per VM 

On Tue, Jul 03, 2012 at 10:28:17PM +0200, Stefan Priebe wrote: 
> For testing purposes i had: 
> 
> 4x dedicated sheep nodes: 
> - Intel Xeon E5 8 Core 3,6Ghz 
> - 64GB Memory 
> - 10GBE 
> - 4x Intel 240GB SSD (write: 250MB/s and 25.000 iop/s random 4k) 

exact git revision of sheepdog 

> On the KVM Machine i used a sheep process acting as the gateway with 
> the following options: -G -i 32 -g 32 -W 

> On each sheep node i used one sheep per disk: -g 32 -i 32 -W 

filesystems on the nodes? 

I/O scheduler in the guest and the host? 

sheep git revision? qemu release? 


In general I doubt anyone has optimized sheepdog for iops and low 
latency at this moment as other things have kept people. There's 
some relatively low hanging fruit like avoiding additional copies 
in the gateway, but your numbers still sound very low. 

Can you also do a perf record -g on both a storage node and the 
kvm box to see if there's anything interesting on them? 

-- 
sheepdog-users mailing lists 
sheepdog-users at lists.wpkg.org 
http://lists.wpkg.org/mailman/listinfo/sheepdog-users 



-- 

-- 



	

Alexandre D e rumier 

Ingénieur Systèmes et Réseaux 


Fixe : 03 20 68 88 85 

Fax : 03 20 68 90 88 


45 Bvd du Général Leclerc 59100 Roubaix 
12 rue Marivaux 75002 Paris