Hi Christoph, thanks for your reply. Am 09.07.2012 16:49, schrieb Christoph Hellwig: > Before I would like to let you try 29bfdbd6a95fdf8d827e177046dbab12ee342611 > or earlier? I already tried that. It gives me the same results. >>> Can you also do a perf record -g on both a storage node and the >>> kvm box to see if there's anything interesting on them? >> >> Option -g is not known by my perf command? > > perf record -g records the callgraph, and it's been there for a long > time. Did you try that above line or something like perf -g record? *argh* i really tried -g record ;-( i'm on holiday right now so it will take some days. What is the expected output of this command? Should i attach it? >> is a perf record sleep 10 enough? Should i upload then the data file >> somewhere? > > Would be nice to get a slightly long run. How long >> Snapshot of perf top from KVM host: >> 14.96% [kernel] [k] _raw_spin_lock > > With the callchains we could expand what spinlock we hammer here. ?! >> 8.13% kvm [.] 0x00000000001d8084 > > Also if kvm/qemu is self-build can you build it with -g to get debug > info? If not see if there is a qemu-dbg/kvm-dbg or similar package for > your distribution. It is self build - i'll do a debug. >> Snapshot of perf top from sheep node (not acting as the gateway / >> target for kvm): >> 2,78% libgcc_s.so.1 [.] 0x000000000000e72b >> 2,21% [kernel] [k] __schedule >> 2,14% [kernel] [k] ahci_port_intr >> 2,08% [kernel] [k] _raw_spin_lock >> 1,77% [kernel] [k] _raw_spin_lock_irqsave >> 1,15% [kernel] [k] ahci_interrupt >> 1,11% [kernel] [k] ahci_scr_read >> 0,94% [kernel] [k] kmem_cache_alloc >> 0,90% [kernel] [k] _raw_spin_lock_irq >> 0,81% [kernel] [k] menu_select >> 0,76% [kernel] [k] _raw_spin_unlock_irqrestore >> 0,73% libpthread-2.11.3.so [.] pthread_mutex_lock >> 0,70% libc-2.11.3.so [.] vfprintf > > Mostly not spending any noticable CPU time, which is quite interesting. Yes the sheep acting as gateway is taking 20x times more CPU than the node sheep processess. Stefan |