[sheepdog-users] dog {cluster,vdi} check needs too much ram

[Hitoshi Mitake]

At Sun, 01 Jun 2014 22:33:53 +0200,
Marcin Mirosław wrote:
> 
> W dniu 2014-05-31 17:05, Hitoshi Mitake pisze:
> > At Sat, 31 May 2014 00:33:38 +0200,
> > Marcin Mirosław wrote:
> > 
> > Hi Marcin,
> > 
> >>
> >> Hi!
> >> I'm locally launching three sheeps on three disks, next:
> >> # dog vdi create -P -c 2:1 -v test 1T
> >>   0.9 % [>
> >>                                                ] 9.2 GB / 1.0 TB    ^C
> >> and I'm hitting ctrl+c to stop command. Now I can see:
> >> # dog vdi list
> >>   Name        Id    Size    Used  Shared    Creation time   VDI id
> >> Copies  Tag
> >>   test         0  1.0 TB  9.2 GB  0.0 MB 2014-05-31 00:19   7c2b25    2:1
> >>
> >> and I started wondering if vdi isn't corrupted:
> >>
> >> # dog vdi check test
> >>   7.3 % [========>
> >>                                                ] 75 GB / 1.0 TB     Killed
> >>
> >> # dog cluster check
> >> fix vdi test
> >>   3.4 % [===>
> >>                                                ] 34 GB / 1.0 TB     Killed
> >>
> >> In dmesg there is:
> >>> [ 2577.036700] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
> >>> [ 2577.036711] [  546]     0   546     6461      120      14        0         -1000 udevd
> >>> [ 2577.036716] [ 1040]     0  1040   120489      193      34        0             0 rsyslogd
> >>> [ 2577.036719] [ 1062]     0  1062     1062       35       5        0             0 acpid
> >>> [ 2577.036723] [ 1100]     0  1100     4979      148      14        0             0 crond
> >>> [ 2577.036726] [ 1364]     0  1364      579       17       3        0             0 busybox
> >>> [ 2577.036730] [ 1439]    40  1439    49058     4903      28        0             0 named
> >>> [ 2577.036733] [ 1474]   123  1474     7298       79      18        0             0 ntpd
> >>> [ 2577.036736] [ 1491]     0  1491     2253       44       9        0             0 rsync
> >>> [ 2577.036740] [ 1509]     0  1509    11638      163      26        0             0 smbd
> >>> [ 2577.036743] [ 1514]     0  1514     7924      109      19        0             0 nmbd
> >>> [ 2577.036746] [ 1515]     0  1515     7843       94      19        0             0 nmbd
> >>> [ 2577.036750] [ 1529]     0  1529    11638      163      25        0             0 smbd
> >>> [ 2577.036753] [ 1534]     0  1534     7525      119      20        0         -1000 sshd
> >>> [ 2577.036756] [ 1551]     0  1551     3703       39      11        0             0 agetty
> >>> [ 2577.036759] [ 1552]     0  1552     3703       38      11        0             0 agetty
> >>> [ 2577.036763] [ 1553]     0  1553     3703       38      10        0             0 agetty
> >>> [ 2577.036766] [ 1554]     0  1554     3703       39      11        0             0 agetty
> >>> [ 2577.036769] [ 1555]     0  1555     3703       39      11        0             0 agetty
> >>> [ 2577.036772] [ 1556]     0  1556     3703       40      11        0             0 agetty
> >>> [ 2577.036776] [ 1557]     0  1557    24818      273      39        0             0 sshd
> >>> [ 2577.036779] [ 1563]     0  1563     5320      847      15        0             0 bash
> >>> [ 2577.036802] [ 1333]     0  1333     3450       75      11        0             0 tmux
> >>> [ 2577.036805] [ 1335]     0  1335     7293     1842      18        0             0 tmux
> >>> [ 2577.036808] [ 1336]     0  1336     5337      858      15        0             0 bash
> >>> [ 2577.036812] [ 1389]     0  1389     5330      843      14        0             0 bash
> >>> [ 2577.036815] [22751]     0 22751     5372      856      15        0             0 bash
> >>> [ 2577.036821] [ 8286]   109  8286   369574     1204      56        0             0 sheep
> >>> [ 2577.036824] [ 8287]   109  8287     8617       47      12        0             0 sheep
> >>> [ 2577.036827] [ 8531]   109  8531   418726     2380      63        0             0 sheep
> >>> [ 2577.036830] [ 8532]   109  8532     8617       47      12        0             0 sheep
> >>> [ 2577.036834] [ 8799]   109  8799   353185     2295      54        0             0 sheep
> >>> [ 2577.036837] [ 8800]   109  8800     8617       46      12        0             0 sheep
> >>> [ 2577.036841] [ 5941]     0  5941    24095    14199      50        0             0 emerge
> >>> [ 2577.036845] [ 6134]   250  6134     1061       37       7        0             0 sandbox
> >>> [ 2577.036848] [ 6136]   250  6136     6081      496      15        0             0 ebuild.sh
> >>> [ 2577.036852] [ 6153]   250  6153     6087      518      14        0             0 ebuild.sh
> >>> [ 2577.036855] [ 6175]   250  6175     5846      255      15        0             0 configure
> >>> [ 2577.036859] [ 6432]     0  6430   372619   296187     601        0             0 dog
> >>> [ 2577.036863] [ 6456]   250  6456     5846      254      14        0             0 configure
> >>> [ 2577.036866] [ 6457]   250  6457    18555       70       9        0             0 conftest
> >>> [ 2577.036869] Out of memory: Kill process 6432 (dog) score 753 or sacrifice child
> >>> [ 2577.036872] Killed process 6432 (dog) total-vm:1490476kB, anon-rss:1184744kB, file-rss:4kB
> >>
> >>
> >> Hmm, I did another test. Even I don't hit ctrl+c while creating vdi dog
> >> clucter check is killed by OOM.
> >> Box has 1.5G RAM.
> >> Is it expected that dog cluster check needs so much memory?
> >> sheepdog at 77fa9112
> > 
> > Yes, "dog vdi check" needs amount of memory when it checks large
> > VDI. The problem is caused because dog doesn't manage worker threads
> > well. Could you add an issue to our tracker?
> > https://bugs.launchpad.net/sheepdog-project/+bugs
> > 
> > Solving this problem is not so difficult but I have other tasks,
> > sorry...
> 
> Hi Hitoshi!
> I've added to TODO:)
> https://bugs.launchpad.net/sheepdog-project/+bug/1325419
> Thanks, Marcin

Looks good, thanks :)
Hitoshi