[sheepdog] [PATCH 0/2] avoid heavy disk I/O in the main thread
Hitoshi Mitake
mitake.hitoshi at gmail.com
Sun Oct 26 12:36:16 CET 2014
At Fri, 24 Oct 2014 22:52:43 +0900,
tuji wrote:
>
> Hi,
>
> > Tuji-san, I updated the avoid-blocking branch for reducing memory
> > consumption. Could you backport it to 0.7.6 and test again?
>
> I've tested last avoid-blocking onto 0.7.6.
> It seems ok.
> It was not avoid blocking completely, but it is getting better!
> (refer log1)
> I found that elapsed time of default_cleanup was 6sec for 1TB disk
> recovry.
>
> By the way, I found other blocking during this test. it was happen when
> start sheepdog process at new noder.
> Disk I/O was blocked since 19:51:40 to 19:52:40.
> 19:51:40 is start time of sheepdog by new node.
> 19:52:40 is the time when recovery was just started (refer log2)
>
> Because there are many object files, so I guess it needs long time to
> move obj to .stale and it cause blocking.
Thanks a lot for your testing and analysis. Could you stamp your
Tested-by tag to the v3 patchset?
I'll work on remove another blocking issue later.
Thanks,
Hitoshi
>
>
>
> ==log1==
> Oct 24 10:17:17 INFO [main] recover_object_main(624) object c7ab8e00001ada is recovered (263966/263967)
> Oct 24 10:17:17 INFO [main] recover_object_main(624) object c7ab8e0000027e is recovered (263967/263967)
> Oct 24 10:17:17 INFO [main] default_cleanup(168) cleaning pruge_state_dir
> Oct 24 10:17:23 INFO [main] default_cleanup(170) cleaning pruge_state_dir done
>
> procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ ---timestamp---
> 0 3 0 64512 64836 741012 0 0 12 58172 296 352 1 14 0 85 0 2014-10-24 10:17:10 JST
> 0 3 0 71696 64832 733860 0 0 112 55452 216 230 1 11 0 88 0 2014-10-24 10:17:11 JST
> 0 3 0 65252 64832 739952 0 0 0 51408 190 181 0 10 0 90 0 2014-10-24 10:17:12 JST
> 0 1 0 68480 64832 737216 0 0 296 62572 261 235 2 13 0 85 0 2014-10-24 10:17:13 JST
> 0 2 0 71328 64820 733684 0 0 0 37224 178 179 1 7 0 92 0 2014-10-24 10:17:14 JST
> 0 3 0 63264 64824 742696 0 0 0 30792 172 249 1 10 0 89 0 2014-10-24 10:17:15 JST
> 0 3 0 71816 64824 732916 0 0 0 24136 89 115 1 3 0 96 0 2014-10-24 10:17:16 JST
> 0 3 0 71956 64824 733332 0 0 0 9584 119 146 1 4 0 95 0 2014-10-24 10:17:17 JST
> 0 3 0 63524 64824 741704 0 0 0 21672 113 111 1 6 0 93 0 2014-10-24 10:17:18 JST
> 0 3 0 63508 64824 741700 0 0 0 0 62 79 1 1 0 98 0 2014-10-24 10:17:19 JST
> 0 3 0 63508 64824 741700 0 0 0 0 52 78 1 1 0 98 0 2014-10-24 10:17:20 JST
> 0 3 0 63508 64824 741700 0 0 0 0 50 82 1 1 0 98 0 2014-10-24 10:17:21 JST
> 0 3 0 63508 64824 741700 0 0 0 0 58 79 1 1 0 98 0 2014-10-24 10:17:22 JST
> 0 3 0 63508 64824 741700 0 0 0 0 52 76 1 1 0 98 0 2014-10-24 10:17:23 JST
> 2 2 0 69712 64820 734736 0 0 156 105840 273 164 1 21 0 77 1 2014-10-24 10:17:24 JST
> 2 3 0 62384 64828 741864 0 0 124 115948 407 366 1 27 0 71 1 2014-10-24 10:17:25 JST
> 2 1 0 62528 64828 743024 0 0 0 111620 369 308 0 23 0 77 0 2014-10-24 10:17:26 JST
> 0 2 0 63892 64824 741328 0 0 296 122968 361 271 1 25 0 73 1 2014-10-24 10:17:27 JST
> 0 2 0 62660 64820 742612 0 0 0 92744 300 201 1 20 0 79 0 2014-10-24 10:17:28 JST
> 0 2 0 71824 64816 733144 0 0 156 87192 276 177 2 18 0 80 0 2014-10-24 10:17:29 JST
> 0 3 0 71828 64820 732912 0 0 124 76140 264 191 1 17 0 82 0 2014-10-24 10:17:30 JST
> 0 3 0 67612 64816 737592 0 0 0 18264 109 129 1 4 0 95 0 2014-10-24 10:17:31 JST
> 0 3 0 67852 64816 738096 0 0 0 33264 169 171 1 9 0 90 0 2014-10-24 10:17:32 JST
> 0 3 0 60796 64816 744524 0 0 296 23184 129 152 1 5 0 94 0 2014-10-24 10:17:33 JST
>
>
> ==log2==
> Oct 23 19:51:39 INFO [main] md_add_disk(141) /home/sheepdog/obj, nr 1
> Oct 23 19:51:39 INFO [main] send_join_request(770) IPv4 ip:10.0.0.3 port:7000
> Oct 23 19:51:40 INFO [main] main(887) sheepdog daemon (version 0.7.6) started
> Oct 23 19:52:39 ALERT [rw] get_vdi_copy_number(104) copy number for 0 not found, set 3
> Oct 23 19:52:39 ALERT [rw] get_vdi_copy_number(104) copy number for 0 not found, set 3
> Oct 23 19:52:40 INFO [main] recover_object_main(624) object c7ab8e0002d944 is recovered (1/263967)
> Oct 23 19:52:41 INFO [main] recover_object_main(624) object c7ab8e000091ee is recovered (2/263967)
> Oct 23 19:52:41 INFO [main] recover_object_main(624) object c7ab8e0003af7f is recovered (3/263967)
> Oct 23 19:52:41 INFO [main] recover_object_main(624) object c7ab8e00025f80 is recovered (4/263967)
> Oct 23 19:52:41 INFO [main] recover_object_main(624) object c7ab8e00004d0c is recovered (5/263967)
> Oct 23 19:52:41 INFO [main] recover_object_main(624) object c7ab8e00004b06 is recovered (6/263967)
> Oct 23 19:52:41 INFO [main] recover_object_main(624) object c7ab8e0002df48 is recovered (7/263967)
> Oct 23 19:52:42 INFO [main] recover_object_main(624) object c7ab8e0003c7c3 is recovered (8/263967)
>
>
> >
> > Thanks,
> > Hitoshi
>
> **********************************************
> 株式会社エーティーワークス 辻 正博
> 〒106-6137
> 東京都港区六本木6-10-1 六本木ヒルズ森タワー37F
> TEL：03-3497-0505 FAX：03-3497-0508
> http://www.atworks.co.jp/
> E-mail: tuji at atworks.co.jp
> **********************************************
>
More information about the sheepdog
mailing list