[sheepdog] [sheepdog/sheepdog] bd1ce8: sheep: Validate -p/--port, -z/--zone and -V/--vnod...

Wed Feb 8 06:52:22 CET 2017

  Branch: refs/heads/stable-0.9
  Home:   https://github.com/sheepdog/sheepdog
  Commit: bd1ce84ae65795d1eeb0427b833bde6228b1e9b1
      https://github.com/sheepdog/sheepdog/commit/bd1ce84ae65795d1eeb0427b833bde6228b1e9b1
  Author: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
  Date:   2017-01-24 (Tue, 24 Jan 2017)

  Changed paths:
    M include/util.h
    M lib/util.c
    M sheep/sheep.c

  Log Message:
  -----------
  sheep: Validate -p/--port, -z/--zone and -V/--vnodes values strictly

This commit adds two functions str_to_u32 and str_to_u16 to lib/util.c.
These functions convert a string into a 64-bit integer with strtoll.
Then they check whether a converted value is in the valid range or not and,
if something goes wrong because of an empty string, an invalid character,
a negative value or overflow, sets errno accordingly.

Using them, this enforces strict range validation of -p/--port, -z/--zone
and -V/--vnodes.

Signed-off-by: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
(cherry picked from commit c19170a25b0a08997e2786c4e9784d9e1604ed39)

Conflicts:
	include/util.h
	sheep/sheep.c

  Commit: 08d47496f8dd73258c45299209588f537a3ee812
      https://github.com/sheepdog/sheepdog/commit/08d47496f8dd73258c45299209588f537a3ee812
  Author: Hitoshi Mitake <mitake.hitoshi at lab.ntt.co.jp>
  Date:   2017-01-24 (Tue, 24 Jan 2017)

  Changed paths:
    M include/work.h
    M lib/work.c

  Log Message:
  -----------
  lib: a new workqueue type WQ_FIXED

The main target of the new type is experimental purpose
(e.g. benchmarking, scalability evaluation).

Signed-off-by: Hitoshi Mitake <mitake.hitoshi at lab.ntt.co.jp>
(cherry picked from commit 9f0bd44c2acacd21488f084d0b9801cbaaf7cc94)

  Commit: 93b19f455e54df7d65e4a061283b9838259dd0cd
      https://github.com/sheepdog/sheepdog/commit/93b19f455e54df7d65e4a061283b9838259dd0cd
  Author: Hitoshi Mitake <mitake.hitoshi at lab.ntt.co.jp>
  Date:   2017-01-24 (Tue, 24 Jan 2017)

  Changed paths:
    M sheep/sheep.c

  Log Message:
  -----------
  sheep: add a new option for using fixed workqueues for stable performance

This commit adds a new option -w to sheep for using fixed workqueues.
Example of usage:
$ sheep -w gway=100,net=300

If the option isn't passed, sheep uses unlimited queues but it is
highly dicouraged for stable and high IOPS.

Based on @luckychen 's report, the patch doubles IOPS with well
configured parameters: 40,000 -> 83,460 IOPS

Signed-off-by: Hitoshi Mitake <mitake.hitoshi at lab.ntt.co.jp>
(cherry picked from commit c84b4fc1021df87bba161b758fb068a4333b9ad6)

Conflicts:
	sheep/sheep.c

  Commit: d2b2cbdddafeb643f9565ce0e3f42cc7e90dd1c3
      https://github.com/sheepdog/sheepdog/commit/d2b2cbdddafeb643f9565ce0e3f42cc7e90dd1c3
  Author: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
  Date:   2017-01-24 (Tue, 24 Jan 2017)

  Changed paths:
    M sheep/sheep.c

  Log Message:
  -----------
  sheep: change fixed workqueue option from -w to -q

Because -w is already used for object cache option in stable-0.9.
Note that this commit is only for stable-0.9.

Signed-off-by: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>

  Commit: d7d692e483f12baffb2281852220ae008c6be462
      https://github.com/sheepdog/sheepdog/commit/d7d692e483f12baffb2281852220ae008c6be462
  Author: Hitoshi Mitake <mitake.hitoshi at gmail.com>
  Date:   2017-01-24 (Tue, 24 Jan 2017)

  Changed paths:
    M sheep/sheep.c

  Log Message:
  -----------
  sheep: create correct work queues

Signed-off-by: Hitoshi Mitake <mitake.hitoshi at lab.ntt.co.jp>
(cherry picked from commit 4f967e9523db5a9be353629a805cf426ff1c821c)

  Commit: d464634f2a59e07267cee9138ba76ba5c296f383
      https://github.com/sheepdog/sheepdog/commit/d464634f2a59e07267cee9138ba76ba5c296f383
  Author: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
  Date:   2017-01-24 (Tue, 24 Jan 2017)

  Changed paths:
    M include/work.h
    M lib/work.c
    M sheep/sheep.c

  Log Message:
  -----------
  Remove WQ_UNLIMITED and change the limit of threads for WQ_DYNAMIC

This patch deletes WQ_UNLIMITED option because, with this option,
sheep may consume a very large amount of memory (in some cases,
to be killed by OOM-killer) by creating worker threads infinitely.

This patch also changes the upper limit for the number of threads
for WQ_DYNAMIC option from "#nodes*2" to "max(#nodes,#cores,16)*2".

Signed-off-by: Teruaki Ishizaki <ishizaki.teruaki at lab.ntt.co.jp>
Signed-off-by: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
(cherry picked from commit cbe755af178566fa2aa8b0b18e2b8d50c1e74ec7)

Conflicts:
	dog/benchmark.c
	tests/unit/lib/test_work.c
	tests/unit/sheep/test_recovery.c

  Commit: 371f772f0d9effda0b2eddaf7fbb34ed49b36b7c
      https://github.com/sheepdog/sheepdog/commit/371f772f0d9effda0b2eddaf7fbb34ed49b36b7c
  Author: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
  Date:   2017-01-24 (Tue, 24 Jan 2017)

  Changed paths:
    M include/work.h
    M lib/work.c
    M sheep/sheep.c

  Log Message:
  -----------
  sheep: add a new option for setting threads for dynamic workqueues

This commit adds a new option -x to sheep for configuring the maximum
number of threads for dynamic workqueues.

Example:

  # Set the max number of threads for dynamic workqueues to 500
  $ sheep -x 500

If this is not passed, sheep uses a default formula for configuring
the maximum number (that is, max(#nodes,#cores,16)*2).

This is for tuning performance, so please be careful to use. Sheep
works very slowly if you give a too small number, or may be shot by
OOM-killer under heavy load if you give a huge number.

Signed-off-by: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
(cherry picked from commit dd924bdb67b0f5fa668afb24a1152be0c93b1796)

Conflicts:
	lib/work.c

  Commit: fc7d12dc8a15896ece5d3c47a5a7dbf7d6e00629
      https://github.com/sheepdog/sheepdog/commit/fc7d12dc8a15896ece5d3c47a5a7dbf7d6e00629
  Author: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
  Date:   2017-01-24 (Tue, 24 Jan 2017)

  Changed paths:
    M lib/work.c

  Log Message:
  -----------
  sheep: fix logic for setting the max number of threads for dynamic WQ

This commit makes a minor change on "doubling growth". If the double
of the current number of threads for a dynamic queue exceeds the
maximum number, that workqueue grows into that maximum number.

Signed-off-by: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
(cherry picked from commit c42b9d5f0cd5998911cc271f31dbb934380c4ddf)

  Commit: 28a2b128aa7654e4166100b8a16917c52ee211a8
      https://github.com/sheepdog/sheepdog/commit/28a2b128aa7654e4166100b8a16917c52ee211a8
  Author: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
  Date:   2017-01-24 (Tue, 24 Jan 2017)

  Changed paths:
    M include/util.h
    M include/work.h
    M lib/util.c
    M lib/work.c
    M sheep/sheep.c

  Log Message:
  -----------
  Merge branch 'for-v0.9.5-fix-max-threads-2' into for-v0.9.5-2

  Commit: bdb9f11d726ddfe732580345c685ad73418afc55
      https://github.com/sheepdog/sheepdog/commit/bdb9f11d726ddfe732580345c685ad73418afc55
  Author: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
  Date:   2017-01-24 (Tue, 24 Jan 2017)

  Changed paths:
    M sheep/ops.c

  Log Message:
  -----------
  sheep: remove object list cache of ledger when removed from store

Each object list cache should be removed when it is removed from
object store. Inode and data ones are done so by peer_remove_obj.
However, ledger objects are not so by peer_decref_object, and
remaining cache leads to unnecessary recovery trial and error log.

This commit lets sheep remove object list cache of ledger when it is
removed from store.

Fix #341

Signed-off-by: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
(cherry picked from commit 254730a6992f74ad635db4417e493de4b45afe6b)

  Commit: 4693ab4940f97a20c9613a3bce8f21c319039f4e
      https://github.com/sheepdog/sheepdog/commit/4693ab4940f97a20c9613a3bce8f21c319039f4e
  Author: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
  Date:   2017-01-24 (Tue, 24 Jan 2017)

  Changed paths:
    M dog/common.c
    M dog/farm/farm.c
    M dog/farm/object_tree.c
    M dog/vdi.c
    M lib/sd_inode.c
    M sheep/gateway.c
    M sheep/http/kv.c
    M sheep/http/oalloc.c
    M sheep/nfs/fs.c
    M sheep/nfs/nfs.c
    M sheep/object_list_cache.c
    M sheep/ops.c
    M sheep/recovery.c
    M sheep/request.c
    M sheep/store/common.c
    M sheep/store/plain_store.c
    M sheep/store/tree_store.c
    M sheep/vdi.c
    M sheepfs/volume.c
    M tests/functional/028.out
    M tests/functional/042.out
    M tests/functional/080.out

  Log Message:
  -----------
  Print object ID in '"%016" PRIx64'

This commit lets Sheepdog print object ID in 16-digit zero-padded
hexadecimal, that is, '"%016" PRIx64'.

I think this is operator-friendly because s/he can tell whether
that object is a data object (00...), an inode object (80...), a
ledger object (08...) and so on, simply by looking at the printed.

Fix #340

Signed-off-by: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
(cherry picked from commit e882d76dd632168f680dc31af81cfbdbc42225d9)

Conflicts:
	dog/benchmark.c
	sheep/gateway.c
	sheep/group.c
	sheep/recovery.c
	sheep/request.c
	sheep/vdi.c

  Commit: fecd47a8c206c5a985967f681f79920a94f8cb2c
      https://github.com/sheepdog/sheepdog/commit/fecd47a8c206c5a985967f681f79920a94f8cb2c
  Author: Kazuhisa Hara <khara at sios.com>
  Date:   2017-02-08 (Wed, 08 Feb 2017)

  Changed paths:
    M sheepdog.spec.in

  Log Message:
  -----------
  rpm: support for systemd and sysvinit

In the current spec.in, rpmbuild fails in sysvinit distribution(ex: el6).
Therefore, set behavior in the case of systemd, and explicitly divide it
from sysvinit behavior.

Signed-off-by: Kazuhisa Hara <khara at sios.com>
(cherry picked from commit 454b84a0dc7f61bccff5c435f7a742c9bd14e94f)

Conflicts:
	sheepdog.spec.in

  Commit: 37f282d129c3afe54a0c73cf48cd98ff999a0c9b
      https://github.com/sheepdog/sheepdog/commit/37f282d129c3afe54a0c73cf48cd98ff999a0c9b
  Author: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>
  Date:   2017-02-08 (Wed, 08 Feb 2017)

  Changed paths:
    M CHANGELOG.md
    M configure.ac

  Log Message:
  -----------
  sheepdog stable release v0.9.5_rc0

Signed-off-by: Takashi Menjo <menjo.takashi at lab.ntt.co.jp>

Compare: https://github.com/sheepdog/sheepdog/compare/144ea5d24df1...37f282d129c3