[sheepdog] [RFC v5 026/126] python: add commit-per-subsystem.py
Marc-André Lureau
marcandre.lureau at gmail.com
Fri Nov 8 22:18:18 CET 2019
Hi
On Fri, Oct 11, 2019 at 9:11 PM Vladimir Sementsov-Ogievskiy
<vsementsov at virtuozzo.com> wrote:
>
> Add script to automatically commit tree-wide changes per-subsystem.
Oh interesting! I guess it could use a --help or a larger commit
message to explain a bit what it does (I imagine from the rest of the
series, but someone looking at the script without context may wonder;)
You could also fix some pep8/pylint/pycodestyle
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov at virtuozzo.com>
> ---
>
> CC: Gerd Hoffmann <kraxel at redhat.com>
> CC: "Gonglei (Arei)" <arei.gonglei at huawei.com>
> CC: Eduardo Habkost <ehabkost at redhat.com>
> CC: Igor Mammedov <imammedo at redhat.com>
> CC: Laurent Vivier <lvivier at redhat.com>
> CC: Amit Shah <amit at kernel.org>
> CC: Kevin Wolf <kwolf at redhat.com>
> CC: Max Reitz <mreitz at redhat.com>
> CC: John Snow <jsnow at redhat.com>
> CC: Ari Sundholm <ari at tuxera.com>
> CC: Pavel Dovgalyuk <pavel.dovgaluk at ispras.ru>
> CC: Paolo Bonzini <pbonzini at redhat.com>
> CC: Stefan Hajnoczi <stefanha at redhat.com>
> CC: Fam Zheng <fam at euphon.net>
> CC: Stefan Weil <sw at weilnetz.de>
> CC: Ronnie Sahlberg <ronniesahlberg at gmail.com>
> CC: Peter Lieven <pl at kamp.de>
> CC: Eric Blake <eblake at redhat.com>
> CC: "Denis V. Lunev" <den at openvz.org>
> CC: Markus Armbruster <armbru at redhat.com>
> CC: Alberto Garcia <berto at igalia.com>
> CC: Jason Dillaman <dillaman at redhat.com>
> CC: Wen Congyang <wencongyang2 at huawei.com>
> CC: Xie Changlong <xiechanglong.d at gmail.com>
> CC: Liu Yuan <namei.unix at gmail.com>
> CC: "Richard W.M. Jones" <rjones at redhat.com>
> CC: Jeff Cody <codyprime at gmail.com>
> CC: "Marc-André Lureau" <marcandre.lureau at redhat.com>
> CC: "Daniel P. Berrangé" <berrange at redhat.com>
> CC: Richard Henderson <rth at twiddle.net>
> CC: Greg Kurz <groug at kaod.org>
> CC: "Michael S. Tsirkin" <mst at redhat.com>
> CC: Marcel Apfelbaum <marcel.apfelbaum at gmail.com>
> CC: Beniamino Galvani <b.galvani at gmail.com>
> CC: Peter Maydell <peter.maydell at linaro.org>
> CC: "Cédric Le Goater" <clg at kaod.org>
> CC: Andrew Jeffery <andrew at aj.id.au>
> CC: Joel Stanley <joel at jms.id.au>
> CC: Andrew Baumann <Andrew.Baumann at microsoft.com>
> CC: "Philippe Mathieu-Daudé" <philmd at redhat.com>
> CC: Antony Pavlov <antonynpavlov at gmail.com>
> CC: Jean-Christophe Dubois <jcd at tribudubois.net>
> CC: Peter Chubb <peter.chubb at nicta.com.au>
> CC: Subbaraya Sundeep <sundeep.lkml at gmail.com>
> CC: Eric Auger <eric.auger at redhat.com>
> CC: Alistair Francis <alistair at alistair23.me>
> CC: "Edgar E. Iglesias" <edgar.iglesias at gmail.com>
> CC: Stefano Stabellini <sstabellini at kernel.org>
> CC: Anthony Perard <anthony.perard at citrix.com>
> CC: Paul Durrant <paul at xen.org>
> CC: Paul Burton <pburton at wavecomp.com>
> CC: Aleksandar Rikalo <arikalo at wavecomp.com>
> CC: Chris Wulff <crwulff at gmail.com>
> CC: Marek Vasut <marex at denx.de>
> CC: David Gibson <david at gibson.dropbear.id.au>
> CC: Cornelia Huck <cohuck at redhat.com>
> CC: Halil Pasic <pasic at linux.ibm.com>
> CC: Christian Borntraeger <borntraeger at de.ibm.com>
> CC: "Hervé Poussineau" <hpoussin at reactos.org>
> CC: Xiao Guangrong <xiaoguangrong.eric at gmail.com>
> CC: Aurelien Jarno <aurelien at aurel32.net>
> CC: Aleksandar Markovic <amarkovic at wavecomp.com>
> CC: Mark Cave-Ayland <mark.cave-ayland at ilande.co.uk>
> CC: Jason Wang <jasowang at redhat.com>
> CC: Laszlo Ersek <lersek at redhat.com>
> CC: Yuval Shaia <yuval.shaia at oracle.com>
> CC: Palmer Dabbelt <palmer at sifive.com>
> CC: Sagar Karandikar <sagark at eecs.berkeley.edu>
> CC: Bastian Koppelmann <kbastian at mail.uni-paderborn.de>
> CC: David Hildenbrand <david at redhat.com>
> CC: Thomas Huth <thuth at redhat.com>
> CC: Eric Farman <farman at linux.ibm.com>
> CC: Matthew Rosato <mjrosato at linux.ibm.com>
> CC: Hannes Reinecke <hare at suse.com>
> CC: Michael Walle <michael at walle.cc>
> CC: Artyom Tarasenko <atar4qemu at gmail.com>
> CC: Stefan Berger <stefanb at linux.ibm.com>
> CC: Samuel Thibault <samuel.thibault at ens-lyon.org>
> CC: Alex Williamson <alex.williamson at redhat.com>
> CC: Tony Krowiak <akrowiak at linux.ibm.com>
> CC: Pierre Morel <pmorel at linux.ibm.com>
> CC: Michael Roth <mdroth at linux.vnet.ibm.com>
> CC: Hailiang Zhang <zhang.zhanghailiang at huawei.com>
> CC: Juan Quintela <quintela at redhat.com>
> CC: "Dr. David Alan Gilbert" <dgilbert at redhat.com>
> CC: Luigi Rizzo <rizzo at iet.unipi.it>
> CC: Giuseppe Lettieri <g.lettieri at iet.unipi.it>
> CC: Vincenzo Maffione <v.maffione at gmail.com>
> CC: Jan Kiszka <jan.kiszka at siemens.com>
> CC: Anthony Green <green at moxielogic.com>
> CC: Stafford Horne <shorne at gmail.com>
> CC: Guan Xuetao <gxt at mprc.pku.edu.cn>
> CC: Max Filippov <jcmvbkbc at gmail.com>
> CC: qemu-block at nongnu.org
> CC: integration at gluster.org
> CC: sheepdog at lists.wpkg.org
> CC: qemu-arm at nongnu.org
> CC: xen-devel at lists.xenproject.org
> CC: qemu-ppc at nongnu.org
> CC: qemu-s390x at nongnu.org
> CC: qemu-riscv at nongnu.org
>
> python/commit-per-subsystem.py | 204 +++++++++++++++++++++++++++++++++
> 1 file changed, 204 insertions(+)
> create mode 100755 python/commit-per-subsystem.py
>
> diff --git a/python/commit-per-subsystem.py b/python/commit-per-subsystem.py
> new file mode 100755
> index 0000000000..2ccf84cb15
> --- /dev/null
> +++ b/python/commit-per-subsystem.py
> @@ -0,0 +1,204 @@
> +#!/usr/bin/env python3
> +#
> +# Copyright (c) 2019 Virtuozzo International GmbH
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +import subprocess
> +import sys
> +import os
> +import glob
> +
> +
> +def git_add(pattern):
> + subprocess.run(['git', 'add', pattern])
> +
> +
> +def git_commit(msg):
> + subprocess.run(['git', 'commit', '-m', msg], capture_output=True)
> +
> +
> +def git_changed_files():
> + ret = subprocess.check_output(['git', 'diff', '--name-only'], encoding='utf-8').split('\n')
> + if ret[-1] == '':
> + del ret[-1]
> + return ret
> +
> +
> +maintainers = sys.argv[1]
> +message = sys.argv[2].strip()
> +
> +subsystem = None
> +
> +remap = {
> + 'Block layer core': 'block',
> + 'Block Jobs': 'block',
> + 'Dirty Bitmaps': 'block',
> + 'Block QAPI, monitor, command line': 'block',
> + 'Block I/O path': 'block',
> + 'Throttling infrastructure': 'block',
> + 'Architecture support': 's390x',
> + 'Guest CPU Cores (KVM)': 'kvm',
> + 'Guest CPU Cores (Xen)': 'xen',
> + 'Guest CPU cores (TCG)': 'tcg',
> + 'Network Block Device (NBD)': 'nbd',
> + 'Parallel NOR Flash devices': 'pflash',
> + 'Firmware configuration (fw_cfg)': 'fw_cfg',
> + 'Block SCSI subsystem': 'scsi',
> + 'Network device backends': 'net',
> + 'Netmap network backend': 'net',
> + 'Host Memory Backends': 'hostmem',
> + 'Cryptodev Backends': 'cryptodev',
> + 'QEMU Guest Agent': 'qga',
> + 'COLO Framework': 'colo',
> + 'Command line option argument parsing': 'cmdline',
> + 'Character device backends': 'chardev'
> +}
> +
> +
> +class Maintainers:
> + def add(self, subsystem, path, mapper, mapper_name, glob_count=1):
> + if subsystem in remap:
> + subsystem = remap[subsystem]
> + if subsystem not in self.subsystems:
> + self.subsystems.append(subsystem)
> +
> + if path[-1] == '/':
> + path = path[:-1]
> +
> + if path in mapper:
> + if mapper[path][1] == glob_count:
> + print('Warning: "{}" both in "{}" and "{}" in {} mapper with '
> + 'same glob-count={}. {} ignored for this path.'.format(
> + path, mapper[path][0], subsystem, mapper_name, glob_count,
> + subsystem))
> + return
> + if mapper[path][1] < glob_count:
> + # silently ignore worse match
> + return
> +
> + mapper[path] = (subsystem, glob_count)
> +
> + def __init__(self, file_name):
> + self.map_file = {}
> + self.map_glob_file = {}
> + self.map_dir = {}
> + self.map_glob_dir = {}
> + self.map_unmaintained_dir = {
> + 'python': ('python', 1),
> + 'hw/misc': ('misc', 1)
> + }
> + self.subsystems = ['python', 'misc']
> + subsystem = None
> +
> + with open(file_name) as f:
> + mode2 = False
> + prevline = ''
> + for line in f:
> + line = line.rstrip()
> + if not line:
> + continue
> + if len(line) >= 2 and line[1] == ':':
> + if line[0] == 'F':
> + fname = line[3:]
> + if fname in ['*', '*/']:
> + continue
> + if os.path.isfile(fname):
> + self.add(subsystem, fname, self.map_file, 'file')
> + elif os.path.isdir(fname):
> + self.add(subsystem, fname, self.map_dir, 'dir')
> + else:
> + paths = glob.glob(fname)
> + if not paths:
> + print('Warning: nothing corresponds to "{}"'.format(fname))
> + continue
> +
> + n = len(paths)
> + for f in paths:
> + if os.path.isfile(f):
> + self.add(subsystem, f, self.map_glob_file, 'glob-file', n)
> + else:
> + assert os.path.isdir(f)
> + self.add(subsystem, f, self.map_glob_dir, 'glob-dir', n)
> + elif line[:3] == '---':
> + subsystem = prevline
> + if subsystem == 'Devices':
> + mode2 = True
> + elif mode2:
> + subsystem = line
> + prevline = line
> +
> + def find_in_map_dir(self, file_name, mapper):
> + while file_name != '' and file_name not in mapper:
> + file_name = os.path.dirname(file_name)
> +
> + return None if file_name == '' else mapper[file_name][0]
> +
> + def find_in_map_file(self, file_name, mapper):
> + if file_name in mapper:
> + return mapper[file_name][0]
> +
> + def find_subsystem(self, file_name):
> + s = self.find_in_map_file(file_name, self.map_file)
> + if s is not None:
> + return s
> +
> + s = self.find_in_map_file(file_name, self.map_glob_file)
> + if s is not None:
> + return s
> +
> + s = self.find_in_map_dir(file_name, self.map_dir)
> + if s is not None:
> + return s
> +
> + s = self.find_in_map_dir(file_name, self.map_glob_dir)
> + if s is not None:
> + return s
> +
> + s = self.find_in_map_dir(file_name, self.map_unmaintained_dir)
> + if s is not None:
> + return s
> +
> + self.subsystems.append(file_name)
> + return file_name
> +
> +
> +def commit(subsystem):
> + msg = subsystem
> + if msg in remap:
> + msg = remap[msg]
> + msg += ': ' + message
> + git_commit(msg)
> +
> +mnt = Maintainers(maintainers)
> +res = {}
> +for f in git_changed_files():
> + s = mnt.find_subsystem(f)
> + if s in res:
> + res[s].append(f)
> + else:
> + res[s] = [f]
> +
> +for s in mnt.subsystems:
> + if s in res:
> + print(s)
> + for f in res[s]:
> + print(' ', f)
> +
> +for s in mnt.subsystems:
> + if s in res:
> + for f in res[s]:
> + git_add(f)
> + commit(s)
> --
> 2.21.0
>
>
--
Marc-André Lureau
More information about the sheepdog
mailing list