From patchwork Tue Jan 24 17:30:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 18555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6CE7C54EB4 for ; Tue, 24 Jan 2023 17:30:20 +0000 (UTC) Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [217.70.178.231]) by mx.groups.io with SMTP id smtpd.web11.21807.1674581410995931484 for ; Tue, 24 Jan 2023 09:30:11 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=TUQJuuw+; spf=pass (domain: bootlin.com, ip: 217.70.178.231, mailfrom: alexis.lothore@bootlin.com) Received: (Authenticated sender: alexis.lothore@bootlin.com) by mail.gandi.net (Postfix) with ESMTPSA id 6D23C100014; Tue, 24 Jan 2023 17:30:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1674581408; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0iqyCglTBYEymg7i9LJt+g77cXOAz9g9NYm0ZGvdJ8M=; b=TUQJuuw+Hq6RA7JsbtWwzHsK3qJ/BRp+ggHqCNAgCKzNBa0kWuNah/se5vPS8uU/KyslG5 XRdlS4xBDovdTwW+noIz3WfL5+akp5nckV11qHH3uu6xSNf7OKHQnzyMFz5pH7wfQL6mGO 0MMU8lnY8d0SWGPxt+dU60nZWPJlOLMoayKfbUseYCxzczzw8rPMPv58861oq0MQH1fuPb /qdFEfxsB35P+MQwbVf6KKXTn4iLCGGV3u6DizP1kope7evUr59ATkOCZegh2N0Ibsc56T 7eTlobQDIDMlcTPLb618CBl0YO/ZGNBUQvInqzzyMixR0mEmmZ1eJfvVV0hHww== From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: yocto@lists.yoctoproject.org Cc: alexandre.belloni@bootlin.com, thomas.petazzoni@bootlin.com, =?utf-8?q?A?= =?utf-8?q?lexis_Lothor=C3=A9?= Subject: [autobuilder][PATCH v3 1/5] scripts/send_qa_email.py: Rename send-qa-email to send_qa_email.py Date: Tue, 24 Jan 2023 18:30:13 +0100 Message-Id: <20230124173017.134017-2-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230124173017.134017-1-alexis.lothore@bootlin.com> References: <20230124173017.134017-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 24 Jan 2023 17:30:20 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/59047 This rename fixes send-qa-email regarding python guidelines, which allows to import the script content in another script (for unit testing purpose for example) Signed-off-by: Alexis Lothoré --- scripts/{send-qa-email => send_qa_email.py} | 0 scripts/shared-repo-unpack | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) rename scripts/{send-qa-email => send_qa_email.py} (100%) diff --git a/scripts/send-qa-email b/scripts/send_qa_email.py similarity index 100% rename from scripts/send-qa-email rename to scripts/send_qa_email.py diff --git a/scripts/shared-repo-unpack b/scripts/shared-repo-unpack index b826c87..8e7ec95 100755 --- a/scripts/shared-repo-unpack +++ b/scripts/shared-repo-unpack @@ -61,7 +61,7 @@ for repo in sorted(repos.keys()): utils.printheader("Copying in repo %s" % repo) utils.mkdir(targetsubdir) if args.target in ["a-full", "a-quick"]: - # full/quick need all repo data due to send-qa-email + # full/quick need all repo data due to send_qa_email.py subprocess.check_call(["tar", "-I", "zstd", "-C", targetsubdir, "-xf", "%s.tar.zst" % args.cache_dir]) else: subprocess.check_call(["tar", "-I", "zstd", "-C", targetsubdir, "-xf", "%s.tar.zst" % args.cache_dir, "./" + repo]) From patchwork Tue Jan 24 17:30:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 18553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2B87C54E94 for ; Tue, 24 Jan 2023 17:30:20 +0000 (UTC) Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [217.70.178.231]) by mx.groups.io with SMTP id smtpd.web10.21715.1674581411298237297 for ; Tue, 24 Jan 2023 09:30:11 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=EERe9Pdp; spf=pass (domain: bootlin.com, ip: 217.70.178.231, mailfrom: alexis.lothore@bootlin.com) Received: (Authenticated sender: alexis.lothore@bootlin.com) by mail.gandi.net (Postfix) with ESMTPSA id 158C610000C; Tue, 24 Jan 2023 17:30:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1674581410; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K9chiPRL9QPiOLWmtVRfP8n3+tuQpfvuY9gwD6GQ6ls=; b=EERe9Pdpo7uOqVykoesKLmwc8WTp23dRNJl25ka66u66Vf7Q60kVl+IdO88KHVFxg1tf6k mKtAyZGGzoj91jE9nXN8Hdmbq4IOyPtzk/XVIhwy1aErLKWR+rn6bLFr62tRwtMWbc+m1h Fdz4wEB90alKpmrKDL4ftnmobqmWyuFqLm9rzMTm0+Jx0rB8o8/h/Tssp/wSXOZMNnoE7E m9FDFc15YZFHxgIIdsYzNg3QpJSB+NC/5BLYxutXfgY+PuhXv6I0YVekfkqsLA1Mv7MvaE qLLBuRe/xxhT98n+AzXGwm7QeJFlC2tDSt6YGpaOTibwdhwIAPRKA5GeYxpJRA== From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: yocto@lists.yoctoproject.org Cc: alexandre.belloni@bootlin.com, thomas.petazzoni@bootlin.com, =?utf-8?q?A?= =?utf-8?q?lexis_Lothor=C3=A9?= Subject: [autobuilder][PATCH v3 2/5] scripts/send_qa_email.py: Wrap send_qa_email.py content in function Date: Tue, 24 Jan 2023 18:30:14 +0100 Message-Id: <20230124173017.134017-3-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230124173017.134017-1-alexis.lothore@bootlin.com> References: <20230124173017.134017-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 24 Jan 2023 17:30:20 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/59048 This wrapping allows importing send_qa_email.py in another script without executing the QA emailing routine. This is useful for unit testing the script internal functions Signed-off-by: Alexis Lothoré --- scripts/send_qa_email.py | 287 ++++++++++++++++++++------------------- 1 file changed, 145 insertions(+), 142 deletions(-) diff --git a/scripts/send_qa_email.py b/scripts/send_qa_email.py index d2f63dc..4023918 100755 --- a/scripts/send_qa_email.py +++ b/scripts/send_qa_email.py @@ -15,150 +15,153 @@ import tempfile import utils -parser = utils.ArgParser(description='Process test results and optionally send an email about the build to prompt QA to begin testing.') - -parser.add_argument('send', - help="True to send email, otherwise the script will display a message and exit") -parser.add_argument('repojson', - help="The json file containing the repositories to use") -parser.add_argument('sharedrepodir', - help="The shared repos directory (to resolve the repo revision hashes)") -parser.add_argument('-p', '--publish-dir', - action='store', - help="Where the artefacts were published") -parser.add_argument('-R', '--results-dir', - action='store', - help="Where the test results were published") -parser.add_argument('-r', '--release', - action='store', - help="The build/release 'name' for release purposes (optional)") - -args = parser.parse_args() - -scriptsdir = os.path.dirname(os.path.realpath(__file__)) -ourconfig = utils.loadconfig() - -with open(args.repojson) as f: - repos = json.load(f) - -resulttool = os.path.dirname(args.repojson) + "/build/scripts/resulttool" - -buildtoolsdir = os.path.dirname(args.repojson) + "/build/buildtools" -if os.path.exists(buildtoolsdir): - utils.enable_buildtools_tarball(buildtoolsdir) - -repodir = os.path.dirname(args.repojson) + "/build/repos" - -if 'poky' in repos and os.path.exists(resulttool) and args.results_dir: - # Need the finalised revisions (not 'HEAD') - targetrepodir = "%s/poky" % (repodir) - revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip() - branch = repos['poky']['branch'] - repo = repos['poky']['url'] - - extraopts = None - basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch) - if basebranch: - extraopts = " --branch %s --commit %s" % (branch, revision) - if comparebranch: - extraopts = extraopts + " --branch2 %s" % (comparebranch) - elif basebranch: - print("No comparision branch found, comparing to %s" % basebranch) - extraopts = extraopts + " --branch2 %s" % basebranch - - report = subprocess.check_output([resulttool, "report", args.results_dir]) - with open(args.results_dir + "/testresult-report.txt", "wb") as f: - f.write(report) - - tempdir = tempfile.mkdtemp(prefix='sendqaemail.') - try: - cloneopts = [] +def send_qa_email(): + parser = utils.ArgParser(description='Process test results and optionally send an email about the build to prompt QA to begin testing.') + + parser.add_argument('send', + help="True to send email, otherwise the script will display a message and exit") + parser.add_argument('repojson', + help="The json file containing the repositories to use") + parser.add_argument('sharedrepodir', + help="The shared repos directory (to resolve the repo revision hashes)") + parser.add_argument('-p', '--publish-dir', + action='store', + help="Where the artefacts were published") + parser.add_argument('-R', '--results-dir', + action='store', + help="Where the test results were published") + parser.add_argument('-r', '--release', + action='store', + help="The build/release 'name' for release purposes (optional)") + + args = parser.parse_args() + + scriptsdir = os.path.dirname(os.path.realpath(__file__)) + ourconfig = utils.loadconfig() + + with open(args.repojson) as f: + repos = json.load(f) + + resulttool = os.path.dirname(args.repojson) + "/build/scripts/resulttool" + + buildtoolsdir = os.path.dirname(args.repojson) + "/build/buildtools" + if os.path.exists(buildtoolsdir): + utils.enable_buildtools_tarball(buildtoolsdir) + + repodir = os.path.dirname(args.repojson) + "/build/repos" + + if 'poky' in repos and os.path.exists(resulttool) and args.results_dir: + # Need the finalised revisions (not 'HEAD') + targetrepodir = "%s/poky" % (repodir) + revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip() + branch = repos['poky']['branch'] + repo = repos['poky']['url'] + + extraopts = None + basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch) + if basebranch: + extraopts = " --branch %s --commit %s" % (branch, revision) if comparebranch: - cloneopts = ["--branch", comparebranch] + extraopts = extraopts + " --branch2 %s" % (comparebranch) elif basebranch: - cloneopts = ["--branch", basebranch] - try: - subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir, "--depth", "5"] + cloneopts) - except subprocess.CalledProcessError: - print("No comparision branch found, falling back to master") - subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir, "--depth", "5"]) + print("No comparision branch found, comparing to %s" % basebranch) + extraopts = extraopts + " --branch2 %s" % basebranch - # If the base comparision branch isn't present regression comparision won't work - # at least until we can tell the tool to ignore internal branch information - if basebranch: + report = subprocess.check_output([resulttool, "report", args.results_dir]) + with open(args.results_dir + "/testresult-report.txt", "wb") as f: + f.write(report) + + tempdir = tempfile.mkdtemp(prefix='sendqaemail.') + try: + cloneopts = [] + if comparebranch: + cloneopts = ["--branch", comparebranch] + elif basebranch: + cloneopts = ["--branch", basebranch] try: - subprocess.check_call(["git", "rev-parse", "--verify", basebranch], cwd=tempdir) + subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir, "--depth", "5"] + cloneopts) except subprocess.CalledProcessError: - # Doesn't exist so base it off master - # some older hosts don't have git branch old new - subprocess.check_call(["git", "checkout", "master"], cwd=tempdir) - subprocess.check_call(["git", "branch", basebranch], cwd=tempdir) - subprocess.check_call(["git", "checkout", basebranch], cwd=tempdir) - extraopts = None - - subprocess.check_call([resulttool, "store", args.results_dir, tempdir]) - if comparebranch: - subprocess.check_call(["git", "push", "--all", "--force"], cwd=tempdir) - subprocess.check_call(["git", "push", "--tags", "--force"], cwd=tempdir) - elif basebranch: - subprocess.check_call(["git", "push", "--all"], cwd=tempdir) - subprocess.check_call(["git", "push", "--tags"], cwd=tempdir) - - if extraopts: - regreport = subprocess.check_output([resulttool, "regression-git", tempdir] + extraopts.split()) - with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f: - f.write(regreport) - - finally: - subprocess.check_call(["rm", "-rf", tempdir]) - pass - -if args.send.lower() != 'true' or not args.publish_dir or not args.release: - utils.printheader("Not sending QA email") - sys.exit(0) - -buildhashes = "" -for repo in sorted(repos.keys()): - # gplv2 is no longer built/tested in master - if repo == "meta-gplv2": - continue - # Need the finalised revisions (not 'HEAD') - targetrepodir = "%s/%s" % (repodir, repo) - revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip() - buildhashes += "%s: %s\n" % (repo, revision) - -web_root = utils.getconfig('WEBPUBLISH_DIR', ourconfig) -web_url = utils.getconfig('WEBPUBLISH_URL', ourconfig) - -email = "" -mailto = utils.getconfig("QAMAIL_TO", ourconfig) -if mailto: - email += "To: " + mailto + "\n" -mailcc = utils.getconfig("QAMAIL_CC", ourconfig) -if mailcc: - email += "Cc: " + mailcc + "\n" -mailbcc = utils.getconfig("QAMAIL_BCC", ourconfig) -if mailbcc: - email += "Bcc: " + mailbcc + "\n" - -email += "Subject: " + "QA notification for completed autobuilder build (%s)\n" % args.release -email += '''\n -A build flagged for QA (%s) was completed on the autobuilder and is available at:\n\n - %s\n\n -Build hash information: \n -%s - -\nThis is an automated message from the Yocto Project Autobuilder\nGit: git://git.yoctoproject.org/yocto-autobuilder2\nEmail: richard.purdie@linuxfoundation.org\n - -''' % (args.release, args.publish_dir.replace(web_root, web_url), buildhashes) - -# Store a copy of the email in case it doesn't reach the lists -with open(os.path.join(args.publish_dir, "qa-email"), "wb") as qa_email: - qa_email.write(email.encode('utf-8')) - -utils.printheader("Sending QA email") -env = os.environ.copy() -# Many distros have sendmail in */sbin -env["PATH"] = env["PATH"] + ":/usr/sbin:/sbin" -subprocess.check_call('echo "' + email +' " | sendmail -t', shell=True, env=env) - + print("No comparision branch found, falling back to master") + subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir, "--depth", "5"]) + + # If the base comparision branch isn't present regression comparision won't work + # at least until we can tell the tool to ignore internal branch information + if basebranch: + try: + subprocess.check_call(["git", "rev-parse", "--verify", basebranch], cwd=tempdir) + except subprocess.CalledProcessError: + # Doesn't exist so base it off master + # some older hosts don't have git branch old new + subprocess.check_call(["git", "checkout", "master"], cwd=tempdir) + subprocess.check_call(["git", "branch", basebranch], cwd=tempdir) + subprocess.check_call(["git", "checkout", basebranch], cwd=tempdir) + extraopts = None + + subprocess.check_call([resulttool, "store", args.results_dir, tempdir]) + if comparebranch: + subprocess.check_call(["git", "push", "--all", "--force"], cwd=tempdir) + subprocess.check_call(["git", "push", "--tags", "--force"], cwd=tempdir) + elif basebranch: + subprocess.check_call(["git", "push", "--all"], cwd=tempdir) + subprocess.check_call(["git", "push", "--tags"], cwd=tempdir) + + if extraopts: + regreport = subprocess.check_output([resulttool, "regression-git", tempdir] + extraopts.split()) + with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f: + f.write(regreport) + + finally: + subprocess.check_call(["rm", "-rf", tempdir]) + pass + + if args.send.lower() != 'true' or not args.publish_dir or not args.release: + utils.printheader("Not sending QA email") + sys.exit(0) + + buildhashes = "" + for repo in sorted(repos.keys()): + # gplv2 is no longer built/tested in master + if repo == "meta-gplv2": + continue + # Need the finalised revisions (not 'HEAD') + targetrepodir = "%s/%s" % (repodir, repo) + revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip() + buildhashes += "%s: %s\n" % (repo, revision) + + web_root = utils.getconfig('WEBPUBLISH_DIR', ourconfig) + web_url = utils.getconfig('WEBPUBLISH_URL', ourconfig) + + email = "" + mailto = utils.getconfig("QAMAIL_TO", ourconfig) + if mailto: + email += "To: " + mailto + "\n" + mailcc = utils.getconfig("QAMAIL_CC", ourconfig) + if mailcc: + email += "Cc: " + mailcc + "\n" + mailbcc = utils.getconfig("QAMAIL_BCC", ourconfig) + if mailbcc: + email += "Bcc: " + mailbcc + "\n" + + email += "Subject: " + "QA notification for completed autobuilder build (%s)\n" % args.release + email += '''\n + A build flagged for QA (%s) was completed on the autobuilder and is available at:\n\n + %s\n\n + Build hash information: \n + %s + + \nThis is an automated message from the Yocto Project Autobuilder\nGit: git://git.yoctoproject.org/yocto-autobuilder2\nEmail: richard.purdie@linuxfoundation.org\n + + ''' % (args.release, args.publish_dir.replace(web_root, web_url), buildhashes) + + # Store a copy of the email in case it doesn't reach the lists + with open(os.path.join(args.publish_dir, "qa-email"), "wb") as qa_email: + qa_email.write(email.encode('utf-8')) + + utils.printheader("Sending QA email") + env = os.environ.copy() + # Many distros have sendmail in */sbin + env["PATH"] = env["PATH"] + ":/usr/sbin:/sbin" + subprocess.check_call('echo "' + email +' " | sendmail -t', shell=True, env=env) + +if __name__ == "__main__": + send_qa_email() From patchwork Tue Jan 24 17:30:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 18556 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E53F8C54EED for ; Tue, 24 Jan 2023 17:30:20 +0000 (UTC) Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [217.70.178.231]) by mx.groups.io with SMTP id smtpd.web10.21716.1674581412255942821 for ; Tue, 24 Jan 2023 09:30:12 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=phid/NXb; spf=pass (domain: bootlin.com, ip: 217.70.178.231, mailfrom: alexis.lothore@bootlin.com) Received: (Authenticated sender: alexis.lothore@bootlin.com) by mail.gandi.net (Postfix) with ESMTPSA id 1E58A10000A; Tue, 24 Jan 2023 17:30:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1674581410; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=scLRPPhNoq2NH4vsF/JuTur6g3LDUTeuH8fwBb8q3Jk=; b=phid/NXbKgxZPSOb5e+oOJehNwCbMydi+U9ud3UzhLpB8iLsZBpR6QAXFPpJgfI6/ujoml oMuqPzgzN0Pok3X7rsZaXcvn584iWlK/UK2zjIuimqd82NjxVhGpEgdZISPXJUUNLnzHLA DDT8S9HXi0bLld8duQnUgdwytI+aQCnYSkajAzDSrjQexLDCqC2W4PC4W26eTG69PmQyzE w0e0ELQcB0ZlXgehemrmtB3SHkMjqNmmag0lHBWQXQ/qQ8fKOSSWL6s3kjC78HZx+2ie3y 4L2XMjlejscwaGv4ZSvSSi6s7GHAeEuSS1bLWBKpRrL5uOx/hrs3h3fro+d6qg== From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: yocto@lists.yoctoproject.org Cc: alexandre.belloni@bootlin.com, thomas.petazzoni@bootlin.com, =?utf-8?q?A?= =?utf-8?q?lexis_Lothor=C3=A9?= Subject: [autobuilder][PATCH v3 3/5] scripts/send-qa-email: Generate regression reports against most relevant release Date: Tue, 24 Jan 2023 18:30:15 +0100 Message-Id: <20230124173017.134017-4-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230124173017.134017-1-alexis.lothore@bootlin.com> References: <20230124173017.134017-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 24 Jan 2023 17:30:20 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/59049 Instead of only generating regressions reports against HEAD of relevant branch, compute most relevant tag (ie : release) against which we can check for regressions. General rules introduced are the following : - milestone release is checked against previous milestone if possible, otherwise against major release - point release is checked against previous point release if possible, otherwise against major release - major release is checked against previous major release - a non release build is checked against base branch Examples : - 4.1.2.rc1 is checked against yocto-4.1.1 - 4.1.2 is checked against yocto-4.1.1 - 4.1.1.rc1 is checked against yocto-4.1 - 4.1.1 is checked against yocto-4.1 - 4.1 is checked against yocto-4.0 - 4.1.rc4 is checked against yocto-4.0 - 4.1_M2.rc1 is checked against 4.1_M1 - 4.1_M2 is checked against 4.1_M1 - 4.1_M1.rc1 is checked against yocto-4.0 - 4.1_M1 is checked against yocto-4.0 Signed-off-by: Alexis Lothoré --- scripts/send_qa_email.py | 86 +++++++++++++++++++++++++++++++++------- scripts/utils.py | 47 ++++++++++++++++++++++ 2 files changed, 118 insertions(+), 15 deletions(-) diff --git a/scripts/send_qa_email.py b/scripts/send_qa_email.py index 4023918..199fe4e 100755 --- a/scripts/send_qa_email.py +++ b/scripts/send_qa_email.py @@ -9,11 +9,79 @@ import json import os import sys import subprocess -import errno import tempfile +import re import utils +def is_non_release_version(version): + p = re.compile('\d{8}-\d+') + return p.match(version) is not None + +def get_previous_tag(targetrepodir, version): + previousversion = None + previousmilestone = None + if version: + if is_non_release_version(version): + return subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=targetrepodir).decode('utf-8').strip() + compareversion, comparemilestone, _ = utils.get_version_from_string(version) + compareversionminor = compareversion[-1] + # After ignoring rc part, if we get a minor to 0 on point release (e.g 4.0.0), + # reject last digit since such versions do not exist + if len(compareversion) == 3 and compareversionminor == 0: + compareversion = compareversion[:-1] + + # Process milestone if not first in current release + if comparemilestone and comparemilestone > 1: + previousversion = compareversion + previousmilestone = comparemilestone-1 + # Process first milestone or release if not first in major release + elif compareversionminor > 0: + previousversion = compareversion[:-1] + [compareversion[-1] - 1] + # Otherwise : format it as tag (which must exist) and search previous tag + else: + comparetagstring = utils.get_tag_from_version(compareversion, comparemilestone) + return subprocess.check_output(["git", "describe", "--abbrev=0", comparetagstring + "^"], cwd=targetrepodir).decode('utf-8').strip() + + return utils.get_tag_from_version(previousversion, previousmilestone) + + # All other cases : merely check against latest tag reachable + defaultbaseversion, _, _ = utils.get_version_from_string(subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=targetrepodir).decode('utf-8').strip()) + return utils.get_tag_from_version(defaultbaseversion, None) + +def get_sha1(targetrepodir, revision): + return subprocess.check_output(["git", "rev-list", "-n", "1", revision], cwd=targetrepodir).decode('utf-8').strip() + +def fetch_testresults(resultdir, revision): + rawtags = subprocess.check_output(["git", "ls-remote", "--refs", "--tags", "origin", f"*{revision}*"], cwd=resultdir).decode('utf-8').strip() + if not rawtags: + raise Exception(f"No reference found for commit {revision} in {resultdir}") + for ref in [rawtag.split()[1] for rawtag in rawtags.splitlines()]: + print(f"Fetching matching revisions: {ref}") + subprocess.check_call(["git", "fetch", "--depth", "1", "origin", f"{ref}:{ref}"], cwd=resultdir) + + +def generate_regression_report(resulttool, targetrepodir, basebranch, resultdir, outputdir, yoctoversion): + baseversion = get_previous_tag(targetrepodir, yoctoversion) + baserevision = get_sha1(targetrepodir, baseversion) + comparerevision = get_sha1(targetrepodir, basebranch) + print(f"Compare version : {basebranch} ({comparerevision})") + print(f"Base tag : {baseversion} ({baserevision})") + + try: + """ + Results directory is likely a shallow clone : + we need to fetch results corresponding to base revision before + running resulttool + """ + fetch_testresults(resultdir, baserevision) + regreport = subprocess.check_output([resulttool, "regression-git", "-B", basebranch, "--commit", baserevision, "--commit2", comparerevision, resultdir]) + with open(outputdir + "/testresult-regressions-report.txt", "wb") as f: + f.write(regreport) + except subprocess.CalledProcessError as e: + error = str(e) + print(f"Error while generating report between {baserevision} and {comparerevision} : {error}") + def send_qa_email(): parser = utils.ArgParser(description='Process test results and optionally send an email about the build to prompt QA to begin testing.') @@ -57,16 +125,7 @@ def send_qa_email(): branch = repos['poky']['branch'] repo = repos['poky']['url'] - extraopts = None basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch) - if basebranch: - extraopts = " --branch %s --commit %s" % (branch, revision) - if comparebranch: - extraopts = extraopts + " --branch2 %s" % (comparebranch) - elif basebranch: - print("No comparision branch found, comparing to %s" % basebranch) - extraopts = extraopts + " --branch2 %s" % basebranch - report = subprocess.check_output([resulttool, "report", args.results_dir]) with open(args.results_dir + "/testresult-report.txt", "wb") as f: f.write(report) @@ -95,7 +154,6 @@ def send_qa_email(): subprocess.check_call(["git", "checkout", "master"], cwd=tempdir) subprocess.check_call(["git", "branch", basebranch], cwd=tempdir) subprocess.check_call(["git", "checkout", basebranch], cwd=tempdir) - extraopts = None subprocess.check_call([resulttool, "store", args.results_dir, tempdir]) if comparebranch: @@ -105,10 +163,8 @@ def send_qa_email(): subprocess.check_call(["git", "push", "--all"], cwd=tempdir) subprocess.check_call(["git", "push", "--tags"], cwd=tempdir) - if extraopts: - regreport = subprocess.check_output([resulttool, "regression-git", tempdir] + extraopts.split()) - with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f: - f.write(regreport) + if basebranch: + generate_regression_report(resulttool, targetrepodir, basebranch, tempdir, args.results_dir, args.release) finally: subprocess.check_call(["rm", "-rf", tempdir]) diff --git a/scripts/utils.py b/scripts/utils.py index c0ad14e..444b3ab 100644 --- a/scripts/utils.py +++ b/scripts/utils.py @@ -478,3 +478,50 @@ def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False): pass subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"]) enable_buildtools_tarball(btdir) + +def get_string_from_version(version, milestone=None, rc=None): + """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists, + those are major releases + """ + if len(version) == 3 and version[-1] == 0: + version = version[:-1] + + result = ".".join(list(map(str, version))) + if milestone: + result += "_M" + str(milestone) + if rc: + result += ".rc" + str(rc) + return result + +def get_tag_from_version(version, milestone): + if not milestone: + return "yocto-" + get_string_from_version(version, milestone) + return get_string_from_version(version, milestone) + + +def get_version_from_string(raw_version): + """ Get version as list of int from raw_version. + + Raw version _can_ be prefixed by "yocto-", + Raw version _can_ be suffixed by "_MX" + Raw version _can_ be suffixed by ".rcY" + """ + version = None + milestone = None + rc = None + if raw_version[:6] == "yocto-": + raw_version = raw_version[6:] + raw_version = raw_version.split(".") + if raw_version[-1][:2] == "rc": + rc = int(raw_version[-1][-1]) + raw_version = raw_version[:-1] + if raw_version[-1][-3:-1] == "_M": + milestone = int(raw_version[-1][-1]) + raw_version = raw_version[:-1] + [raw_version[-1][:-3]] + version = list(map(int, raw_version)) + """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists, + those are major releases + """ + if len(version) == 3 and version[-1] == 0: + version = version[:-1] + return version, milestone, rc \ No newline at end of file From patchwork Tue Jan 24 17:30:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 18554 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E554FC61D97 for ; Tue, 24 Jan 2023 17:30:20 +0000 (UTC) Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [217.70.178.231]) by mx.groups.io with SMTP id smtpd.web11.21810.1674581412848687431 for ; Tue, 24 Jan 2023 09:30:13 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=JvUiWLH7; spf=pass (domain: bootlin.com, ip: 217.70.178.231, mailfrom: alexis.lothore@bootlin.com) Received: (Authenticated sender: alexis.lothore@bootlin.com) by mail.gandi.net (Postfix) with ESMTPSA id 18E8E100005; Tue, 24 Jan 2023 17:30:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1674581411; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O27gm8E4iexRV+sgKCmhhPoxhI1GxmAMVzkX6zlAsDg=; b=JvUiWLH7CD44hz5wYOJCViNS+358oTIN5FGuuYDsl7wQ6ttUwCTlfbgOldBtED284NkA+R y4v8HNXIzS5aPwyctPlU6fvB08E6d80ewHxIhg9oge6JdqLl5T6FilEqwz3MPZ2gi/ed/o jOyZFmErzU00FclfCyelk/6aspXUaABPT6n0u2YJg9MEWOWpfGKazVHtUSfgQ7buTVzHd6 V2XOTVUWDMFQght392TSQjd9lO5fOjNODqPbrUAx1gL65DiI2mY1hx3eqlrCD5MVpEMjNf 3hV5u4otkYA8RLfP8RSOgAT1Vj5s1h0oaqMsVEISe6tDeorZyFYc/WZVF/bcaQ== From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: yocto@lists.yoctoproject.org Cc: alexandre.belloni@bootlin.com, thomas.petazzoni@bootlin.com, =?utf-8?q?A?= =?utf-8?q?lexis_Lothor=C3=A9?= Subject: [autobuilder][PATCH v3 4/5] scripts/send_qa_email.py: add unit tests on previous version computation Date: Tue, 24 Jan 2023 18:30:16 +0100 Message-Id: <20230124173017.134017-5-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230124173017.134017-1-alexis.lothore@bootlin.com> References: <20230124173017.134017-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 24 Jan 2023 17:30:20 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/59050 The "previous version" computation bring many edge cases depending on the version under release. Add a basic test suite to validate currently implemented computation strategy and to prevent mistakes when ediiting it in the future Signed-off-by: Alexis Lothoré --- scripts/test_send_qa_email.py | 56 +++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100755 scripts/test_send_qa_email.py diff --git a/scripts/test_send_qa_email.py b/scripts/test_send_qa_email.py new file mode 100755 index 0000000..41e895e --- /dev/null +++ b/scripts/test_send_qa_email.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +""" +Test suite for send_qa_email.py + +The suite needs a valid poky clone to run since it will +fetch and return revisions from remote repository. To run the suite, +set POKY_PATH environment variable accordingly: +`POKY_PATH=~/src/poky ./scripts/test_send_qa_email.py` +""" +import os +import sys +import unittest +import send_qa_email + + +class TestVersion(unittest.TestCase): + test_data_get_version = [ + {"input": {"version": "4.1.2"}, "expected": "yocto-4.1.1"}, + {"input": {"version": "4.1"}, "expected": "yocto-4.0"}, + {"input": {"version": "4.1.1"}, "expected": "yocto-4.1"}, + {"input": {"version": "4.1_M2"}, "expected": "4.1_M1"}, + {"input": {"version": "4.1_M1"}, "expected": "yocto-4.0"}, + {"input": {"version": "4.1.1.rc1"}, "expected": "yocto-4.1"}, + {"input": {"version": "4.1.2.rc1"}, "expected": "yocto-4.1.1"}, + {"input": {"version": "4.1_M3.rc1"}, "expected": "4.1_M2"}, + {"input": {"version": "4.1_M3.rc4"}, "expected": "4.1_M2"}, + {"input": {"version": "4.1_M1.rc1"}, "expected": "yocto-4.0"}, + {"input": {"version": "4.1_M1.rc4"}, "expected": "yocto-4.0"}, + {"input": {"version": "4.1.rc4"}, "expected": "yocto-4.0"} + ] + + test_data_get_sha1 = [ + {"input": "yocto-4.0", "expected": "00cfdde791a0176c134f31e5a09eff725e75b905"}, + {"input": "4.1_M1", "expected": "95066dde6861ee08fdb505ab3e0422156cc24fae"}, + ] + + def test_versions(self): + for data in self.test_data_get_version: + test_name = data["input"]["version"] + with self.subTest(f"Test {test_name} previous tag"): + self.assertEqual(send_qa_email.get_previous_tag(os.environ.get( + "POKY_PATH"), data["input"]["version"]), data["expected"]) + + def test_get_sha1(self): + for data in self.test_data_get_sha1: + test_name = data["input"] + with self.subTest(f"Test SHA1 from {test_name}"): + self.assertEqual(send_qa_email.get_sha1(os.environ.get( + "POKY_PATH"), data["input"]), data["expected"]) + + +if __name__ == '__main__': + if os.environ.get("POKY_PATH") is None: + print("Please set POKY_PATH to proper poky clone location before running tests") + sys.exit(1) + unittest.main() From patchwork Tue Jan 24 17:30:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alexis_Lothor=C3=A9?= X-Patchwork-Id: 18552 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D65AFC54EAA for ; Tue, 24 Jan 2023 17:30:20 +0000 (UTC) Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [217.70.178.231]) by mx.groups.io with SMTP id smtpd.web10.21717.1674581413449122554 for ; Tue, 24 Jan 2023 09:30:13 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=gm1 header.b=A6y59hk4; spf=pass (domain: bootlin.com, ip: 217.70.178.231, mailfrom: alexis.lothore@bootlin.com) Received: (Authenticated sender: alexis.lothore@bootlin.com) by mail.gandi.net (Postfix) with ESMTPSA id AD347100011; Tue, 24 Jan 2023 17:30:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=gm1; t=1674581412; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VurhaKUgO6RioXU9lh3dsiTVfz4fhnavENsn7/i2svI=; b=A6y59hk4ClIpa0rZbWfV1zOTP0fg0U09I+AmPHBxDpGHuEzJiUwhNaNTEWx7IpgzaiH/0U 1fWrN+mqMZF0cqJ3dqrPsX+dZwax0hxPuG75Z/lF2u5mO2mThdXtvPK1wL8pVrXL520fVE 0mGiiA1P6UzoXcXR8rZVmcOtwlPTiniL77Fm4+gERkkNt5Zs9AZrbNu+yftuG7ktL9Outk gKgO9yIWED7pRAIgLpJ0wl87tuYBOFM8i1EX2I8cYj6X8dvvkCPEMrjDM+v4SYVQIFgmEp n5mqYrvrXizc9OQFT+ITl8rFX2nvpGWQUVRCHrfmVvBD+MV5ymafSw2kgEAe9Q== From: =?utf-8?q?Alexis_Lothor=C3=A9?= To: yocto@lists.yoctoproject.org Cc: alexandre.belloni@bootlin.com, thomas.petazzoni@bootlin.com, =?utf-8?q?A?= =?utf-8?q?lexis_Lothor=C3=A9?= Subject: [autobuilder][PATCH v3 5/5] scripts/send-qa-email: add symlink to preserve compatibility with autobuilder Date: Tue, 24 Jan 2023 18:30:17 +0100 Message-Id: <20230124173017.134017-6-alexis.lothore@bootlin.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230124173017.134017-1-alexis.lothore@bootlin.com> References: <20230124173017.134017-1-alexis.lothore@bootlin.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 24 Jan 2023 17:30:20 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/59051 Merely renaming send-qa-email to send_qa_email.py would imply to update yocto-autobuilder2 scripts, which can be daunting. In order to preserve compatibility with current configuration, add a symlink pointing to the new script Signed-off-by: Alexis Lothoré --- scripts/send-qa-email | 1 + 1 file changed, 1 insertion(+) create mode 120000 scripts/send-qa-email diff --git a/scripts/send-qa-email b/scripts/send-qa-email new file mode 120000 index 0000000..d82d07f --- /dev/null +++ b/scripts/send-qa-email @@ -0,0 +1 @@ +send_qa_email.py \ No newline at end of file