From patchwork Tue Feb 22 15:20:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 4065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF1E9C433F5 for ; Tue, 22 Feb 2022 15:20:29 +0000 (UTC) Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by mx.groups.io with SMTP id smtpd.web12.13827.1645543229158385353 for ; Tue, 22 Feb 2022 07:20:29 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=G27jr2Mb; spf=softfail (domain: sakoman.com, ip: 209.85.216.44, mailfrom: steve@sakoman.com) Received: by mail-pj1-f44.google.com with SMTP id cp23-20020a17090afb9700b001bbfe0fbe94so2969989pjb.3 for ; Tue, 22 Feb 2022 07:20:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=32YOcb7ettdBysa/MTBsL/T0Hgdmcp8kIEKpySYDfNc=; b=G27jr2MbBCrmg+Wjq/yHED+Ye0uGy7RTPdF0+JxX0xZ4LsGl/N5vs8p3Udi3FOQoON jnyDfNGCwBvoELI0sFlJ/CXaB24YmTfKcZBT74FyYi5UP9UVUDPjWPU/1vfe9rDNiMXe EhOK+hIptOF34cpzvPT+SBKiHiSVpeNd6j3gTEBCl/v+G9yiDhbLVXRmm3MaQjS2NDKe ZtDWWPBH5quLQ0cNBW3BPJ8C8tSUpOdEi2hirYdQhZ3A+xLQNkeFPrn4A/zDcHtk613W rCnqhyLSt5TFUPFnpowEKJ4YcJLJDhUodjMWuiKiqPUTfZygl7ktzu/Q4tpwna3Vk7E2 bz1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=32YOcb7ettdBysa/MTBsL/T0Hgdmcp8kIEKpySYDfNc=; b=su2Q62uEQxK+hY5vPSYeLNOJeqEiUOjNzUXBRgzDXjLeXPxdrYC1+q7uPK5RPLOtdb YIIkwxJ2NOBm2vw2CzuESckLdl9DW681bE/pTKDD0OruJFrmfw4fJG0BkITOjfEGiKRC Ed9jJXPcI9FYuYK5J+cDdv3niRI1s0fIbbRQd8R/teBcdCQ1R+jyX8t/4arGBCZNPqjt X4ptZ3iYZxF8a+CuSkuQ4x094QjYMKhYq2DjE8zpGxSd3zA4IuAxUF6RUu+rwfoZnY7g oXzh04tU38NRM0JR5a5lpfeMXBXuC6+y+d3paZ14v3+ow6+GrKzaeeSVF7+AgKBA3EjO lb+Q== X-Gm-Message-State: AOAM533SuBBQZWDdMkKgHjsnQ1AcAZ+eVpo32uGm6qMOfDdlqFYQ0dH5 vt9L3A5GGRdvdMdOiBU4Jz9u5+qLV1rFjbuy X-Google-Smtp-Source: ABdhPJzTIUaWx4iDLpA9wtkNOIbJDNlnu4ejwUgyKF+JH9cIcNW+83cnpEkOchjoiF20lmLTdOo1QQ== X-Received: by 2002:a17:90b:28d:b0:1bc:6db:5eb8 with SMTP id az13-20020a17090b028d00b001bc06db5eb8mr4617045pjb.155.1645543228262; Tue, 22 Feb 2022 07:20:28 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id i9sm3118160pjd.11.2022.02.22.07.20.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Feb 2022 07:20:27 -0800 (PST) From: Steve Sakoman To: yocto@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][dunfell 1/3] scripts/prepare-shared-repos: Use tmpfs for speed Date: Tue, 22 Feb 2022 05:20:12 -1000 Message-Id: <92e0019207d59a8b04f340e4b8c17efc430ef83c.1645543004.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 22 Feb 2022 15:20:29 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/56242 From: Richard Purdie Signed-off-by: Richard Purdie (cherry picked from commit 298a10575851d501204fe1ee0d1dcbcec37a66cd) Signed-off-by: Steve Sakoman --- scripts/prepare-shared-repos | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos index 8400d09..1573f85 100755 --- a/scripts/prepare-shared-repos +++ b/scripts/prepare-shared-repos @@ -32,7 +32,7 @@ with open(args.repojson) as f: stashdir = utils.getconfig("REPO_STASH_DIR", ourconfig) -with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/tmp") as tempdir: +with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuild/tmp") as tempdir: for repo in sorted(repos.keys()): utils.printheader("Intially fetching repo %s" % repo) utils.fetchgitrepo(tempdir, repo, repos[repo], stashdir) From patchwork Tue Feb 22 15:20:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 4066 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFAF3C433F5 for ; Tue, 22 Feb 2022 15:20:32 +0000 (UTC) Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by mx.groups.io with SMTP id smtpd.web10.13978.1645543231753603067 for ; Tue, 22 Feb 2022 07:20:31 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=kdrcMMDU; spf=softfail (domain: sakoman.com, ip: 209.85.210.174, mailfrom: steve@sakoman.com) Received: by mail-pf1-f174.google.com with SMTP id l19so12369126pfu.2 for ; Tue, 22 Feb 2022 07:20:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=P8ejh+o/b2I9hV0mll9tISxi0+3IBTCIaNQkdsDaHco=; b=kdrcMMDUnN1DU+XSCpB0CoQ78wMH1zTknqPPayx54z8ltueG69u9XW8lFPwwW6CrLg vcGY4h44/yealRc/7D+iOxyfDUvZcNS7+fiFAVCvRckGe0Ooo1naS2OLgNUrvlLcmGbF 5NogsIvYYfva5T/8Okg/rndZ1wUJiJ6M35dnS3jAKnsYJCu+hVi0cgjoih5a+a7pSH6A JALnUWPn99W3RJ22w8tTG8DzK1gp9dm6umBgwppmRxc37a1Dx03pXuWx2+zPyaKkcPxS j9N+h2uMVHNd6Hbfktw5KsT6YQzRBFm8EIyWUaiYH37ec8kKM/LnPUH0pQdggeTCBaqk Vrrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P8ejh+o/b2I9hV0mll9tISxi0+3IBTCIaNQkdsDaHco=; b=TogalETEOn+huAcd/dNtvgFZX3fT4JB09QHZ20+AcAsRy1IPY5EwUMS1HMh13K8CxR LmYZ+nWDHOtOJOhoAtU3WO8iOUASltk/c5OPKAvllsxTfQHg3sc340XH1Vs+eU2Oeg7C rtTgALUgywhb5Scy7ez/494Q96x7IbYeVq4vSSpsPRi9UB4/CMGOOWZHgVJ6JHzt+D/H KEXjzJroXj9hFLH11oLQ/E8u4+u//yx+hQAU763zpgbMh6SWouP9dlgAWpCqvRfF1OfE aqEVkM5vXaPO9ZxXPebP9dduxw3mYZnSbg5nXIqKdJeNwa8gqAw0TAC3u1japzwetDfc afWw== X-Gm-Message-State: AOAM532B2ofXQBwPEqbhic1efW9UhnklqkRt6eaBjoGe6jLqhgmoZI8P +8mXLVeOiMi8sSpw6SDXlHUk25d3QMioXIi7 X-Google-Smtp-Source: ABdhPJy7Qp6PtD/obVHGvz+dyQyJ4A5s7ZcmpldSCLcmz8skPYd+kzTmz8X6UiDqs0TkayVOaGLDGQ== X-Received: by 2002:a65:5347:0:b0:374:87b6:549b with SMTP id w7-20020a655347000000b0037487b6549bmr1898546pgr.537.1645543230694; Tue, 22 Feb 2022 07:20:30 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id i9sm3118160pjd.11.2022.02.22.07.20.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Feb 2022 07:20:30 -0800 (PST) From: Steve Sakoman To: yocto@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][dunfell 2/3] prepare-shared-repos: Make it clear when rsync starts in logs Date: Tue, 22 Feb 2022 05:20:13 -1000 Message-Id: <03d4d5afcaec6ec4990329e6db2b81654c9d8495.1645543004.git.steve@sakoman.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 22 Feb 2022 15:20:32 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/56243 From: Richard Purdie Signed-off-by: Richard Purdie (cherry picked from commit 8e996a95a8902b40380dd477ecb606cc969cdee9) Signed-off-by: Steve Sakoman --- scripts/prepare-shared-repos | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos index 1573f85..c221e69 100755 --- a/scripts/prepare-shared-repos +++ b/scripts/prepare-shared-repos @@ -39,4 +39,5 @@ with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuil if args.publish_dir: utils.publishrepo(tempdir, repo, args.publish_dir) + utils.printheader("Running rsync") subprocess.check_call("rsync -a " + tempdir + "/* " + args.sharedsrcdir, shell=True) From patchwork Tue Feb 22 15:20:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Sakoman X-Patchwork-Id: 4067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC009C433EF for ; Tue, 22 Feb 2022 15:20:35 +0000 (UTC) Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by mx.groups.io with SMTP id smtpd.web11.13911.1645543234562894526 for ; Tue, 22 Feb 2022 07:20:34 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@sakoman-com.20210112.gappssmtp.com header.s=20210112 header.b=oEpmGwHI; spf=softfail (domain: sakoman.com, ip: 209.85.215.181, mailfrom: steve@sakoman.com) Received: by mail-pg1-f181.google.com with SMTP id w37so10909489pga.7 for ; Tue, 22 Feb 2022 07:20:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sakoman-com.20210112.gappssmtp.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=KQBXUlxJKKgdZCWYbZXZsbQK67pDGYWyPQrQIqMCGoE=; b=oEpmGwHISb5NPQ5VUpNppOqH1g7xlAmqVznTvivz9FEkmEs7imCaitpU/m6hw4qPDV B+cZ8O30NAnh8b7/rCUQYl7SiBE/rOFwBeMCt6OP4sHHtL3f+cD8u8jYaCweMhRrPUDA Xg1Gf8Tcfs2/FPLw6x7vwcIg39cA5leC2cit+fYBu/fyhxla2+iGxUxeSiBvzH8jueX/ 5mK74Iz8wH/ysZpAvJX3OHbEwC42sNBX17jR13yXAmGsOp7RUfoJ7XxgpWTyuQUOrjfE bzF1sbOYkEWAaGt1rGcPJ1xaxYYaqfYC7uKvfHnegks2fAl2viAtFnqIovK4sb5YzCgm QDVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KQBXUlxJKKgdZCWYbZXZsbQK67pDGYWyPQrQIqMCGoE=; b=TNS4SBD+2qbZiGdym9LtvKJRDWdxEsUsljlOoq6MvYO2NWOykQLweVW9jtKn8rlegI I+73Y+UKSB2iETX9BFYOFirgNUb0VeOi6K7E9oUBN88RNp+0hDePA/SUPUukncSHOg1I +ep7Eh2TJzIMN7xndprqKDGoi9aE07wNYDTGe7+cItLqI0oHNOBnKqGIQ2eRxVYPxnYb +x6J/lA4tfBFG++uixx9GaUTBDUYr8SFo8/OtalPZ0CpAGZMnk0E4mnMLQogKwdTPB4R /jcA6moSs2clmoo0H7UyXeiRlAYjF2s86vDhYVTAw2Xpc4qBm4OsImLL9ohD6yNu/chQ h5yw== X-Gm-Message-State: AOAM532gLK9cp6FBp02eiYzD/620qDjj0tVgn+8adSmluGbZm41Evstx kBNKsC7mfI6gbZyIfBpJlFdZ8LTaLLTmZgQ2 X-Google-Smtp-Source: ABdhPJwVlcLoVre0Ix2xHY5MyFyEFc1viil2XzQ3SdIdk28YCzrpfbj8RMNeQvp8ocG/HykrQxdTqA== X-Received: by 2002:a63:6c01:0:b0:36f:b8de:31b4 with SMTP id h1-20020a636c01000000b0036fb8de31b4mr20020777pgc.603.1645543233598; Tue, 22 Feb 2022 07:20:33 -0800 (PST) Received: from hexa.router0800d9.com (dhcp-72-253-6-214.hawaiiantel.net. [72.253.6.214]) by smtp.gmail.com with ESMTPSA id i9sm3118160pjd.11.2022.02.22.07.20.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Feb 2022 07:20:33 -0800 (PST) From: Steve Sakoman To: yocto@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][dunfell 3/3] shared-repos: Use tar instead of rsync for speed Date: Tue, 22 Feb 2022 05:20:14 -1000 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 22 Feb 2022 15:20:35 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/56244 From: Richard Purdie The rysnc of 20,000 files (650MB) onto the nas is slow taking ~3 minutes at idle and worse at load. This is due to the number of files which is a pain point for NFS. This piece of the build is also a bottleneck since the rest of a build depends on it happening. If we switch to zstd compressed tar, it takes 2.49s. Other compression methods were much slower but zstd seems 'accptable' and speeds things up too. Signed-off-by: Richard Purdie (cherry picked from commit aff49e938ee34e1fc5a2954e3e22a4ca1ae9ac7b) Signed-off-by: Steve Sakoman --- scripts/prepare-shared-repos | 4 ++-- scripts/send-qa-email | 6 ++++-- scripts/shared-repo-unpack | 9 ++++++--- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos index c221e69..a5bc0da 100755 --- a/scripts/prepare-shared-repos +++ b/scripts/prepare-shared-repos @@ -39,5 +39,5 @@ with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuil if args.publish_dir: utils.publishrepo(tempdir, repo, args.publish_dir) - utils.printheader("Running rsync") - subprocess.check_call("rsync -a " + tempdir + "/* " + args.sharedsrcdir, shell=True) + utils.printheader("Creating shared src tarball") + subprocess.check_call("tar -I zstd -cf " + args.sharedsrcdir.rstrip("/") + ".tar.zst ./*", shell=True, cwd=tempdir) diff --git a/scripts/send-qa-email b/scripts/send-qa-email index 1b69307..bc594df 100755 --- a/scripts/send-qa-email +++ b/scripts/send-qa-email @@ -45,9 +45,11 @@ buildtoolsdir = os.path.dirname(args.repojson) + "/build/buildtools" if os.path.exists(buildtoolsdir): utils.enable_buildtools_tarball(buildtoolsdir) +repodir = os.path.dirname(args.repojson) + "/build/repos" + if 'poky' in repos and os.path.exists(resulttool) and args.results_dir: # Need the finalised revisions (not 'HEAD') - targetrepodir = "%s/poky" % (args.sharedrepodir) + targetrepodir = "%s/poky" % (repodir) revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip() branch = repos['poky']['branch'] repo = repos['poky']['url'] @@ -116,7 +118,7 @@ if args.send.lower() != 'true' or not args.publish_dir or not args.release: buildhashes = "" for repo in sorted(repos.keys()): # Need the finalised revisions (not 'HEAD') - targetrepodir = "%s/%s" % (args.sharedrepodir, repo) + targetrepodir = "%s/%s" % (repodir, repo) revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip() buildhashes += "%s: %s\n" % (repo, revision) diff --git a/scripts/shared-repo-unpack b/scripts/shared-repo-unpack index f08efa8..f7f87af 100755 --- a/scripts/shared-repo-unpack +++ b/scripts/shared-repo-unpack @@ -50,11 +50,14 @@ needrepos_baseddirs = [r.split('/')[0] for r in needrepos] for repo in sorted(repos.keys()): if repo not in needrepos_baseddirs: continue - targetrepodir = "%s/%s" % (targetsubdir, repo) if args.cache_dir: utils.printheader("Copying in repo %s" % repo) - utils.mkdir(targetrepodir) - subprocess.check_call(["rsync", "-a", "%s/%s" % (args.cache_dir, repo), targetsubdir]) + utils.mkdir(targetsubdir) + if args.target in ["a-full", "a-quick"]: + # full/quick need all repo data due to send-qa-email + subprocess.check_call(["tar", "-I", "zstd", "-C", targetsubdir, "-xf", "%s.tar.zst" % args.cache_dir]) + else: + subprocess.check_call(["tar", "-I", "zstd", "-C", targetsubdir, "-xf", "%s.tar.zst" % args.cache_dir, "./" + repo]) else: utils.printheader("Fetching repo %s" % repo) utils.fetchgitrepo(targetsubdir, repo, repos[repo], stashdir)