From patchwork Mon Feb 28 14:16:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 4440 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FECFC433F5 for ; Mon, 28 Feb 2022 14:16:39 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mx.groups.io with SMTP id smtpd.web08.20524.1646057797323338719 for ; Mon, 28 Feb 2022 06:16:38 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="unable to parse pub key" header.i=@intel.com header.s=intel header.b=SQLmvmvZ; spf=pass (domain: intel.com, ip: 192.55.52.136, mailfrom: anuj.mittal@intel.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646057797; x=1677593797; h=from:to:subject:date:message-id:mime-version: content-transfer-encoding; bh=Wt7iFn5q+TTY/K27FkCuyDaNGg+mKQ1v2rXDk3G6P4o=; b=SQLmvmvZj3xCMJsRdnhwgd7r9T2X1L84ozPKy3juwIXfV5oynA/fE9Aj ERFxAImeoG6r5LJ03Ennv8MHJi5a9ZVUXRWzRJc0caQwAuZ+z+1JuMCiD CrDt6K1DtqKAcVBqWDOyunmACLWuwGxyfdJw0W0ICVtJIiIhAvO67IOA/ Rt2S29e0mhHypahfTEE9L0iXQ1TLc71BtFIoUXMASTi5JbjnH68GvXJSP QIsz++zk7PCj7GYF+uehEfscBITF+RwhLqk8aivCFqYf3+T9HMopPVJxl Kxfw6b0n/5QoJcdO/coKPpYEl32zsS+ENg1mPxTRrtPc1AqE3qv5RR7Ax Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10271"; a="232862733" X-IronPort-AV: E=Sophos;i="5.90,142,1643702400"; d="scan'208";a="232862733" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2022 06:16:36 -0800 X-IronPort-AV: E=Sophos;i="5.90,142,1643702400"; d="scan'208";a="534459489" Received: from jtang17-mobl1.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.213.141.22]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2022 06:16:34 -0800 From: Anuj Mittal To: yocto@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][hardknott 1/3] scripts/prepare-shared-repos: Use tmpfs for speed Date: Mon, 28 Feb 2022 22:16:27 +0800 Message-Id: <20220228141629.7688-1-anuj.mittal@intel.com> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 28 Feb 2022 14:16:39 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/56317 From: Richard Purdie Signed-off-by: Richard Purdie (cherry picked from commit 298a10575851d501204fe1ee0d1dcbcec37a66cd) Signed-off-by: Anuj Mittal --- scripts/prepare-shared-repos | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos index 8400d09..1573f85 100755 --- a/scripts/prepare-shared-repos +++ b/scripts/prepare-shared-repos @@ -32,7 +32,7 @@ with open(args.repojson) as f: stashdir = utils.getconfig("REPO_STASH_DIR", ourconfig) -with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/tmp") as tempdir: +with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuild/tmp") as tempdir: for repo in sorted(repos.keys()): utils.printheader("Intially fetching repo %s" % repo) utils.fetchgitrepo(tempdir, repo, repos[repo], stashdir) From patchwork Mon Feb 28 14:16:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 4439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FF0EC433FE for ; Mon, 28 Feb 2022 14:16:39 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mx.groups.io with SMTP id smtpd.web11.20214.1646057798754817658 for ; Mon, 28 Feb 2022 06:16:38 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="unable to parse pub key" header.i=@intel.com header.s=intel header.b=Zy4G49lL; spf=pass (domain: intel.com, ip: 192.55.52.136, mailfrom: anuj.mittal@intel.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646057798; x=1677593798; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=Dj5sSLsJffMKqc5uawbFL+Hr0N7dreYEkSh6JujP1yY=; b=Zy4G49lL6rroQoA72fG1gJkQlQmOIZHe+tTpjP0AHs10dSCrTISU6UIi rVFk1r0MIjgS/T/SXRm8kItaqYTdXsnulpBWkQQlC3lWEh9I6I3Lk5+Q3 21sM9l37WiL6eJkW76vUPXXoxtN3HCHIif8qZZKe5axSNP1m/Et0xiI5T mkh2LfEUEz/eQNwA3lubhQbb5mFoeo6+NyDHdB57FP8tvEHKJhxefezd7 2UMI132XI5++x4QMEpSmGnvLGlEkx0QDFQITSyhh5R/dr+WTQ7kcwYXKY oiZoaHW6pw9Oirg2Bzhw/I6gRBqYr8Gzukspjd5lnBBnyw21crMWEzRRU g==; X-IronPort-AV: E=McAfee;i="6200,9189,10271"; a="232862741" X-IronPort-AV: E=Sophos;i="5.90,142,1643702400"; d="scan'208";a="232862741" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2022 06:16:37 -0800 X-IronPort-AV: E=Sophos;i="5.90,142,1643702400"; d="scan'208";a="534459502" Received: from jtang17-mobl1.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.213.141.22]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2022 06:16:36 -0800 From: Anuj Mittal To: yocto@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][hardknott 2/3] prepare-shared-repos: Make it clear when rsync starts in logs Date: Mon, 28 Feb 2022 22:16:28 +0800 Message-Id: <20220228141629.7688-2-anuj.mittal@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220228141629.7688-1-anuj.mittal@intel.com> References: <20220228141629.7688-1-anuj.mittal@intel.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 28 Feb 2022 14:16:39 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/56318 From: Richard Purdie Signed-off-by: Richard Purdie (cherry picked from commit 8e996a95a8902b40380dd477ecb606cc969cdee9) Signed-off-by: Anuj Mittal --- scripts/prepare-shared-repos | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos index 1573f85..c221e69 100755 --- a/scripts/prepare-shared-repos +++ b/scripts/prepare-shared-repos @@ -39,4 +39,5 @@ with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuil if args.publish_dir: utils.publishrepo(tempdir, repo, args.publish_dir) + utils.printheader("Running rsync") subprocess.check_call("rsync -a " + tempdir + "/* " + args.sharedsrcdir, shell=True) From patchwork Mon Feb 28 14:16:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mittal, Anuj" X-Patchwork-Id: 4441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B0BAC433EF for ; Mon, 28 Feb 2022 14:16:41 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mx.groups.io with SMTP id smtpd.web10.20077.1646057800355963486 for ; Mon, 28 Feb 2022 06:16:40 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="unable to parse pub key" header.i=@intel.com header.s=intel header.b=Us+pMB4S; spf=pass (domain: intel.com, ip: 192.55.52.136, mailfrom: anuj.mittal@intel.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646057800; x=1677593800; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=k9W4DJ5A61sw1DucsSVyR4zeBBDyPL9DIWxVMsJ3SBM=; b=Us+pMB4Sw1zqU923EJg49JEHvQrD9jB7T84SxUSYjHBJFSfSZJhHhGCo 6wBX5Kjw/hYTwfmB4Iu/doFlGc0CGSMcynSEQTDkuRYEUA+mK/8E9mpY1 3a1G1i6TMJTQ/XGOewgEwubWb2XVkN6y1b3ftkxBq9vQ4qm5fg0kHVmr2 MyGXZabQhmkmruf5fJMPuUHbO58ItG9NIfP7kwpeoJxwCxSBqjlNN9lqI JKIWCTDD5YuzdsmRrmCjdbRXNHRlpvyVlSnuPdnchOaAFcxPWGmJ/JhuQ VY4yOM9HyDq0Gsmbmy25nV5e13hgPEqpZS8i96LOlR2hHm5z6jeDGN3Kr g==; X-IronPort-AV: E=McAfee;i="6200,9189,10271"; a="232862745" X-IronPort-AV: E=Sophos;i="5.90,142,1643702400"; d="scan'208";a="232862745" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2022 06:16:40 -0800 X-IronPort-AV: E=Sophos;i="5.90,142,1643702400"; d="scan'208";a="534459507" Received: from jtang17-mobl1.gar.corp.intel.com (HELO anmitta2-mobl3.intel.com) ([10.213.141.22]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2022 06:16:38 -0800 From: Anuj Mittal To: yocto@lists.yoctoproject.org Subject: [yocto-autobuilder-helper][hardknott 3/3] shared-repos: Use tar instead of rsync for speed Date: Mon, 28 Feb 2022 22:16:29 +0800 Message-Id: <20220228141629.7688-3-anuj.mittal@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220228141629.7688-1-anuj.mittal@intel.com> References: <20220228141629.7688-1-anuj.mittal@intel.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 28 Feb 2022 14:16:41 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/yocto/message/56319 From: Richard Purdie The rysnc of 20,000 files (650MB) onto the nas is slow taking ~3 minutes at idle and worse at load. This is due to the number of files which is a pain point for NFS. This piece of the build is also a bottleneck since the rest of a build depends on it happening. If we switch to zstd compressed tar, it takes 2.49s. Other compression methods were much slower but zstd seems 'accptable' and speeds things up too. Signed-off-by: Richard Purdie (cherry picked from commit aff49e938ee34e1fc5a2954e3e22a4ca1ae9ac7b) Signed-off-by: Anuj Mittal --- scripts/prepare-shared-repos | 4 ++-- scripts/send-qa-email | 6 ++++-- scripts/shared-repo-unpack | 9 ++++++--- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos index c221e69..a5bc0da 100755 --- a/scripts/prepare-shared-repos +++ b/scripts/prepare-shared-repos @@ -39,5 +39,5 @@ with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuil if args.publish_dir: utils.publishrepo(tempdir, repo, args.publish_dir) - utils.printheader("Running rsync") - subprocess.check_call("rsync -a " + tempdir + "/* " + args.sharedsrcdir, shell=True) + utils.printheader("Creating shared src tarball") + subprocess.check_call("tar -I zstd -cf " + args.sharedsrcdir.rstrip("/") + ".tar.zst ./*", shell=True, cwd=tempdir) diff --git a/scripts/send-qa-email b/scripts/send-qa-email index 1b69307..bc594df 100755 --- a/scripts/send-qa-email +++ b/scripts/send-qa-email @@ -45,9 +45,11 @@ buildtoolsdir = os.path.dirname(args.repojson) + "/build/buildtools" if os.path.exists(buildtoolsdir): utils.enable_buildtools_tarball(buildtoolsdir) +repodir = os.path.dirname(args.repojson) + "/build/repos" + if 'poky' in repos and os.path.exists(resulttool) and args.results_dir: # Need the finalised revisions (not 'HEAD') - targetrepodir = "%s/poky" % (args.sharedrepodir) + targetrepodir = "%s/poky" % (repodir) revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip() branch = repos['poky']['branch'] repo = repos['poky']['url'] @@ -116,7 +118,7 @@ if args.send.lower() != 'true' or not args.publish_dir or not args.release: buildhashes = "" for repo in sorted(repos.keys()): # Need the finalised revisions (not 'HEAD') - targetrepodir = "%s/%s" % (args.sharedrepodir, repo) + targetrepodir = "%s/%s" % (repodir, repo) revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip() buildhashes += "%s: %s\n" % (repo, revision) diff --git a/scripts/shared-repo-unpack b/scripts/shared-repo-unpack index f08efa8..f7f87af 100755 --- a/scripts/shared-repo-unpack +++ b/scripts/shared-repo-unpack @@ -50,11 +50,14 @@ needrepos_baseddirs = [r.split('/')[0] for r in needrepos] for repo in sorted(repos.keys()): if repo not in needrepos_baseddirs: continue - targetrepodir = "%s/%s" % (targetsubdir, repo) if args.cache_dir: utils.printheader("Copying in repo %s" % repo) - utils.mkdir(targetrepodir) - subprocess.check_call(["rsync", "-a", "%s/%s" % (args.cache_dir, repo), targetsubdir]) + utils.mkdir(targetsubdir) + if args.target in ["a-full", "a-quick"]: + # full/quick need all repo data due to send-qa-email + subprocess.check_call(["tar", "-I", "zstd", "-C", targetsubdir, "-xf", "%s.tar.zst" % args.cache_dir]) + else: + subprocess.check_call(["tar", "-I", "zstd", "-C", targetsubdir, "-xf", "%s.tar.zst" % args.cache_dir, "./" + repo]) else: utils.printheader("Fetching repo %s" % repo) utils.fetchgitrepo(targetsubdir, repo, repos[repo], stashdir)