From patchwork Tue May 7 11:46:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 43322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68A0EC25B4F for ; Tue, 7 May 2024 11:46:45 +0000 (UTC) Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by mx.groups.io with SMTP id smtpd.web11.9481.1715082404613855114 for ; Tue, 07 May 2024 04:46:44 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=lT1w1CKj; spf=pass (domain: gmail.com, ip: 209.85.218.44, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-a59ece5e18bso155562266b.2 for ; Tue, 07 May 2024 04:46:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715082403; x=1715687203; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Pb/HD8X57q4/zKrajzDDTirW/Am7nAjY/W0zVeVRmcA=; b=lT1w1CKjs9EC6yphX08jp4Pfs++tsXxskeCkjyL0vYr7xH2UYpxhpcXjQQpEMT6qOB GEEKW6q6ztNR0SvwQvRIAeUHvZsIr2Rii+k6VO5nLmjbjkWu39lv0/4zgDs4FYTrAvfP SvFOAaVk6mMozDGWzC0cWLIRHyCqhyGeyIxXM9Z+WiThVNGGPFm2EFrGhOQEf7XZTqt+ YA8KRVHh58hPfVE9FK8tJT4KJrovRarjOJVObwRFCickvlM3pxST+zbYfUVHMR4URlv0 90VeiY1yPeWyQ8SicwjXzNjVhTTQt1rsssdOJ0DSFGkjubRKRZhofBxSxleBu0o1qpzD m1Ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715082403; x=1715687203; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Pb/HD8X57q4/zKrajzDDTirW/Am7nAjY/W0zVeVRmcA=; b=F4N3hCWARn4qexB4Ir5cU0M1iIE2jlbPZYoo+HqitFW5PT9pBIOqWdewDI9P/ypJsz Ae1PCoOoRvR8LVvFai00vGk8McyeO7rSQPTAYMTwFpmC7quW7ygEOGnws/Rdc8fSr3sY ibClt7Q0E4ChCp4GVhSwBnposagzL81IUUeoDKALnyf3rkpoVmx/4yCmUDPoLW2EMD1U J4giR6azGby5/MoCY2NF/z/QjSoVWndFFtTTp9H+TEsRV1+570fpG1hQAM20vOFr9fzw 6Zv/meqhBpgx2yId5VfWmuEjdmQkFEle071FOkDFn/XuPpgNcJ3axXD+sDGTMmEWBPsC QPOg== X-Gm-Message-State: AOJu0YxydX6Ez5OK2nUW7ByydKjsg8VRwMtDtj2Ige6qM6jNnQD+IMtz UiB6pBGMLLwJV6xnsMHHxcOemAYN0F6LhcDuQyCxo1XKgEmuUZVIAlI96w== X-Google-Smtp-Source: AGHT+IGQ8yyxcwE6aIneN1dP1gLhVsBX34mDZqFuliP7FLaKVy2/q5BfcwwDPmQQwPNfgxkBx6UMOg== X-Received: by 2002:a50:cd93:0:b0:568:9b13:77fe with SMTP id p19-20020a50cd93000000b005689b1377femr11827426edi.37.1715082402661; Tue, 07 May 2024 04:46:42 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id et4-20020a056402378400b00572d255e342sm5268215edb.10.2024.05.07.04.46.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 04:46:42 -0700 (PDT) From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 1/9] bblayers/makesetup: raise exceptions when errors happen Date: Tue, 7 May 2024 13:46:21 +0200 Message-Id: <20240507114629.2591882-1-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 07 May 2024 11:46:45 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/199084 From: Alexander Kanavin Otherwise the calling code can only issue a generic, unhelpful erorr message, and it's difficult to tell what went wrong if logger.error output is obscured or redirected. Signed-off-by: Alexander Kanavin --- meta/lib/bblayers/makesetup.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py index 99d59737602..8148b0d9ff4 100644 --- a/meta/lib/bblayers/makesetup.py +++ b/meta/lib/bblayers/makesetup.py @@ -48,8 +48,9 @@ class MakeSetupPlugin(LayerPlugin): if l_name == 'workspace': continue if l_ismodified: - logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l_name,path=l_path)) - return + e = "Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l_name,path=l_path) + logger.error(e) + raise Exception(e) repo_path = oe.buildcfg.get_metadata_git_toplevel(l_path) if self._is_submodule(repo_path): @@ -63,8 +64,9 @@ class MakeSetupPlugin(LayerPlugin): if repo_path == destdir_repo: repos[repo_path]['contains_this_file'] = True if not repos[repo_path]['git-remote']['remotes'] and not repos[repo_path]['contains_this_file']: - logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) - return + e = "Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path) + logger.error(e) + raise Exception(e) top_path = os.path.commonpath([os.path.dirname(r) for r in repos.keys()]) From patchwork Tue May 7 11:46:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 43323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B465C25B4F for ; Tue, 7 May 2024 11:46:55 +0000 (UTC) Received: from mail-lf1-f43.google.com (mail-lf1-f43.google.com [209.85.167.43]) by mx.groups.io with SMTP id smtpd.web10.9477.1715082405680189281 for ; Tue, 07 May 2024 04:46:46 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=R89XfWOv; spf=pass (domain: gmail.com, ip: 209.85.167.43, mailfrom: alex.kanavin@gmail.com) Received: by mail-lf1-f43.google.com with SMTP id 2adb3069b0e04-51f4d2676d1so3373456e87.3 for ; Tue, 07 May 2024 04:46:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715082404; x=1715687204; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CW40iCFvydPp5MoN2fDg/DPqApO8GWg4L/F8hC/qqkA=; b=R89XfWOvt8DwMiNAp6Vp08j+HoVrtHQrjuz8VBCcvplBDjocsRGc/H9bH8dmBv1IuG sEnVCSU6be7VMgwQrWJErOm5WqIiGTu1WwuRizeRW8l4OtuB0OOYov2OhQcmBKtAT7ER ljg1geCL/36iKm/rw9XcR2dA/sVItRH1DieHviL40w6PYvqaBpf4/5y0fKUeUpvF0gV6 qOgavacS+Cac4vyo4+Ehd226/KWwmx7GZaBo8ycryYZXwzx0x+mtjsIFMaOH7765B3B2 UAgeE2Mrr2rRGkMuzomz7gbRHWZrqkb8MTxIUfcxodfa8CxddWiT4RT7skJSv9YTUmT+ WM4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715082404; x=1715687204; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CW40iCFvydPp5MoN2fDg/DPqApO8GWg4L/F8hC/qqkA=; b=MwXSmbgaFIcuHSedMBY7zpqHU/80o7dRerAtSBkncZfyDTzB9aY7ib4NeRwjXWP4fp I12Cx8/N/stKTIJWo6p7hBUnItEKniYninH65PpPQ4qcsaChVsb/xllaQJyu/QLQpiU+ +TYO2DkYq9n8wJERiGEsFHJhh7Ac0zozHUtOFbimkODWwXynqxEa5v0Kn8ioh6LBRbmS +LdWjrwuq7pNmPvIz85Nxsm47nxYVUY6qAQt1E5WKN0x5QuSSBHx7lS3fWzS/uH8qCVZ r2zZK9nth0LKn3ZhpK65FdHVlL8qh3hdoGYl15xI6gEgmNF1azqugWLP0IjHi3ZUHgXP D51Q== X-Gm-Message-State: AOJu0YwJgGWhNk9kWiDqrtEadaiwvgQobVTkQlEZISNVxEi1oNdRoUle ZvfPYFgwEnLaqR3Uo6oRzYq+U3sEa0VoKjjlNJuvfXgwwGRlwKm94KPafA== X-Google-Smtp-Source: AGHT+IG1umuQfqmZ6Xgd+SumBgG93hN8TfjA6+2ZyfdV2dYiPsUMxR3ckxG+GjkzdWn3y0aic0mHVQ== X-Received: by 2002:a19:4357:0:b0:51d:5e16:517a with SMTP id m23-20020a194357000000b0051d5e16517amr7402014lfj.48.1715082403260; Tue, 07 May 2024 04:46:43 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id et4-20020a056402378400b00572d255e342sm5268215edb.10.2024.05.07.04.46.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 04:46:43 -0700 (PDT) From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 2/9] bblayers/makesetup: include local repository paths in discovered layer repo data Date: Tue, 7 May 2024 13:46:22 +0200 Message-Id: <20240507114629.2591882-2-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240507114629.2591882-1-alex.kanavin@gmail.com> References: <20240507114629.2591882-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 07 May 2024 11:46:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/199085 From: Alexander Kanavin This can be useful when the caller needs to know where the repos actually are on local disk (for example to make a local copy of them or obtain additional information about them). This field however should be removed from the json output in oe-setup-layers plugin, as it is host-specific and not part of the schema. Signed-off-by: Alexander Kanavin --- meta/lib/bblayers/makesetup.py | 1 + meta/lib/bblayers/setupwriters/oe-setup-layers.py | 2 ++ 2 files changed, 3 insertions(+) diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py index 8148b0d9ff4..48dc73c288d 100644 --- a/meta/lib/bblayers/makesetup.py +++ b/meta/lib/bblayers/makesetup.py @@ -76,6 +76,7 @@ class MakeSetupPlugin(LayerPlugin): repos_nopaths[r_nopath] = repos[r] r_relpath = os.path.relpath(r, top_path) repos_nopaths[r_nopath]['path'] = r_relpath + repos_nopaths[r_nopath]['originpath'] = r return repos_nopaths def do_make_setup(self, args): diff --git a/meta/lib/bblayers/setupwriters/oe-setup-layers.py b/meta/lib/bblayers/setupwriters/oe-setup-layers.py index 59ca968ff33..56c929471e6 100644 --- a/meta/lib/bblayers/setupwriters/oe-setup-layers.py +++ b/meta/lib/bblayers/setupwriters/oe-setup-layers.py @@ -85,6 +85,8 @@ class OeSetupLayersWriter(): if not os.path.exists(args.destdir): os.makedirs(args.destdir) repos = parent.make_repo_config(args.destdir) + for r in repos.values(): + del r['originpath'] json = {"version":"1.0","sources":repos} if not repos: err = "Could not determine layer sources" From patchwork Tue May 7 11:46:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 43328 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94397C25B78 for ; Tue, 7 May 2024 11:46:55 +0000 (UTC) Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) by mx.groups.io with SMTP id smtpd.web10.9478.1715082405919969932 for ; Tue, 07 May 2024 04:46:46 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=J+4Uq12L; spf=pass (domain: gmail.com, ip: 209.85.167.46, mailfrom: alex.kanavin@gmail.com) Received: by mail-lf1-f46.google.com with SMTP id 2adb3069b0e04-51f0b6b682fso3120239e87.1 for ; Tue, 07 May 2024 04:46:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715082404; x=1715687204; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s3KjOQMiol6AhvXTwXnrEgF8B6xzHo9Kpiwd0eNmqHM=; b=J+4Uq12Lq+ej3uz5Ja8Rd0jXEek7x6t9RvOR/G3vCcWUDNsuKYjAvm0VUcTDEQ6bf8 wXd0mbRJHxzcioje1FXMkdD5WnpMkaDgJo9ztJjEdP5XoUP89lkjQNJZsBCQ+A8r1hGW qgvSuRV6ndyNO81plJfX0tkn5H7fzOgX0nqkVj2Azhh9ovxKpP4m90Av0uvCQ2o7e/H4 UdqvAO7iqQdGAivj0AXoR6WOTCeMY+MDsIZqtdUu4P7LSOEgUBwGVYgOSZA9eCXItRpm Omg4W/HUW6aDeqd3HWUDviRWtnarjgrcPa7enxrux3BHeISBYePWo2i8syfx4Uh1hiL/ NAhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715082404; x=1715687204; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s3KjOQMiol6AhvXTwXnrEgF8B6xzHo9Kpiwd0eNmqHM=; b=cf5JnYHz2I7KCnlEZx9SC31rExZnWxY/JINYwg3SPGrM7gHrkLbDCflL+iVoUFwoqJ R+sgW+1/+AzFJ9j88CNk67WgPqCKW9bP2Gl0SRB0y42Stlcif2ySeQNLfK7xNuomz6I/ gTubHufjUu6yM2yxhtWaiICMAwWu11nKqkdn86nLz7sA3HxGaVJddhjvZaON3lRJ+j6r bJZVQakkqqfMBAh6cZWddj+aGe8d6Rz7xy7piu3DFngsipa/wJfSf5OgyX2+kKNbAwJB bZgM7AJDgWy4DoxZIESmJlYuSeSmIi7594bdPxScpgD8WGaSPZOZgBsK2JylQTZrpMmW y1uQ== X-Gm-Message-State: AOJu0Yx7iHGKz3kru28MhQ5gip0+GWc6b8mqgEGM/Mr/bTqsRps/NKoa 4wbSHh2xHuzGrd3tUIeNQ0nYpq7OUzEEbgx9FL5TnPo9zBUe3ZqoJ9pbAQ== X-Google-Smtp-Source: AGHT+IEX5+1ObK4d0vxzBx6G3fMT0aqpgat8TRn1ZVXD9E8HjPBlrvuKa1I0MUUKKMA99LHpN/YZiQ== X-Received: by 2002:a05:6512:3d87:b0:51e:876d:17d0 with SMTP id k7-20020a0565123d8700b0051e876d17d0mr14404138lfv.52.1715082403835; Tue, 07 May 2024 04:46:43 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id et4-20020a056402378400b00572d255e342sm5268215edb.10.2024.05.07.04.46.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 04:46:43 -0700 (PDT) From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 3/9] bblayers/makesetup: move check for existence of git remotes to oe-setup-layers plugin Date: Tue, 7 May 2024 13:46:23 +0200 Message-Id: <20240507114629.2591882-3-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240507114629.2591882-1-alex.kanavin@gmail.com> References: <20240507114629.2591882-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 07 May 2024 11:46:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/199086 From: Alexander Kanavin This helps when making a local copy of all the layers: it's fine to not have any remotes configured in that case. Also correct the mistake in accessing 'contains_this_file' property: it may not necessarily exist, and if so, it should default to False. Signed-off-by: Alexander Kanavin --- meta/lib/bblayers/makesetup.py | 4 ---- meta/lib/bblayers/setupwriters/oe-setup-layers.py | 3 +++ 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py index 48dc73c288d..4199b5f0698 100644 --- a/meta/lib/bblayers/makesetup.py +++ b/meta/lib/bblayers/makesetup.py @@ -63,10 +63,6 @@ class MakeSetupPlugin(LayerPlugin): 'describe':oe.buildcfg.get_metadata_git_describe(repo_path)}} if repo_path == destdir_repo: repos[repo_path]['contains_this_file'] = True - if not repos[repo_path]['git-remote']['remotes'] and not repos[repo_path]['contains_this_file']: - e = "Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path) - logger.error(e) - raise Exception(e) top_path = os.path.commonpath([os.path.dirname(r) for r in repos.keys()]) diff --git a/meta/lib/bblayers/setupwriters/oe-setup-layers.py b/meta/lib/bblayers/setupwriters/oe-setup-layers.py index 56c929471e6..8faeabfabc8 100644 --- a/meta/lib/bblayers/setupwriters/oe-setup-layers.py +++ b/meta/lib/bblayers/setupwriters/oe-setup-layers.py @@ -86,6 +86,9 @@ class OeSetupLayersWriter(): os.makedirs(args.destdir) repos = parent.make_repo_config(args.destdir) for r in repos.values(): + if not r['git-remote']['remotes'] and not r.get('contains_this_file', False): + e = "Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=r['originpath']) + raise Exception(e) del r['originpath'] json = {"version":"1.0","sources":repos} if not repos: From patchwork Tue May 7 11:46:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 43330 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA95BC25B77 for ; Tue, 7 May 2024 11:46:55 +0000 (UTC) Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by mx.groups.io with SMTP id smtpd.web11.9483.1715082406382460271 for ; Tue, 07 May 2024 04:46:46 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=Ra6duQRx; spf=pass (domain: gmail.com, ip: 209.85.218.51, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f51.google.com with SMTP id a640c23a62f3a-a59ab4f60a6so649041366b.0 for ; Tue, 07 May 2024 04:46:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715082404; x=1715687204; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e+/zpwJd307h0VIOUObhyMPia3hdzN9VvluxH6MNJ/c=; b=Ra6duQRxTHqaIfsF/08hc1sy1MWeq16JebBD2HDAZR1xq1/gH+uaqww+Io+sivIuoO VO+Nxr6N1Aqc+AmcRVExCA4hPU7x1y9AosmjW45Ptj5QzP9ljr9I5/VFjJitVzo8BbnV Zi0ZMLFpTC7I8xzJa1JUJqmS2jkIDrBbxs5GT+8CRAA3arTk8lmJR7Ma+OdavnxMPj6S fNUGqTly0ZnR3mN8jgNQJXY/SPGpkkLeJ91YB/9hvmzPDF3Pn/ZsEjE2z6MfGizea0P0 0b3Vv3KfRpIc1TNR4hiRcenjXT6EEDulgx1KfQuuTl6TDhUSBm18gPewmp/hXfnEhPPC rUqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715082404; x=1715687204; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e+/zpwJd307h0VIOUObhyMPia3hdzN9VvluxH6MNJ/c=; b=RATBJj0oC+B+vgf2D28pXd2tvwaajwJf99J4DcgisdPM6/qsVib1dQurlbRfSlQBk+ vsu5sBprF2nCbknPKlrCAme8z3F1RjIet8lR6NC5lXX7BXUHTHYfkRSm46uGZBkeVoAf qlMdGkMh8gbfZYFxdpUkFUWsKmyY4LXFi+gACbRdf0TURPuQb5DetfY2CUPMUGaPuix/ hVbHEnmjbs+pNheAgke5p2mC8Kp5Gfh/0WCZLYB6NAwiwVo3v50cQh5O2re34aPuU6Bx UuPA+sNvvzvULfw0J9+Y0aOW8nMEJTbIptSMFnybEv9r7F6XvNRCkbWv0MlPnJfw8to8 jWVQ== X-Gm-Message-State: AOJu0YxxY2Cr8gUF0oi8hEticHk8EpqTzN35I9AVYJlFvXE0+eD5YPEw 4HJpeSlCuuaa5Z7SSh5iQRKXMeEHTO/MAXrO8NGCub3aXDJBwRJbr3pnlQ== X-Google-Smtp-Source: AGHT+IGVXoFh3lMdaQQd45ocQO42BweybTKADcZsW8/MVEHFg9t2o29dIkAknE+jFyRj5dQO2Wrt9Q== X-Received: by 2002:a50:aa95:0:b0:568:cdd8:cf60 with SMTP id q21-20020a50aa95000000b00568cdd8cf60mr12031380edc.8.1715082404411; Tue, 07 May 2024 04:46:44 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id et4-20020a056402378400b00572d255e342sm5268215edb.10.2024.05.07.04.46.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 04:46:44 -0700 (PDT) From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 4/9] bblayers/setupwriters/oe-local-copy: add a 'local copy' plugin for bitbake-layers create-layer-setup Date: Tue, 7 May 2024 13:46:24 +0200 Message-Id: <20240507114629.2591882-4-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240507114629.2591882-1-alex.kanavin@gmail.com> References: <20240507114629.2591882-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 07 May 2024 11:46:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/199087 From: Alexander Kanavin This plugin copies all currently configured layer respositories into a dedicated location on local disk. This is useful for entirely offline layer replication, when the layers are packed and then unpacked from an archive, rather than fetched from git (there can be situations where fetching from git is undesirable or impossible). This plugin will also be used when replicating an entire yocto build (with build config and sstate cache), and is an element of that. This plugin is similar to what esdk does, and it provides that functionality outside of populate_sdk_ext task. It does not reuse esdk code, as that simply copies the layer tree; it's better to use 'git clone' with a path to original repo on local disk, as that will preserve commit history in the copy. Signed-off-by: Alexander Kanavin --- .../bblayers/setupwriters/oe-local-copy.py | 41 +++++++++++++++++++ meta/lib/oeqa/selftest/cases/bblayers.py | 13 ++++++ 2 files changed, 54 insertions(+) create mode 100644 meta/lib/bblayers/setupwriters/oe-local-copy.py diff --git a/meta/lib/bblayers/setupwriters/oe-local-copy.py b/meta/lib/bblayers/setupwriters/oe-local-copy.py new file mode 100644 index 00000000000..8c1ccb67368 --- /dev/null +++ b/meta/lib/bblayers/setupwriters/oe-local-copy.py @@ -0,0 +1,41 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import json + +logger = logging.getLogger('bitbake-layers') + +def plugin_init(plugins): + return OeLocalCopyWriter() + +class OeLocalCopyWriter(): + + def __str__(self): + return "oe-local-copy" + + def do_write(self, parent, args): + """ Writes out a local copy of all the metadata layers (and bitbake) included in a current build. """ + if not os.path.exists(args.destdir): + os.makedirs(args.destdir) + repos = parent.make_repo_config(args.destdir) + if not repos: + raise Exception("Could not determine layer sources") + output = os.path.join(os.path.abspath(args.destdir), args.output_prefix or "layers") + json_f = os.path.join(os.path.abspath(args.destdir), "bundle-repos.json") + + for r in repos.values(): + r['git-remote']['remotes'] = {"origin":{"uri":r["originpath"]}} + + json_data = {"version":"1.0","sources":repos} + with open(json_f, 'w') as f: + json.dump(json_data, f, sort_keys=True, indent=4) + + logger.info("Cloning repositories into {}".format(output)) + bb.process.run('oe-setup-layers --force-bootstraplayer-checkout --destdir {} --jsondata {}'.format(output, json_f)) + + def register_arguments(self, parser): + pass diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 695d17377d4..8b2bc319d50 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -240,3 +240,16 @@ class BitbakeLayers(OESelftestTestCase): self.assertEqual(first_desc_2, '', "Describe not cleared: '{}'".format(first_desc_2)) self.assertEqual(second_rev_2, second_rev_1, "Revision should not be updated: '{}'".format(second_rev_2)) self.assertEqual(second_desc_2, second_desc_1, "Describe should not be updated: '{}'".format(second_desc_2)) + + def test_bitbakelayers_setup_localcopy(self): + testcopydir = os.path.join(self.builddir, 'test-layer-copy') + result = runCmd('bitbake-layers create-layers-setup --writer oe-local-copy {}'.format(testcopydir)) + oe_core_found = False + meta_selftest_found = False + for topdir, subdirs, files in os.walk(testcopydir): + if topdir.endswith('meta/conf') and 'layer.conf' in files: + oe_core_found = True + if topdir.endswith('meta-selftest/conf') and 'layer.conf' in files: + meta_selftest_found = True + self.assertTrue(oe_core_found, "meta/conf/layer.conf not found in {}".format(testcopydir)) + self.assertTrue(meta_selftest_found, "meta-selftest/conf/layer.conf not found in {}".format(testcopydir)) From patchwork Tue May 7 11:46:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 43329 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9446C25B79 for ; Tue, 7 May 2024 11:46:55 +0000 (UTC) Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by mx.groups.io with SMTP id smtpd.web10.9479.1715082406610108761 for ; Tue, 07 May 2024 04:46:46 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=WtTR5Gz1; spf=pass (domain: gmail.com, ip: 209.85.218.51, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f51.google.com with SMTP id a640c23a62f3a-a59e4136010so280269866b.3 for ; Tue, 07 May 2024 04:46:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715082405; x=1715687205; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D60eP3TaT3Vsq4Bx85RYVlPWdsP28ojmQ/3ZERdDwYo=; b=WtTR5Gz1WAggrNj2k+2q/umsi3GRsFiMTn1iYZ17D0mTwxBkMGzgyXU44agx61gtXk ugj/rjrDH2WHCmvBOmwfeufqLHE/kYFd4A/Y9pg+Gl+nf5IE8FnCDB7C0C7Tkhh/Klzy cgFjbeop6XeXOUwudfVLqVElj8yw4atzZvP6YG5KB3gMucSo599xaUXd90xAC7KPNvCu t0D3P32U8ktko2zkaEtOoif2MLjlPa3ayWrMdd6uW44P3edJrPqpz2p1MthpeOUTtHdp dAh7hgPKPY6WBOK1yJPlDrAR97Pj4KjVVpSldRt5VDAxnAtV6iyG5mO+ELOoSpknVJiK aykw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715082405; x=1715687205; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D60eP3TaT3Vsq4Bx85RYVlPWdsP28ojmQ/3ZERdDwYo=; b=dpcaCvxS36Evg3wZW5bvcgm3lXTI2vz2bk1DOOrc5SL7SYyToqgdx3IHioeN1iHw0U nYb4OInC1mecKkFb7KMsOXljBC1VK29PkihkVKtK2EwzIeth1IHJl2r1RlyT+p/FEtP1 lhURCRLS1b+W0wNdzEXkQVzMZUX/Sen2VbWlYdWXRUdNbHv4gnYvTHOGqQy/HtK4ABHg NlRDjQADO9WJd7tyYAM2V/IL623U/hr/ExXNmIY9qLCkUnVpLYhl/bgoV5s234Nu3cyu 2kMA1LwRFo8FqdEF/ZpOs2Wcuig4VECiON6PgmflzBICY29zgi9mf7A78YwtulxN7WjE gSkQ== X-Gm-Message-State: AOJu0YzPa8zvuTBJZpgTmQOQbHJF9K2d0JxO2hnTTzOuIju408FI6HEv P+p8WJ1zA0gHV28oRian+vDzvxo0dDW1PKY76QFiKxa1xB8zcmUQA82v6Q== X-Google-Smtp-Source: AGHT+IE2MegRAHfgJpvnS+StM7PzJ8J8ounvRW92gXGp9A5aziRJAwGQn7E8lSqcPQf7ijRbsMn9JA== X-Received: by 2002:a50:d61c:0:b0:572:9dbf:1de5 with SMTP id x28-20020a50d61c000000b005729dbf1de5mr7530767edi.40.1715082404918; Tue, 07 May 2024 04:46:44 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id et4-20020a056402378400b00572d255e342sm5268215edb.10.2024.05.07.04.46.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 04:46:44 -0700 (PDT) From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 5/9] lib/oe/sstatesig.py: add an option to make a unihash cache copy Date: Tue, 7 May 2024 13:46:25 +0200 Message-Id: <20240507114629.2591882-5-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240507114629.2591882-1-alex.kanavin@gmail.com> References: <20240507114629.2591882-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 07 May 2024 11:46:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/199088 From: Alexander Kanavin This is already performed in eSDK creation; the change allows also making a copy with a standalone 'bitbake -S unihash-cache-copy targets...' command, which is useful for replicating builds on a different build host (simply coping sstate objects is not enough if they need to be discovered through unihash redirections). Signed-off-by: Alexander Kanavin --- meta/lib/oe/sstatesig.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py index a46e5502ab3..5dfd180e7f2 100644 --- a/meta/lib/oe/sstatesig.py +++ b/meta/lib/oe/sstatesig.py @@ -147,6 +147,10 @@ class SignatureGeneratorOEBasicHashMixIn(object): sigfile = os.getcwd() + "/locked-sigs.inc" bb.plain("Writing locked sigs to %s" % sigfile) self.dump_lockedsigs(sigfile) + if 'unihash-cache-copy' in options: + cachefile = os.getcwd() + "/bb_unihashes.dat" + bb.plain("Writing unihash cache to %s" % cachefile) + bb.parse.siggen.copy_unitaskhashes(os.getcwd()) return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigs(dataCache, options) From patchwork Tue May 7 11:46:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 43325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82CA3C10F1A for ; Tue, 7 May 2024 11:46:55 +0000 (UTC) Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) by mx.groups.io with SMTP id smtpd.web11.9484.1715082407648621227 for ; Tue, 07 May 2024 04:46:48 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=cAWZwe4r; spf=pass (domain: gmail.com, ip: 209.85.167.42, mailfrom: alex.kanavin@gmail.com) Received: by mail-lf1-f42.google.com with SMTP id 2adb3069b0e04-51ae2e37a87so3997072e87.2 for ; Tue, 07 May 2024 04:46:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715082406; x=1715687206; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HCGqm3ObVK1AKU+eEmtnRmX2xq7TtVO4XadeBblwsNw=; b=cAWZwe4r/1Bt2ygMxe+8YH+PYWCvEzIW9anoZvGw98tLHYs4TnGLQH4OncM/iGUAow n/5KrwU3bbv67tRzJnjhhNiydYJpUaVdFi6NIkggzxGwFT3gvB8zGFirPTGpMRVhMWOD SG7vdRMhveLWUNFF6TOIxYUwObmH1wThOzlB9IqXxHGnBg7F5l2W1WhoYEV57w4w4TkL kqgyM+xljDm4VX+VJD4XUARAt6WQ+nmW/QVsO2VQuDhyaeQUdFU062V31oO82lcrLBiW pBUYiJJBrmfgLiVADnSxFYhlxB0C9WQ0ecOF/tM6aBzg3l1VtLU8fRNSfA/6VGVXsZdc 6FlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715082406; x=1715687206; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HCGqm3ObVK1AKU+eEmtnRmX2xq7TtVO4XadeBblwsNw=; b=U3Y8OY+vPyr+geWSPF9zr2VWfnboLhJWyMPG/7u4Zq8VKqkgPrwvcm+/Ou3xdS1tuv LfDDkAnu0ZtKwM5mzdBbTWI9wGjtWHfRKcVt9W/Lbo/tFZdzAjhC06Q2xImtknRFy2YH A4J2s5wP9QQ7M1UtyUmQI549OeOHFDZjKRPCN8oL2jd+BVQWlzf0yieouABOWwliXk13 SUkk9BL4ErF/1RmagmeLaFQ/VtnrRodOfaQAs0GWjn9Ch1GHy6bO5L6C7v6IKBjplaEu q36o8hEmJy68C6VNvk9Bf4ausHQVRTd9c4pM/xAsVKBoKM6+Nh9eU6mRFvzXcdwxJfZo bXrQ== X-Gm-Message-State: AOJu0YzCMz37dcMOy3EID+365KWxWC7NZu+KxmmR2Vg8AiOE/jQp0EII u6KJbObSTg5DCp5cbP2RXBxCNb0PLZb6drKT/1GLmYHtKLuOQSue5nRGgw== X-Google-Smtp-Source: AGHT+IHEJgvZ2GB1RJvhA9SnHZE5ykb44lltJ/oCx7iZO6qJUX1oGwFyH1SpTq+8jwfnR7TXFFv16Q== X-Received: by 2002:a05:6512:32ac:b0:51e:543c:a45d with SMTP id q12-20020a05651232ac00b0051e543ca45dmr8810641lfe.20.1715082405732; Tue, 07 May 2024 04:46:45 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id et4-20020a056402378400b00572d255e342sm5268215edb.10.2024.05.07.04.46.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 04:46:45 -0700 (PDT) From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 6/9] sstatesig/dump_sigs: allow setting output directory explicitly from command line option Date: Tue, 7 May 2024 13:46:26 +0200 Message-Id: <20240507114629.2591882-6-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240507114629.2591882-1-alex.kanavin@gmail.com> References: <20240507114629.2591882-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 07 May 2024 11:46:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/199089 From: Alexander Kanavin The default (current directory) is unfortunately non-deterministic, as the operation is performed from bitbake-server and not from bitbake process directly. If the server was started in a different directory it will write the files there, and not where bitbake command requesting those files was started. Signed-off-by: Alexander Kanavin --- meta/lib/oe/sstatesig.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py index 5dfd180e7f2..f885bb54f64 100644 --- a/meta/lib/oe/sstatesig.py +++ b/meta/lib/oe/sstatesig.py @@ -143,14 +143,19 @@ class SignatureGeneratorOEBasicHashMixIn(object): super().set_taskdata(data[3:]) def dump_sigs(self, dataCache, options): + outdir = os.getcwd() + for o in options: + if o.startswith('outdir'): + outdir = o.split('=')[1] + if 'lockedsigs' in options: - sigfile = os.getcwd() + "/locked-sigs.inc" + sigfile = outdir + "/locked-sigs.inc" bb.plain("Writing locked sigs to %s" % sigfile) self.dump_lockedsigs(sigfile) if 'unihash-cache-copy' in options: - cachefile = os.getcwd() + "/bb_unihashes.dat" + cachefile = outdir + "/bb_unihashes.dat" bb.plain("Writing unihash cache to %s" % cachefile) - bb.parse.siggen.copy_unitaskhashes(os.getcwd()) + bb.parse.siggen.copy_unitaskhashes(outdir) return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigs(dataCache, options) From patchwork Tue May 7 11:46:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 43326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83C16C25B75 for ; Tue, 7 May 2024 11:46:55 +0000 (UTC) Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by mx.groups.io with SMTP id smtpd.web11.9485.1715082408053641953 for ; Tue, 07 May 2024 04:46:48 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=a2jiCJLp; spf=pass (domain: gmail.com, ip: 209.85.218.51, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f51.google.com with SMTP id a640c23a62f3a-a59ece5e18bso155582466b.2 for ; Tue, 07 May 2024 04:46:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715082406; x=1715687206; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OYdVYWukUb9hI4Es+l65oTcvNxvuIIsk3tNl85AZGw4=; b=a2jiCJLpyxqj8GFgVrS2XS5JmUtIGo7N4bCaclHyaS9R2xW9A4RxIWr5OdSBZQfHKz 6yw7HokqSA+0Pwlt01d9XEbcTmsJb+V1ddkvjkHFzpEofvFD8vUGd30PLjeZFxVJj9Zi MsJjXapXUTcYDFjlFH4etx6HF6sedYSxd1nOTZ9qa4FvIxFg22qtsokAUmiKnfeVSDhK 8UxXDwcbjQG0BKaI3gu1vKP+JOyGfqBBKUDIPZ23bhauQ2KiRRn64c8ZoDqZjJsIg0W6 4rTO+iVYnYnUjS7f/q5ElMg9fSrjqZl9G8kQ/GTe5X3+tVM16VvJsbLm+iKpfmcBhJiN 4ZwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715082406; x=1715687206; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OYdVYWukUb9hI4Es+l65oTcvNxvuIIsk3tNl85AZGw4=; b=H/DBljIIcjYsbZBI7hM38FaTIwxiTnYTutbvtEcrY1mb0hctCP+k0hj+jFffNfj1Fa IWpJFvl6HGud8YrS4tW8jRW+AZE+DZjD4TZNl7AjlxHc97N0gtHw6Bm0VkW0Qch1EtTO D9c6X3zfOEZ9wpOwLoNxDeiuw6GUQcqGvXF2whXQynLcypM+5DbpwIGBwmSK+QXKWc7j 1QCRBzTPuvcpcMrdfg6dIW/iJ+U7GHhCykN1Dk+1aLSZMWTBCXl/zjiHPntrTYCY8EMI Eq2n/YzlP0OrwJgs9EBDJnfISkeq2KiFYXcBCT+UgjPOjkN0UynSLKGirAEDlvRAqeJL cSJA== X-Gm-Message-State: AOJu0YyWiXxs5wGZfQ5EDouYdBnaUpFyIqiG6U5664lbfjBYIDnaPLXy dbT6QQCKfINSdFmY8lcbxkJb18TT02Nn+45QlLpyyN/nT1qfN5POnhsFgw== X-Google-Smtp-Source: AGHT+IG6vuCKqpjt2dykORMfvBn1wgAkrY+vL3EgSoEgmqZrNxi7ChlTl7spGYneHLqSJ1rXm22HVA== X-Received: by 2002:a50:cd5d:0:b0:56c:1696:58a8 with SMTP id d29-20020a50cd5d000000b0056c169658a8mr11655120edj.0.1715082406279; Tue, 07 May 2024 04:46:46 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id et4-20020a056402378400b00572d255e342sm5268215edb.10.2024.05.07.04.46.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 04:46:46 -0700 (PDT) From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 7/9] selftest/sstatetests: separate sstate presence check into its own class Date: Tue, 7 May 2024 13:46:27 +0200 Message-Id: <20240507114629.2591882-7-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240507114629.2591882-1-alex.kanavin@gmail.com> References: <20240507114629.2591882-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 07 May 2024 11:46:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/199090 From: Alexander Kanavin This allows reusing the check function in tests other than CDN/local mirror ones, such as sstate bundle testing. Signed-off-by: Alexander Kanavin --- meta/lib/oeqa/selftest/cases/sstatetests.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index 86d6cd7464f..8f843139ebc 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -917,8 +917,7 @@ INHERIT += "base-do-configure-modified" """, expected_sametmp_output, expected_difftmp_output) -@OETestTag("yocto-mirrors") -class SStateMirrors(SStateBase): +class SStateCheckObjectPresence(SStateBase): def check_bb_output(self, output, exceptions, check_cdn): def is_exception(object, exceptions): for e in exceptions: @@ -960,6 +959,8 @@ class SStateMirrors(SStateBase): self.assertEqual(len(failed_urls), missing_objects, "Amount of reported missing objects does not match failed URLs: {}\nFailed URLs:\n{}\nFetcher diagnostics:\n{}".format(missing_objects, "\n".join(failed_urls), "\n".join(failed_urls_extrainfo))) self.assertEqual(len(failed_urls), 0, "Missing objects in the cache:\n{}\nFetcher diagnostics:\n{}".format("\n".join(failed_urls), "\n".join(failed_urls_extrainfo))) +@OETestTag("yocto-mirrors") +class SStateMirrors(SStateCheckObjectPresence): def run_test(self, machine, targets, exceptions, check_cdn = True, ignore_errors = False): # sstate is checked for existence of these, but they never get written out to begin with exceptions += ["{}.*image_qa".format(t) for t in targets.split()] From patchwork Tue May 7 11:46:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 43327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9422FC25B76 for ; Tue, 7 May 2024 11:46:55 +0000 (UTC) Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) by mx.groups.io with SMTP id smtpd.web10.9480.1715082408509365177 for ; Tue, 07 May 2024 04:46:48 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=beyZ7+hb; spf=pass (domain: gmail.com, ip: 209.85.218.52, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f52.google.com with SMTP id a640c23a62f3a-a59c04839caso625467866b.2 for ; Tue, 07 May 2024 04:46:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715082407; x=1715687207; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qZuV6K7EP+DvEfVyw1ILZ0ycOKrtDUL+3lvt0f+lOGA=; b=beyZ7+hbShjtKVuq4MyQc0Dzya5sAtVOaLi+AJ1CFTXalsVJLukoPDCDfXe/YhfGvs FFRW6vVjOr+IhAamRbateGp5Z3GYwGxrDX5o7rx2TYpklyNRvLLD3m3ShlHVTPXl9xGk Y4TZXQmiqAqBiHsk0pv3zz4U3WOpufoxtulwgOQAqZwo2OJaN0M/iaT28Bfpmjf1LN8s WbF5X5xi7tbAiGPVo4gmv2fQwNKy32r+dhAg15j6OCfD0K4w1XPxEV9LPfO95mmWtGtC Rpvt/OsJBuNvO+zIJeiuUh2SQu5pJT2LrMjRvU7/59QGT6nv94AO1nV2iohPqKxgW6eM s/Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715082407; x=1715687207; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qZuV6K7EP+DvEfVyw1ILZ0ycOKrtDUL+3lvt0f+lOGA=; b=sTcwBo0Hv4b4JA2fMvAOSxCoQ9biSjxWIgNqutwW8FudRJ76ZnEfeSF3cpeHpWyhSY racurjG3FSA9Fg79IF9UQnlpNIomIP8Cfgo7dfGWAh0tWhpc1KdSaBsw5SCqIBKsuz+I U4KORMZ9PT3o9TLtuiDv1jjmoHAxxJJVq4EAFdExxT2xqGEdz3uA/Lmw1B5ewnCXnBsL 17iIAn8k8VKLzNj9uA+EykXxQwhMAMUt8oExeGzn8RDfH6k+j7jqMrIQFyrc/Yw42DYJ m3IDD9So3GCgGQaWT1/zYWKwD/Y8DVoBQX/GX89zmXsqXJrkF3fVV/Ld6yaiahtjREzh 079g== X-Gm-Message-State: AOJu0YypA7zZIJAoJ5g60VIMvBKDZcUTOxhXtbJ4jRGN0lIOIACO3Fxm yRLMdW4uadIB7L63U1brblwCo95wylpDujpwG6Bb9721jjdUDOIuFW7QkA== X-Google-Smtp-Source: AGHT+IGwHbhpeDdAEf5QGzmTvyLwjc2+MiPB758IivfDbwb723wdSbd11DbpDavGvU3CsjmgDp+BlQ== X-Received: by 2002:a50:ab5e:0:b0:570:1de6:8a2d with SMTP id t30-20020a50ab5e000000b005701de68a2dmr8045902edc.10.1715082406863; Tue, 07 May 2024 04:46:46 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id et4-20020a056402378400b00572d255e342sm5268215edb.10.2024.05.07.04.46.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 04:46:46 -0700 (PDT) From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 8/9] selftest/sstatetests: move exception list for cache objects to the parent class Date: Tue, 7 May 2024 13:46:28 +0200 Message-Id: <20240507114629.2591882-8-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240507114629.2591882-1-alex.kanavin@gmail.com> References: <20240507114629.2591882-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 07 May 2024 11:46:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/199091 From: Alexander Kanavin This is useful not just in cdn/local cache tests, but also in the build bundle test, and so should be applied by default. Signed-off-by: Alexander Kanavin --- meta/lib/oeqa/selftest/cases/sstatetests.py | 26 ++++++++++----------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index 8f843139ebc..94ad6e38b68 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -918,13 +918,24 @@ INHERIT += "base-do-configure-modified" expected_sametmp_output, expected_difftmp_output) class SStateCheckObjectPresence(SStateBase): - def check_bb_output(self, output, exceptions, check_cdn): + def check_bb_output(self, output, targets, exceptions, check_cdn): def is_exception(object, exceptions): for e in exceptions: if re.search(e, object): return True return False + # sstate is checked for existence of these, but they never get written out to begin with + exceptions += ["{}.*image_qa".format(t) for t in targets.split()] + exceptions += ["{}.*deploy_source_date_epoch".format(t) for t in targets.split()] + exceptions += ["{}.*image_complete".format(t) for t in targets.split()] + exceptions += ["linux-yocto.*shared_workdir"] + # these get influnced by IMAGE_FSTYPES tweaks in yocto-autobuilder-helper's config.json (on x86-64) + # additionally, they depend on noexec (thus, absent stamps) package, install, etc. image tasks, + # which makes tracing other changes difficult + exceptions += ["{}.*create_spdx".format(t) for t in targets.split()] + exceptions += ["{}.*create_runtime_spdx".format(t) for t in targets.split()] + output_l = output.splitlines() for l in output_l: if l.startswith("Sstate summary"): @@ -962,17 +973,6 @@ class SStateCheckObjectPresence(SStateBase): @OETestTag("yocto-mirrors") class SStateMirrors(SStateCheckObjectPresence): def run_test(self, machine, targets, exceptions, check_cdn = True, ignore_errors = False): - # sstate is checked for existence of these, but they never get written out to begin with - exceptions += ["{}.*image_qa".format(t) for t in targets.split()] - exceptions += ["{}.*deploy_source_date_epoch".format(t) for t in targets.split()] - exceptions += ["{}.*image_complete".format(t) for t in targets.split()] - exceptions += ["linux-yocto.*shared_workdir"] - # these get influnced by IMAGE_FSTYPES tweaks in yocto-autobuilder-helper's config.json (on x86-64) - # additionally, they depend on noexec (thus, absent stamps) package, install, etc. image tasks, - # which makes tracing other changes difficult - exceptions += ["{}.*create_spdx".format(t) for t in targets.split()] - exceptions += ["{}.*create_runtime_spdx".format(t) for t in targets.split()] - if check_cdn: self.config_sstate(True) self.append_config(""" @@ -988,7 +988,7 @@ MACHINE = "{}" bitbake("-S none {}".format(targets)) if ignore_errors: return - self.check_bb_output(result.output, exceptions, check_cdn) + self.check_bb_output(result.output, targets, exceptions, check_cdn) def test_cdn_mirror_qemux86_64(self): exceptions = [] From patchwork Tue May 7 11:46:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kanavin X-Patchwork-Id: 43324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C9DFC41513 for ; Tue, 7 May 2024 11:46:55 +0000 (UTC) Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by mx.groups.io with SMTP id smtpd.web10.9481.1715082409183452288 for ; Tue, 07 May 2024 04:46:49 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=lDFL/WWB; spf=pass (domain: gmail.com, ip: 209.85.218.51, mailfrom: alex.kanavin@gmail.com) Received: by mail-ej1-f51.google.com with SMTP id a640c23a62f3a-a599a298990so808706566b.2 for ; Tue, 07 May 2024 04:46:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715082407; x=1715687207; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zLpiWCSWWWhjuvEiZq9ugoHBjyr0DDTXq7mVI5herZY=; b=lDFL/WWBTflX10YasaeSfexFgE4OMxRZMLoXNQ5BwDT15NTnG/ah6uBJRSwSFLhWzI jXKsBSXdYSbL2IsODuQH8yW8iVoVIJU16cwIDp3DVqsM/3AFEylr/NvMOuG0JyPUqHX8 dhWsFptBZFFRwi/A3yMhgoietuNkbCkmpzWtloBshnMJwywrOj9n///2Kaj/7na0SJBx cJzxWtGqlUuHG2zU0C5yfICWXC8ZDjkFAI40lFj++U9D96Eayn03BvuaeIkJPVVKhIQd NESltoX6LexxCmTdE38rHxCbRJinESea2Gsyx69y0ShgiLJ9+pKgu6gfQwURrv/SEyJE x5sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715082407; x=1715687207; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zLpiWCSWWWhjuvEiZq9ugoHBjyr0DDTXq7mVI5herZY=; b=pDGZJM6YhCgeEuUACDE96klRbKEwC6wgE+1xfuUsRr1FSJKxGO2E6PGF1Jbf3Eck70 awUAyQ8iEjkMyey4Ua1TeMsQZ1I/RrmhenUqWeCZE3TD7WcxIKozGmpkRzYtV/Dd3YaT 7y37+rymUVTZ3aCknpOnx3GjAvZHLf/OBBFs+2IYwWE8g+35aVxMD+yu8bUTIwMyBB/E 3/4Ox7J9SZT0fY6+PANG5L4VUlQTKaCjbAWmpXnYQFuRjAVgozIhusCvEYo2pgyQgvkl 8+jgTePdFzLnv+7VdFPeMHOtTVBRgve6RAPcneAAsHmod9/c93E1RzNbM7sH6Oo+C8zU +HKQ== X-Gm-Message-State: AOJu0YxO0LBuPYfNHIKrx/FIHmbJPAIlFQiekqT6WbUlr0YM4ZCBBAch T1d/zGCXjwMCwJdqsmK8/uG15GXw8AEnv6niVMG1SDYT7SIDnYA8tJmXFg== X-Google-Smtp-Source: AGHT+IEOpjtViVseK++j9RTnJk9h5x1bAIqBhJcEnTT/nYL2GGo0Z2DDNisR02BsyFAIJNXIPinzwA== X-Received: by 2002:a50:d581:0:b0:572:71b2:e200 with SMTP id v1-20020a50d581000000b0057271b2e200mr8163999edi.22.1715082407445; Tue, 07 May 2024 04:46:47 -0700 (PDT) Received: from Zen2.lab.linutronix.de. (drugstore.linutronix.de. [80.153.143.164]) by smtp.gmail.com with ESMTPSA id et4-20020a056402378400b00572d255e342sm5268215edb.10.2024.05.07.04.46.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 04:46:47 -0700 (PDT) From: Alexander Kanavin To: openembedded-core@lists.openembedded.org Cc: Alexander Kanavin Subject: [PATCH 9/9] scripts/oe-replicate-build: add a script that packages and replicates a yocto build elsewhere Date: Tue, 7 May 2024 13:46:29 +0200 Message-Id: <20240507114629.2591882-9-alex.kanavin@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240507114629.2591882-1-alex.kanavin@gmail.com> References: <20240507114629.2591882-1-alex.kanavin@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 07 May 2024 11:46:55 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/199092 From: Alexander Kanavin This scripts takes an existing, active yocto build, and puts pieces of it into a bundle directory, which is then placed in a tarball, and that goes into a self-extracting shell archive. This allows moving the bundle to another machine, or placing it on a network server for downloads. It's like esdk bundles, except inside there is a regular, plain, direct yocto build. The bundle includes: - copies of all layers used in a build, with their git histories - build configuration as a template in a special layer private to the bundle - (this is the best bit) all the sstate needed for the specified target, but *only* that sstate and not a complete copy of the local cache. When someone runs the self-extracting shell archive, everything should 'just work': the layers are restored, and the build set up to use them, the bundled sstate cache, and the template configuration. Alternatively, it's possible to not include the cache, and presumably it will come from a configured sstate mirror. This enables at least two*scratch*three interesting use cases: - moving builds around in single-step fashion: run the script, and give the artifact to someone else via file sharing or usb sticks. - publishing builds on the network, perhaps right next to the sstate cache that can fulfil them. This allows an alternative, much smoother entry point into 'doing yocto', as this guarantees availability of needed cache items, and so builds will be a lot faster than the typical 'compile everything from scratch' yocto experience. Sstate is guaranteed to be available, as there's a record of needed items in each bundle, and that record can be used to prevent pruning of still-needed cache items. - this, in turn, can enable 'binary Yocto distro' implemented with sstate, if there's a higher level tool that creates bundles in a systematic, structured fashion, and another tool that iterates over available bundles, and lets users pick them for building locally. Signed-off-by: Alexander Kanavin --- meta/files/bundle-shar-extract.sh | 55 +++++++++++++++ meta/lib/oeqa/selftest/cases/sstatetests.py | 13 ++++ scripts/oe-replicate-build | 75 +++++++++++++++++++++ 3 files changed, 143 insertions(+) create mode 100644 meta/files/bundle-shar-extract.sh create mode 100755 scripts/oe-replicate-build diff --git a/meta/files/bundle-shar-extract.sh b/meta/files/bundle-shar-extract.sh new file mode 100644 index 00000000000..ae10dbaf065 --- /dev/null +++ b/meta/files/bundle-shar-extract.sh @@ -0,0 +1,55 @@ +#!/bin/bash + +verbose=0 +listcontents=0 +prepare_buildsystem="yes" +while getopts "Dln" OPT; do + case $OPT in + D) + verbose=1 + ;; + l) + listcontents=1 + ;; + n) + prepare_buildsystem="no" + ;; + *) + echo "Usage: $(basename "$0") [-D] [-l] [-n]" + echo " -D Use set -x to see what is going on" + echo " -l list files that will be extracted" + echo " -n Extract files but do not prepare the build system" + exit 1 + ;; + esac +done + +if [ $verbose = 1 ] ; then + set -x +fi + +payload_offset=$(($(grep -na -m1 "^MARKER:$" "$0"|cut -d':' -f1) + 1)) + +if [ "$listcontents" = "1" ] ; then + tail -n +$payload_offset "$0"| tar tv || exit 1 + exit +fi + + +tail -n +$payload_offset "$0"| tar mx --zstd --checkpoint=.2500 || exit 1 + +if [ $prepare_buildsystem = "no" ] ; then + exit +fi + +target_dir=$(basename "$0" .sh) +pushd $target_dir +layers/setup-build setup -c build-config-default -b build --no-shell > setup-build.log +popd + +echo "Each time you wish to use this build in a new shell session, you need to source the environment setup script:" +echo " \$ . $target_dir/build/init-build-env" + +exit 0 + +MARKER: diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index 94ad6e38b68..b59d0d2d1e8 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -1007,3 +1007,16 @@ MACHINE = "{}" def test_local_cache_qemuarm64(self): exceptions = [] self.run_test("qemuarm64", "core-image-minimal core-image-full-cmdline core-image-sato-sdk", exceptions, check_cdn = False) + +class SStateBundles(SStateCheckObjectPresence): + def test_minimal_bundle(self): + targets = "core-image-minimal" + machine = get_bb_var('MACHINE') + bitbake("--runall build {}".format(targets)) + runCmd("oe-replicate-build --targets {}".format(targets)) + extractedbundledir = tempfile.mkdtemp(prefix='bundle-extracted-', dir=self.topdir) + runCmd("../build-bundle.sh", cwd=extractedbundledir) + result = runCmd(". build-bundle/build/init-build-env && MACHINE={} bitbake -DD -n {}".format(machine,targets), cwd=extractedbundledir, shell=True, executable='/bin/bash') + + exceptions = [] + self.check_bb_output(result.output, targets, exceptions, check_cdn=False) diff --git a/scripts/oe-replicate-build b/scripts/oe-replicate-build new file mode 100755 index 00000000000..bb465f0372d --- /dev/null +++ b/scripts/oe-replicate-build @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +import argparse +import json +import os +import subprocess +import shutil + +def _do_bundle(args): + bundledir = args.output_prefix or "build-bundle" + print("Making a self-extracting bundle archive in {}.sh ...".format(bundledir)) + os.mkdir(bundledir) + + subprocess.check_output("bitbake -S outdir={} -S lockedsigs {}".format(os.path.abspath(bundledir),args.targets), shell=True) + sstate_dir = subprocess.check_output("bitbake-getvar --value SSTATE_DIR", shell=True).decode().strip() + nativelsbstring = subprocess.check_output("bitbake-getvar --value NATIVELSBSTRING", shell=True).decode().strip() + if not args.no_sstate: + subprocess.check_output("gen-lockedsig-cache {} {} {} {}".format("locked-sigs.inc", sstate_dir, "build/sstate-cache", nativelsbstring), shell=True, cwd=bundledir) + cachedir = bundledir + "/build/cache" + os.mkdir(cachedir) + subprocess.check_output("bitbake -S outdir={} -S unihash-cache-copy {}".format(os.path.abspath(cachedir),args.targets), shell=True) + + layerdir = "meta-build-config" + subprocess.check_output("bitbake-layers create-layer --add-layer {}".format(layerdir), shell=True, cwd=bundledir) + subprocess.check_output("bitbake-layers save-build-conf {} default".format(layerdir), shell=True, cwd=bundledir) + shutil.copy(os.path.join(os.environ["BUILDDIR"],'conf',"conf-summary.txt"), os.path.join(bundledir,layerdir,'conf/templates/default')) + shutil.copy(os.path.join(os.environ["BUILDDIR"],'conf',"conf-notes.txt"), os.path.join(bundledir,layerdir,'conf/templates/default')) + subprocess.check_output("bitbake-layers remove-layer {}".format(layerdir), shell=True, cwd=bundledir) + + # meta-build-config is then in bblayers.conf.sample, and should be removed from it as it wasn't in the actual build + bblayers = os.path.join(bundledir, layerdir, 'conf/templates/default/bblayers.conf.sample') + with open(bblayers) as f: + lines = f.readlines() + lines = [l for l in lines if os.path.join(bundledir, layerdir) not in l] + with open(bblayers,'w') as f: + f.write(''.join(lines)) + + subprocess.check_output("bitbake-layers create-layers-setup --writer oe-local-copy {}".format(bundledir), shell=True) + + # meta-build-config should however be present in .oe-layers.json, as otherwise oe-setup-build won't be able to discover + # the config template in it + oelayers = os.path.join(bundledir, 'layers', '.oe-layers.json') + with open(oelayers) as f: + json_f = json.load(f) + json_f["layers"].append("../meta-build-config") + with open(oelayers,'w') as f: + json.dump(json_f, f, sort_keys=True, indent=4) + + subprocess.check_output("tar caf {}.tar.zst {}".format(bundledir, bundledir), shell=True) + + corebase = subprocess.check_output("bitbake-getvar --value COREBASE", shell=True).decode().strip() + subprocess.check_output("cp {}/meta/files/bundle-shar-extract.sh {}.sh".format(corebase, bundledir), shell=True) + subprocess.check_output("cat {}.tar.zst >> {}.sh".format(bundledir, bundledir), shell=True) + subprocess.check_output("chmod +x {}.sh".format(bundledir), shell=True) + + if not args.keep_tmp: + shutil.rmtree(bundledir) + os.remove("{}.tar.zst".format(bundledir)) + +parser = argparse.ArgumentParser(description="A script that bundles up everything needed to replicate a yocto build elsewhere (including appropriate portions of sstate cache) into a self-contained shell archive.") + +parser.add_argument('--targets', required=True, help="Bitbake targets that the bundle should be made for.") +parser.add_argument('--output-prefix', help='File name prefix for the output files, if the default (build-bundle) is undesirable.') +parser.add_argument('--no-sstate', action='store_true', help='Do not include sstate cache into the bundle.') +parser.add_argument('--keep-tmp', action='store_true', help='Keep intermediate output: unpacked bundle directory, and compressed tarball (in addition to the final self-extracting shell archuve.') + +args = parser.parse_args() + +_do_bundle(args) +