From patchwork Thu Jun 6 14:03:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joshua Watt X-Patchwork-Id: 44769 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09182C27C54 for ; Thu, 6 Jun 2024 14:06:33 +0000 (UTC) Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) by mx.groups.io with SMTP id smtpd.web11.15392.1717682789051000088 for ; Thu, 06 Jun 2024 07:06:29 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=msn+echK; spf=pass (domain: gmail.com, ip: 209.85.166.53, mailfrom: jpewhacker@gmail.com) Received: by mail-io1-f53.google.com with SMTP id ca18e2360f4ac-7eb0116faf8so28483339f.3 for ; Thu, 06 Jun 2024 07:06:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717682787; x=1718287587; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HBGCFefJYrzHwbZwdoU3smbeRn37Ywr/NYfTupVi4Ig=; b=msn+echK/xQQbqjkog28wQWR4o4baHY8o2ttdQl5HO2arfpzyJaZpLqr7dUDlMWZ+d BUv4z6PcHMr1gMUBxhBY422y9O3gCOQhEgwS6glaW1RH1kf1WxkUiigWgimYpbyegL9g UTTOSaWTv2Kv2DFiazcVjKiRvVVv+FFwKjJmmtw+KaGr/ZtJaCYhJXfBh/bQItxUyZPB h6pgAqB0xhdyop369BxhHZHopN2dUDoaMWR0PR2SP4tLtje77oe5uzZ4Cvn3XWFTffy7 nw+SYmLreI15KxDReovQo6c53pJN7S+YbKBqaj5btwCS/SnbBClGAG2oAwHUL/kkmOlL 40PQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717682787; x=1718287587; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HBGCFefJYrzHwbZwdoU3smbeRn37Ywr/NYfTupVi4Ig=; b=mKQBl1TGZK7JYpCtQSDH6G3k9WRrcpvwcwtAzkibBPE2JV+sgtihcjZ7T7N3WcBoPb K7IYJ9/DddE73Bvp3Niq96qLzwuJqV7Fg7420hXr0X/StQXvnq8zX90GS61L9t1z/yj9 0VRvfLJ6Nz6PGZq8zIv2n1x1sx3lSvrLK5NpbspUF1Aqn1z4a/9ihl5uA8VX8tbu96VZ fQxwr444HUAtg6ZjDAJ0vcBnVsNx1nvtaHziN7GYsRmQnXaL2of3UgE0q2U1gF61Jf5A JWsa88hMe5jxb6vmzx1yU3wZYb8aeAArJo0PYSLpE/kpMKmfw4ZpVeC1RmZOri5X+94m YwGg== X-Gm-Message-State: AOJu0Yzbvc+ffzHedKhq8jHhT6ldvmGoa5+vY8sjzghOSYrJyC/LjrbY bRB1VL64/rVtHYqUjDUu419UuvBelczyeWfvMU05L3na3faluggvUfuqGw== X-Google-Smtp-Source: AGHT+IFRWEwdCNAK6mhg+E1P70XJ5xJlSZb65ni1Mo1Rf1LO8WVi8b/pmKT9/ieXf9GL/JjkzDj+TQ== X-Received: by 2002:a05:6602:1582:b0:7eb:52f1:53d with SMTP id ca18e2360f4ac-7eb52f106a2mr136307339f.13.1717682787125; Thu, 06 Jun 2024 07:06:27 -0700 (PDT) Received: from localhost.localdomain ([2601:282:4300:19e0::3969]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4b7a2070f95sm346411173.0.2024.06.06.07.06.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Jun 2024 07:06:26 -0700 (PDT) From: Joshua Watt X-Google-Original-From: Joshua Watt To: openembedded-core@lists.openembedded.org Cc: Joshua Watt Subject: [OE-core][RFC 1/2] utils: Add write_files_manifest helper Date: Thu, 6 Jun 2024 08:03:03 -0600 Message-ID: <20240606140622.2494668-2-JPEWhacker@gmail.com> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240606140622.2494668-1-JPEWhacker@gmail.com> References: <20240606140622.2494668-1-JPEWhacker@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 06 Jun 2024 14:06:33 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/200403 Adds a helper function to write out a JSON file with a manifest of files Signed-off-by: Joshua Watt --- meta/lib/oe/utils.py | 67 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py index 14a7d07ef01..5437e7a2d9f 100644 --- a/meta/lib/oe/utils.py +++ b/meta/lib/oe/utils.py @@ -8,6 +8,9 @@ import subprocess import multiprocessing import traceback import errno +import json +import hashlib +import re def read_file(filename): try: @@ -540,3 +543,67 @@ def touch(filename): # Handle read-only file systems gracefully if e.errno != errno.EROFS: raise e + + +def write_file_manifest(path, outfile, *, include_hash=True, extract_lic=False, ignore_dirs=[]): + LIC_REGEX = re.compile(rb'^\W*SPDX-License-Identifier:\s*([ \w\d.()+-]+?)(?:\s+\W*)?$', re.MULTILINE) + + manifest = { + "files": [], + "symlinks": [], + "dirs": [], + } + + for root, dirs, files in os.walk(path): + dirs[:] = [d for d in dirs if d not in ignore_dirs] + + for fn in files: + p = os.path.join(root, fn) + stat = os.lstat(p) + if os.path.islink(p): + manifest["symlinks"].append({ + "path": os.path.relpath(p, path), + "target": os.readlink(p), + "mode": stat.st_mode, + }) + continue + + data = { + "path": os.path.relpath(p, path), + "size": stat.st_size, + "mode": stat.st_mode, + } + + if include_hash or extract_lic: + h = hashlib.sha256() + with open(p, "rb") as f: + if extract_lic: + d = f.read(15000) + h.update(d) + + licenses = re.findall(LIC_REGEX, d) + if licenses: + data["licenses"] = [lic.decode('ascii') for lic in licenses] + + if include_hash: + while True: + d = f.read(4096) + if not d: + break + h.update(d) + + if include_hash: + data["sha256"] = h.hexdigest() + + manifest["files"].append(data) + + for dn in dirs: + p = os.path.join(root, dn) + stat = os.lstat(p) + manifest["dirs"].append({ + "path": os.path.relpath(p, path), + "mode": stat.st_mode, + }) + + with open(outfile, "w") as f: + json.dump(manifest, f) From patchwork Thu Jun 6 14:03:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joshua Watt X-Patchwork-Id: 44770 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DBCAC27C5F for ; Thu, 6 Jun 2024 14:06:33 +0000 (UTC) Received: from mail-io1-f43.google.com (mail-io1-f43.google.com [209.85.166.43]) by mx.groups.io with SMTP id smtpd.web11.15393.1717682789771042018 for ; Thu, 06 Jun 2024 07:06:29 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=YyFLCgUz; spf=pass (domain: gmail.com, ip: 209.85.166.43, mailfrom: jpewhacker@gmail.com) Received: by mail-io1-f43.google.com with SMTP id ca18e2360f4ac-7e1fcd2662cso45986839f.2 for ; Thu, 06 Jun 2024 07:06:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717682788; x=1718287588; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Bh1IySEFEKksIaDhxMgS+bHqF53D2RKWomUMJFy4f/0=; b=YyFLCgUz8atfPj33bvAqqVk7Lx/IpF90efGkV7xD5CXy4D54z6sDSaU6AZTBxBXIgW 7Y9Zv9fSzpoVz8H5tiNKA0/uiFv3WtOguM6OKbEDQRjhmmioCIxFEidEUoKznKhvDM+g SYiJtsDpL6M3sj7tNy1+1KNTAr3HFy+Dh9IpXIb68jcbrDTwc4N8k6o/TF0ngGZjb6cz /mkk2hMKLflnILpyzneIjOPgTlVDDLC1er+1P9HnKeyu+nxQJhwroAp5v3n+JEhLJfyO f6Kgm0VMfJIfYqBGA8TzDzwY1tGdMJo2eHDlAqodx66tiUf9rQocTA6YECn/eSf7cowQ rh0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717682788; x=1718287588; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Bh1IySEFEKksIaDhxMgS+bHqF53D2RKWomUMJFy4f/0=; b=T0gUVmuCf9bnZ7Q9//JAEVcXBmlef1y8u2+5mJKnoe8fUhCTGvHXs3YuyPBmJo094r GmRd+iR4Br97WjmVp1sytfIRb8nk/2nX1zcuszFSyZv8D36pdq/8eMIpsPqv8MIOATVo swb0s3258aMCLaCYWCRQZ6igamGOQsy0gN31Ta8AI2Uw31oRmA2qMcZg/3EniYst8Eby t2CiQ4IG72B0WbObhUziOXjmQ8VerXxKZfBlYb9SSSuapje4cD9ht9y5kkMtc0/Cb91B Bqr4vasL7OlX4Z+XXvy1byB5jjC7xJH+UrO7oUjoEaLNY2cFb6n8yn1okYO+MDCj3+lk ZsLQ== X-Gm-Message-State: AOJu0YxSf1QdcrEwo9FMPLJfQTLkGMivY2yQGKkiV7ewlh48HlSWncbl 6fzwcDoj7vBxoAeafF2D1+pIsyYnVrsIqQWNPtvCSEPvbAVchP/k7fvJtQ== X-Google-Smtp-Source: AGHT+IGmfy3F8TKHefGI+IOMU0g9DqN66Y3XnyvCwu/y37Twz/GpUjQE+PXZSnCbiATjK4mOdwS48Q== X-Received: by 2002:a05:6602:29cf:b0:7e1:b3fa:6470 with SMTP id ca18e2360f4ac-7eb3c91b9bfmr577775339f.19.1717682788255; Thu, 06 Jun 2024 07:06:28 -0700 (PDT) Received: from localhost.localdomain ([2601:282:4300:19e0::3969]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4b7a2070f95sm346411173.0.2024.06.06.07.06.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Jun 2024 07:06:27 -0700 (PDT) From: Joshua Watt X-Google-Original-From: Joshua Watt To: openembedded-core@lists.openembedded.org Cc: Joshua Watt Subject: [OE-core][RFC 2/2] classes-global/base: Add manifest to do_unpack Date: Thu, 6 Jun 2024 08:03:04 -0600 Message-ID: <20240606140622.2494668-3-JPEWhacker@gmail.com> X-Mailer: git-send-email 2.43.2 In-Reply-To: <20240606140622.2494668-1-JPEWhacker@gmail.com> References: <20240606140622.2494668-1-JPEWhacker@gmail.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 06 Jun 2024 14:06:33 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/200404 Writes a manifest of files unpacked in do_unpack Signed-off-by: Joshua Watt --- meta/classes-global/base.bbclass | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass index b6940bbb6ff..e543e206d08 100644 --- a/meta/classes-global/base.bbclass +++ b/meta/classes-global/base.bbclass @@ -177,10 +177,12 @@ python base_do_unpack() { try: fetcher = bb.fetch2.Fetch(src_uri, d) - fetcher.unpack(d.getVar('UNPACKDIR')) + fetcher.unpack(unpackdir) except bb.fetch2.BBFetchException as e: bb.fatal("Bitbake Fetcher Error: " + repr(e)) + oe.utils.write_file_manifest(unpackdir, workdir + "/unpack-manifest.json", extract_lic=True, ignore_dirs=[".git"]) + if basedir and os.path.exists(unpackdir + '/' + basedir): # Compatibility magic to ensure ${WORKDIR}/git and ${WORKDIR}/${BP} # as often used in S work as expected.