From patchwork Thu Jun 4 13:34:42 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 89315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFD72CD6E6B for ; Thu, 4 Jun 2026 13:34:53 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.11718.1780580090182078860 for ; Thu, 04 Jun 2026 06:34:50 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=gUOOk3iQ; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.50, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-490b64c8311so8895335e9.3 for ; Thu, 04 Jun 2026 06:34:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1780580088; x=1781184888; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=S07ch3KPnW04Kay19fV7xvy5hpDM0bdmgUfvzsp87R8=; b=gUOOk3iQfayzALrs73/DqMRFggpee7KZ+uE503oHb76kDTzd7epMccbPBSKeYxttrs DzA2fLQxwfUNmVchQLt0cvyRWdAWpoZKuuzs1v/gORmrW4FAkKPbJlkEgzIYsUF7xhAb WCdagCtWH4qZJIc8pXXcBT8tq0MnOLAmtnSDQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780580088; x=1781184888; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=S07ch3KPnW04Kay19fV7xvy5hpDM0bdmgUfvzsp87R8=; b=XQQt4BUTvAscbqMBikFPt3yPfTUX2CzR5ufjbN1dAFn3DANW0pQU3IwQAIzdUacflQ jpvwySIpF+FIrc8eRce3W/eRx1o7bhcuscaOpmYcLQMsobe0EF80eazXvjo2CqICqok3 VTCPd57pfW62GInxhx6w2vN9mwvM/Ce8r9IQZdVY/td1W4WxFI1adH3cDQulCtanuuy9 O1zHDpsuTk2tYIHyb1pg+J0KHGNQkscfz/ykTtP2clNAym8OhsTOVhhDr7mU91e9Evlk CYqvms/RcW7MnseuQNOrUE/0fXa6UcMxEfZts04kBHoA8pHtJvEiGKurgVMGgYupQzrE pi/Q== X-Gm-Message-State: AOJu0YyP6iJg79D+jtLopwbL1/6UnUzstHc7JJdyik3co+4ojCgC5jj1 Vx8rtQaNuUx4OYsD8xtMKp1JdHfbpbWZt4pSrl/W1oE0NuO2GgFF2jiRlTiArsM/IstPy8hokjd jsZAs X-Gm-Gg: Acq92OF+RVwfPJIArfMrMGyZRCARUqcZ+qRnxrRQSi1lZDAAIuklGkOqsaolCf+L/PO RdPYIvfrzqp/BMFIuN+C2sWVR/NnIgiBl4aIqHxffhCMQp8yEdpqOC4jFEsa1oZd/6MyTjYObmu TYl3eJIyFXtYMZExnvFwtsaXG4REhnqgespXML8triK2l0FzDJgwlnUQm5S8IYKLcKUv9/FYQ68 erQ+gj4wvG8UCiiDnJ74sIKmr91OZjYf8n4N2d+tqzZYy5Sw6T9U/XCgdf+TJxNVgMwQrkhEQd3 dxSyeAzd/ynqIvDgO7pelw1Aoi95LWmQXRGmQ8xoMcnxl5Z/QPVp+iXIhrgQJbCs7EvAxwyCyRU HWeP87UeW2namxrsOAHsMGy0MI2ua/fZ9Z/LyF7GREajypqXNCU8PyL4v+vtqlTRPEpxeQifvtr y6yPl6nZkXcyyZW9xEbzqCWwUbduLEZlowSOSjmx14GjLMEyZBSgp1tJcXeZFiHeA= X-Received: by 2002:a05:600c:c490:b0:490:b0df:9eac with SMTP id 5b1f17b1804b1-490b5e94d73mr135690665e9.3.1780580088237; Thu, 04 Jun 2026 06:34:48 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:eed3:d0ae:8f44:9e49]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490bc3d6c65sm71326935e9.12.2026.06.04.06.34.47 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Jun 2026 06:34:47 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/5] fetch2: Reorder unpack code to facilitate changes Date: Thu, 4 Jun 2026 14:34:42 +0100 Message-ID: <20260604133446.2528212-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.53.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 04 Jun 2026 13:34:53 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19605 In order to refactor the function, we need path and unpackdir to be set earlier in the function. This change just moves code to earlier in the function. Signed-off-by: Richard Purdie --- lib/bb/fetch2/__init__.py | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 93d8b1820d8..d94f54d1bc5 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1539,6 +1539,21 @@ class FetchMethod(object): bb.fatal("Invalid value for 'unpack' parameter for %s: %s" % (file, urldata.parm.get('unpack'))) + path = data.getVar('PATH') + + # If 'subdir' param exists, create a dir and use it as destination for unpack cmd + if 'subdir' in urldata.parm: + subdir = urldata.parm.get('subdir') + if os.path.isabs(subdir): + if not os.path.realpath(subdir).startswith(os.path.realpath(rootdir)): + raise UnpackError("subdir argument isn't a subdirectory of unpack root %s" % rootdir, urldata.url) + unpackdir = subdir + else: + unpackdir = os.path.join(rootdir, subdir) + bb.utils.mkdirhier(unpackdir) + else: + unpackdir = rootdir + base, ext = os.path.splitext(file) if ext in ['.gz', '.bz2', '.Z', '.xz', '.lz', '.zst']: efile = os.path.join(rootdir, os.path.basename(base)) @@ -1614,19 +1629,6 @@ class FetchMethod(object): quoted_datafile = shlex.quote(datafile) cmd = 'ar x %s %s && %s -p -f %s && rm %s' % (shlex.quote(file), quoted_datafile, tar_cmd, quoted_datafile, quoted_datafile) - # If 'subdir' param exists, create a dir and use it as destination for unpack cmd - if 'subdir' in urldata.parm: - subdir = urldata.parm.get('subdir') - if os.path.isabs(subdir): - if not os.path.realpath(subdir).startswith(os.path.realpath(rootdir)): - raise UnpackError("subdir argument isn't a subdirectory of unpack root %s" % rootdir, urldata.url) - unpackdir = subdir - else: - unpackdir = os.path.join(rootdir, subdir) - bb.utils.mkdirhier(unpackdir) - else: - unpackdir = rootdir - if not unpack or not cmd: urldata.unpack_tracer.unpack("file-copy", unpackdir) # If file == dest, then avoid any copies, as we already put the file into dest! @@ -1649,7 +1651,6 @@ class FetchMethod(object): if not cmd: return - path = data.getVar('PATH') if path: cmd = "PATH=\"%s\" %s" % (path, cmd) bb.note("Unpacking %s to %s/" % (file, unpackdir)) From patchwork Thu Jun 4 13:34:43 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 89316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38331CD6E77 for ; Thu, 4 Jun 2026 13:34:54 +0000 (UTC) Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.11719.1780580091242758327 for ; Thu, 04 Jun 2026 06:34:51 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=fjdI4fLY; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.41, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-490b3e03939so6287755e9.1 for ; Thu, 04 Jun 2026 06:34:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1780580089; x=1781184889; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=5WZfIfFTkKfF7UF6Z58s5k16OwxGsysCgzW84wlDcJk=; b=fjdI4fLYwTWmj7ZFFUD3KFNkw67Mv3fnx23U7poqkYPueZ2Va30gBEy3Yy6ZEczkAx mcTus80NtUDlIlkWpnDDtGztRZ0qNWEG7pfzMSD8T8gHemDmdiuoMDxEq8f5CvT3+5lU ncdqIxO9XgYcVnYDyAQ044U3j8wGNwm8Gs3Wc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780580089; x=1781184889; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=5WZfIfFTkKfF7UF6Z58s5k16OwxGsysCgzW84wlDcJk=; b=PuJ8/86kOLzofEL7uPahqzBv4ca2iGvezXN6uW2K4ugiFG9NmEFZud/LCx6GHL442Y ACOZzOdhRbDX2OnS3isfxvPGKJ1RQmFQ5nQUZqjJiwPxe421nYjzqsvwlyNrbO9XKflZ wSDEru16CWjAFUiDvB8s5qyfCPwXa8Mx0121YhVvgfCz9DMmfcYlgET4YrnYNMtFaNPB wOVD6bWxP4sA7T2ny7iLqnGURc1PD2T9veHzESYD8CkGHyN/ewKK0fFLMdWSNYHAKIjz 87jn4iWDhuS+bVn3p3I1nnZMqDwqhTyUlTob0H2RPMH5BgXTP3UTeXQi8gIJrNtmlLrb EB3w== X-Gm-Message-State: AOJu0YzZ9eCYIdVci5MN/91ZgYuFJhRabpDvtzKvpDrcbt+0w5ZHZlh/ XRV3QICXCOORQwVhJVINUvoxg3vRrNpYpV+6Lau92RtEvsEeytcXw3z7CltJ4bu/Xy6gd9QcGrR OJk3U X-Gm-Gg: Acq92OEm8GXFzxwdR4XUvZPSBZ6ymQOI0KdczyLRkO6bltdfNqj3JHRgjXLfM4jjBFj QnJOq9MZzqrlEzA4Dy2+h/tR+HdWY3rfPRxk0HRWSRhntpmLKgluisogH+PBDedmthWxZbiOXxd FV4pmIC9icVvfT9FfNRxCFkUfFFnnFmRgDtnYlgO0Vh0ZvCHS8UgQcr/J6mlc3QgscRNJ/ajNny Qg9hj+XGg6MHH3soXL6GPL6LeBxOuT09W8NweJ01Rlw/5tpxpLgkD2Mj3glB08JBdVcW6poGI7k jfmi/V76eP35yNJF630Uw/F2YKrYqzV7H+JPBo2O62rHMkDHECyjeNnaqwXm92MDtNuWKNHKju1 QXmeN3K074w+aSA2RqEK1/S2r5GOwJpM8ulUVUQTtg2LFXgNTfMDCsPW/TQUUmWsvbQj63J5uPU MfVCDbjetEt6HpbswX6EuaHZBAru54KkVpf4uyycfbJMGkWQcEYFqqS1+hscwIjn4= X-Received: by 2002:a05:600c:528a:b0:48f:e230:29f4 with SMTP id 5b1f17b1804b1-490bc506460mr56775315e9.15.1780580089163; Thu, 04 Jun 2026 06:34:49 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:eed3:d0ae:8f44:9e49]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490bc3d6c65sm71326935e9.12.2026.06.04.06.34.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Jun 2026 06:34:48 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/5] fetch: Abstract command execution in unpack() Date: Thu, 4 Jun 2026 14:34:43 +0100 Message-ID: <20260604133446.2528212-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260604133446.2528212-1-richard.purdie@linuxfoundation.org> References: <20260604133446.2528212-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 04 Jun 2026 13:34:54 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19606 Make the command execution a function as we need to do this multiple times in some cases. Signed-off-by: Richard Purdie --- lib/bb/fetch2/__init__.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index d94f54d1bc5..07997946e22 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1554,6 +1554,11 @@ class FetchMethod(object): else: unpackdir = rootdir + def _run(cmd): + ret = subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True, cwd=unpackdir) + if ret != 0: + raise UnpackError("Unpack command %s failed with return value %s" % (cmd, ret), urldata.url) + base, ext = os.path.splitext(file) if ext in ['.gz', '.bz2', '.Z', '.xz', '.lz', '.zst']: efile = os.path.join(rootdir, os.path.basename(base)) @@ -1654,10 +1659,7 @@ class FetchMethod(object): if path: cmd = "PATH=\"%s\" %s" % (path, cmd) bb.note("Unpacking %s to %s/" % (file, unpackdir)) - ret = subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True, cwd=unpackdir) - - if ret != 0: - raise UnpackError("Unpack command %s failed with return value %s" % (cmd, ret), urldata.url) + _run(cmd) if iterate is True: iterate_urldata = urldata From patchwork Thu Jun 4 13:34:44 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 89314 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10DBCCD6E4A for ; Thu, 4 Jun 2026 13:34:54 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.11746.1780580091992824484 for ; Thu, 04 Jun 2026 06:34:52 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=aQs6oRK/; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.50, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-490aebf33e9so4106185e9.3 for ; Thu, 04 Jun 2026 06:34:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1780580090; x=1781184890; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=rizkG9jZuJediubgH/mX6ef/0Hne4R43p5mzR8jB1QE=; b=aQs6oRK/ddtBcWzY4/HP/48eeH0q21Nkv2Dcpj9UFNHxMve1IYHQmcbExma1k38O3y Qb9WgtlT2kJZNd93q966hdcHi6oU8fPfS1XWohDJGtIhyZ6TJZDg92A8fwdXQPXM5XIi UNvTMV5H0/Sd5A/O8Nh1th4p//NivMk0R/xUo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780580090; x=1781184890; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=rizkG9jZuJediubgH/mX6ef/0Hne4R43p5mzR8jB1QE=; b=m9dMoqr10YuATvM00wBqlsOza9oVgJjXTqSaUve6LOJXBQQrd7DDHixZwLPutzfrfu ++JEIZJFx6lG+hjZtro+we/8MN7H9yZrKRY5FNYiMEvOhgEGeazm817oY7UbZYxo4nEI +3JB3lKNoZYJSbsUSJCpbXIx2/etk8f4m/bvh0txHcbmuh3PZpmtRKbyGGnaKeCPi5eN VZRTLDZN3Z7fXkhhMJSRNOUE/gN0jnFQiiicRwGeKq69mXwbcnrhH/WrQBPAY46HjN1f lMY/buwMCm0JAXATWZ6xV7zkhXZq2S/cpN0W0j06liTvjBeexpctk/LoUg86GVPmWJJU cEMQ== X-Gm-Message-State: AOJu0YwMBEVPBBldJ8fShmWJs7s2JUY5EMHkHMIReBur4lGkTxn9nuGk NSAm38DIHSwnNR7CncONGmmc9pTNU1Kyt0sS3/aIVN50al7NbdRchZgBw7ZI41koh9bBcK+0+kA v22FC X-Gm-Gg: Acq92OFL/tq6tmQsIFww9voLlrdYEH7VRmduMLd8QgPZy+/G33C5EP7kfkjEzIFrUtK 4MpIKEgm/9EVLxF5xK94n20VuVWp9uXO89XEEZXWsWhb2dfnsbW6vhfTXkWIR6CdsxcUB9A/mPb Y41ZBAuNrbzZpJTRG4SKu/1wluQuMvotnF972HMFvbRsyjKCbD9DL4cQQT1PdMfD0sFOmzgi39j Sd9uNJapXaGyqCoRaOO+waEHazbERfO+sD09VI4BRvtSrZEGv6BjNB6UWNnbi8e9HWb71LrtyoK LnUOf5p+vPpeMWeeEsEzP4buLLqYwUe2Sv+h8rWXOIDHJkFfJZKdEkvHewD7H+eafdOk75mEIdm kgqb8QOCT0dTp0jKBH9GOI8GB5XyakbU/qHyXEaB+fgiC+BNoCou+nCcejcM90QKzxOpS+nW4wJ 7oYKwyr7nlgysD96Y+yXtGtJvrpHXcOYBU4IZd9UN5xyQNDMQw0Iis35P0CZlghTAIm0UsqOzzH g== X-Received: by 2002:a05:600c:45d2:b0:490:bada:6b15 with SMTP id 5b1f17b1804b1-490bada6c90mr99332305e9.17.1780580090283; Thu, 04 Jun 2026 06:34:50 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:eed3:d0ae:8f44:9e49]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490bc3d6c65sm71326935e9.12.2026.06.04.06.34.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Jun 2026 06:34:49 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/5] fetch2: Improve environment handling in unpack() Date: Thu, 4 Jun 2026 14:34:44 +0100 Message-ID: <20260604133446.2528212-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260604133446.2528212-1-richard.purdie@linuxfoundation.org> References: <20260604133446.2528212-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 04 Jun 2026 13:34:54 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19607 Rather than putting environment variables into command strings, pass them directly to subprocess. This adds PATH to the environment to a subprocess call to 'ar' where it was previously missing too so the function is consistent. Signed-off-by: Richard Purdie --- lib/bb/fetch2/__init__.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 07997946e22..845dd3baab1 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1539,7 +1539,10 @@ class FetchMethod(object): bb.fatal("Invalid value for 'unpack' parameter for %s: %s" % (file, urldata.parm.get('unpack'))) + env = os.environ.copy() path = data.getVar('PATH') + if path: + env['PATH'] = path # If 'subdir' param exists, create a dir and use it as destination for unpack cmd if 'subdir' in urldata.parm: @@ -1555,7 +1558,7 @@ class FetchMethod(object): unpackdir = rootdir def _run(cmd): - ret = subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True, cwd=unpackdir) + ret = subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True, cwd=unpackdir, env=env) if ret != 0: raise UnpackError("Unpack command %s failed with return value %s" % (cmd, ret), urldata.url) @@ -1618,7 +1621,7 @@ class FetchMethod(object): else: cmd = 'rpm2cpio.sh %s | cpio --no-absolute-filenames -id' % (file) elif file.endswith('.deb') or file.endswith('.ipk'): - output = subprocess.check_output(['ar', '-t', file], preexec_fn=subprocess_setup) + output = subprocess.check_output(['ar', '-t', file], preexec_fn=subprocess_setup, env=env) datafile = None valid_datafiles = ('data.tar', 'data.tar.gz', 'data.tar.xz', 'data.tar.zst', 'data.tar.bz2', 'data.tar.lzma') @@ -1656,8 +1659,6 @@ class FetchMethod(object): if not cmd: return - if path: - cmd = "PATH=\"%s\" %s" % (path, cmd) bb.note("Unpacking %s to %s/" % (file, unpackdir)) _run(cmd) From patchwork Thu Jun 4 13:34:45 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 89317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 374FECD6E6D for ; Thu, 4 Jun 2026 13:34:54 +0000 (UTC) Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.11720.1780580093217180323 for ; Thu, 04 Jun 2026 06:34:53 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=SQ+N/q9k; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.50, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-490b613a17bso7216365e9.3 for ; Thu, 04 Jun 2026 06:34:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1780580092; x=1781184892; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=J4Oy67000vqwBAteypMq6XQuxENgxORhY6i4QWCzP1Q=; b=SQ+N/q9kfjPM+Z5WZRwPagubFFy1BcYMOwSf+RL+PKk7iqoStPc/cJyxp3Ws9Y9/4f ILOA+Y8xoJjVgzoCFx+/8ySjDd4qqGptqtv+CWH4mbtRlDto9ej334QB4i3avS6VKLlB +SRdMGIUQwzuWlmOTh1TonbRONVvG4UCbvmIs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780580092; x=1781184892; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=J4Oy67000vqwBAteypMq6XQuxENgxORhY6i4QWCzP1Q=; b=Wmo+pbwmt7tPPgzrFcbKLPMt6SCM9XOYZKZPG1Bwr5dpFi/DxpMR8y0csYvkI057uh Blgd+BQaHOD6Q+qrlvVIA5bQo9NSV4b/G0UTsaPzq5qn2Adxe50CKSKdQ1V77XP0a/a7 +7hzD+SlqaxBcweja1G/BzR4OMPBtdiGgNPwXf9kwYdJHMzKOXQcusye/ESeeh8qVUo8 Wr620PiROVRz+oefSH3Z4Bpp330dCK4cJbMt2QrqSDJwIkh0557F8KpacWUdwL4gHE64 Cq4Oj8bN9AbzJEcynp+PPvw0qI19BjSvwfsXoYuJzpMjh8yjXCMx6jq6l+IdFTj5xmCx ha9g== X-Gm-Message-State: AOJu0YwG7oDb61m7GdWBwXV9NkzEx024m4lvjUd0wFzj3c4ByqaeKMaP 8QPZZ7ib4DQBOd0ItP/Li5toLKa0ZK7V+h5V7dK7EDiVgfYMLRK4AePQe7oSP6uITobmDGqzMAy BBXwg X-Gm-Gg: Acq92OFMc5rULaLibvTXauIJWlCJU2PlrdUXY6tEdJNXTPw0d5nhMrm14aRjLgn15cE jpXS2m8g9fQMcpa0JPz5imJ+oli3bZGkH/nWK5MjA4C/Oi67NFkW8yg5OUu/Nz+Wv7Dk8IaD7as LCfikyFHvTXJ86LS4ciUbeajt61rz3GHrMiy8xnDxEvCDuhR12hupeZ+UHvoIU2har21CoKNZDF cCy6+OoPxXvT5oLz+KFFpnsbJd34CSL3N11sCSYypLVA7TZeDdZiuBqLOYoc0MBS/Z2a85PE35J xDuMXvKI4ifLHkm3OSjNpCdh91oFUM9YJX44MnllJuOudeKojOekjD2+0yFIE7XIm7V8hHM0jyc WxOwgQBBpKd8vdbZM45rThL+2tl6oJ6LlHc0wYkGNfi5TSv6X1jdpWx++nHkumqo98O5thkeTLe qwbwzF65QTs8rG8elBCSnYw+jxGbZMgM/XzdG2FmuiVIr8gfPMZZueJIvf8AVr20NGdTE1ioQAc w== X-Received: by 2002:a05:600c:4e43:b0:490:be8e:c890 with SMTP id 5b1f17b1804b1-490be8ecff4mr41119695e9.13.1780580091454; Thu, 04 Jun 2026 06:34:51 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:eed3:d0ae:8f44:9e49]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490bc3d6c65sm71326935e9.12.2026.06.04.06.34.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Jun 2026 06:34:50 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 4/5] fetch2: Convert parts of unpack() to use parameter lists Date: Thu, 4 Jun 2026 14:34:45 +0100 Message-ID: <20260604133446.2528212-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260604133446.2528212-1-richard.purdie@linuxfoundation.org> References: <20260604133446.2528212-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 04 Jun 2026 13:34:54 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19608 The unpack function makes heavy use of shell pipelines but start the conversion to avoid shell=True where possible. Signed-off-by: Richard Purdie --- lib/bb/fetch2/__init__.py | 39 +++++++++++++++++++++++---------------- 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 845dd3baab1..58359896602 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1558,7 +1558,10 @@ class FetchMethod(object): unpackdir = rootdir def _run(cmd): - ret = subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True, cwd=unpackdir, env=env) + if isinstance(cmd, str): + ret = subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True, cwd=unpackdir, env=env) + else: + ret = subprocess.call(cmd, preexec_fn=subprocess_setup, cwd=unpackdir, env=env) if ret != 0: raise UnpackError("Unpack command %s failed with return value %s" % (cmd, ret), urldata.url) @@ -1570,36 +1573,36 @@ class FetchMethod(object): cmd = None if unpack: - tar_cmd = 'tar --extract --no-same-owner' + tar_cmd = ['tar', '--extract', '--no-same-owner'] if 'striplevel' in urldata.parm: striplevel = urldata.parm['striplevel'] if not striplevel.isdigit(): raise UnpackError("Invalid striplevel parameter: %s" % striplevel, urldata.url) - tar_cmd += ' --strip-components=%s' % striplevel + tar_cmd.append('--strip-components=%s' % striplevel) if file.endswith('.tar'): - cmd = '%s -f %s' % (tar_cmd, file) + cmd = tar_cmd + ['-f', file] elif file.endswith('.tgz') or file.endswith('.tar.gz') or file.endswith('.tar.Z'): - cmd = '%s -z -f %s' % (tar_cmd, file) + cmd = tar_cmd + ['-z', '-f', file] elif file.endswith('.tbz') or file.endswith('.tbz2') or file.endswith('.tar.bz2'): - cmd = 'bzip2 -dc %s | %s -f -' % (file, tar_cmd) + cmd = 'bzip2 -dc %s | %s -f -' % (file, shlex.join(tar_cmd)) elif file.endswith('.gz') or file.endswith('.Z') or file.endswith('.z'): cmd = 'gzip -dc %s > %s' % (file, efile) elif file.endswith('.bz2'): cmd = 'bzip2 -dc %s > %s' % (file, efile) elif file.endswith('.txz') or file.endswith('.tar.xz'): - cmd = 'xz -dc %s | %s -f -' % (file, tar_cmd) + cmd = 'xz -dc %s | %s -f -' % (file, shlex.join(tar_cmd)) elif file.endswith('.xz'): cmd = 'xz -dc %s > %s' % (file, efile) elif file.endswith('.tar.lz'): - cmd = 'lzip -dc %s | %s -f -' % (file, tar_cmd) + cmd = 'lzip -dc %s | %s -f -' % (file, shlex.join(tar_cmd)) elif file.endswith('.lz'): cmd = 'lzip -dc %s > %s' % (file, efile) elif file.endswith('.tar.7z'): - cmd = '7z x -so %s | %s -f -' % (file, tar_cmd) + cmd = '7z x -so %s | %s -f -' % (file, shlex.join(tar_cmd)) elif file.endswith('.7z'): cmd = '7za x -y %s 1>/dev/null' % file elif file.endswith('.tzst') or file.endswith('.tar.zst'): - cmd = 'zstd --decompress --stdout %s | %s -f -' % (file, tar_cmd) + cmd = 'zstd --decompress --stdout %s | %s -f -' % (file, shlex.join(tar_cmd)) elif file.endswith('.zst'): cmd = 'zstd --decompress --stdout %s > %s' % (file, efile) elif file.endswith('.zip') or file.endswith('.jar'): @@ -1608,10 +1611,10 @@ class FetchMethod(object): except ValueError as exc: bb.fatal("Invalid value for 'dos' parameter for %s: %s" % (file, urldata.parm.get('dos'))) - cmd = 'unzip -q -o' + cmd = ['unzip', '-q', '-o'] if dos: - cmd = '%s -a' % cmd - cmd = "%s '%s'" % (cmd, file) + cmd.append('-a') + cmd.append(file) elif file.endswith('.rpm') or file.endswith('.srpm'): if 'extract' in urldata.parm: unpack_file = urldata.parm.get('extract') @@ -1634,8 +1637,12 @@ class FetchMethod(object): raise UnpackError("Unable to unpack deb/ipk package - does not contain supported data.tar* file", urldata.url) else: raise UnpackError("Unable to unpack deb/ipk package - could not list contents", urldata.url) - quoted_datafile = shlex.quote(datafile) - cmd = 'ar x %s %s && %s -p -f %s && rm %s' % (shlex.quote(file), quoted_datafile, tar_cmd, quoted_datafile, quoted_datafile) + bb.note("Unpacking %s to %s/" % (file, unpackdir)) + urldata.unpack_tracer.unpack("archive-extract", unpackdir) + _run(['ar', 'x', file, datafile]) + _run(tar_cmd + ['-p' ,'-f', datafile]) + os.remove(os.path.join(unpackdir, datafile)) + return if not unpack or not cmd: urldata.unpack_tracer.unpack("file-copy", unpackdir) @@ -1652,7 +1659,7 @@ class FetchMethod(object): if urlpath.find("/") != -1: destdir = urlpath.rsplit("/", 1)[0] + '/' bb.utils.mkdirhier("%s/%s" % (unpackdir, destdir)) - cmd = 'cp --force --preserve=timestamps --no-dereference --recursive -H "%s" "%s"' % (file, destdir) + cmd = ['cp', '--force', '--preserve=timestamps', '--no-dereference', '--recursive', '-H', file, destdir] else: urldata.unpack_tracer.unpack("archive-extract", unpackdir) From patchwork Thu Jun 4 13:34:46 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 89318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27A2CCD6E6B for ; Thu, 4 Jun 2026 13:35:04 +0000 (UTC) Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.11721.1780580094125051543 for ; Thu, 04 Jun 2026 06:34:54 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=Y+mcSMs5; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.43, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-490ac357c55so8822635e9.1 for ; Thu, 04 Jun 2026 06:34:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1780580092; x=1781184892; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=yffhqDEzIKFflDNbr/nW0Kv6kK20OLhLHTJ0Cg0Gzts=; b=Y+mcSMs5he6Njkyv3MBfrnXMTC9fH9+dgw/hRWrPp5TgZbjkX9XmP35Kh/1gRLIRPh 0vrxo4fdu0lv/RcSmqgnK83KsUoi6PP2yJ9FscEDtu3swNUlnBwJipNhaL9CY6KIJ5Wi iEk6XSX4JtzAwCt2VEUvxiP1F0RQA/yav6t2w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780580092; x=1781184892; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=yffhqDEzIKFflDNbr/nW0Kv6kK20OLhLHTJ0Cg0Gzts=; b=muifzun02j0OYs5J//Ys0FwAKPta411gdUULvRwofymqww3mUXMRg9XLB7tMsdRrd+ YOSdc+jFbpc3AiKWAxDwp5XoaHSdJE9Cd+P8kEwrYR/z35kX/v1QyRb7NCZXKUspl9Gh IgrZlIRhL9aXULe0JlkUq+Ux1Ef+bRSM9ERC5tJCeyiCqXSq2EdQUZgcgkNm9ktgQJ6Y 5Wn/kFlNR8sGq65U3ieG/VW7YgK8O9t9CmR9yMTqzttxYUvKsEnESOBWzsUbI4p2lR/V Kz/w+hWCGhltF4p5zDhl6u7oXQRGysagG6cImrkT59sbFcgMeOvk48NaBwHOWpvwhbO5 KMOA== X-Gm-Message-State: AOJu0Yy9L5wjhyg7kHOT3BRI3nLWDy+mOmPRuqBCPGBmdeIVGqoEb8+9 4qPSbLHYPjtESSA9X5tQvFRR4VWVKRxg3cInr/MCgteHSHyHkXIJNP4Q/nxlFwe0tG0LXVxDLFG wb68V X-Gm-Gg: Acq92OGQBoww80DIqlGRuL6Ht27hvK2yXquD8e8mP/6ax46OtUz2KuJ604RPhVkuIhk r6vM5LPAlFFuAwUqJ44P+tWIC3etAD1jkvmxZ+U3Ann5mZKCz0MFz8i7lafCxmuD8XIyJPE+20X aYeHdWCe0MXb5wJLQpA0ZD20OFF/HbrgW/zNtfJo2jBavt3SdfoNxWf9ObP9hCfuvhT6MOqKaF6 6ha58j9jpUqGkAWqTutZT4Gbh9h/hNOFwPbEDm1Fnt9hc9hmbVlYdBzHE3OCyoTxsfJoqnbEI3u 3Cpm9nAdxjhM5HYZA9HNSTPRGUYh4Spjoq3kgO88NWU/MQ2lKTCv/vo/sc2TSF0TsCUI4F8Wbzr faqodx0vF+wXqEFxkYObfgwIX3UKHBuYKpsFjkz4uL6IEqEhgNeRNpk2GiNVRpz0T+95pmsBZR7 owj4xee6tJAb/XHqn9SKnMH7CpGLVujHOQE+LUFthQX050ZRPiMy54d9M4A325onI= X-Received: by 2002:a05:600c:3508:b0:490:5cd8:d21c with SMTP id 5b1f17b1804b1-490b5eb3310mr127259955e9.14.1780580092366; Thu, 04 Jun 2026 06:34:52 -0700 (PDT) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:eed3:d0ae:8f44:9e49]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490bc3d6c65sm71326935e9.12.2026.06.04.06.34.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Jun 2026 06:34:51 -0700 (PDT) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 5/5] fetch: Improve tar pipeline unpack commands Date: Thu, 4 Jun 2026 14:34:46 +0100 Message-ID: <20260604133446.2528212-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260604133446.2528212-1-richard.purdie@linuxfoundation.org> References: <20260604133446.2528212-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 04 Jun 2026 13:35:04 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/19609 When originally implemented, tar didn't support many of the compression options so it was easier for the code to use pipelines. tar does support these now, so we can switch to using them. zstd isn't converted since the --zstd option isn't present in tar v1.30 present on RHEL 8 derived distros. Signed-off-by: Richard Purdie --- lib/bb/fetch2/__init__.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 58359896602..088136bb627 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1584,17 +1584,17 @@ class FetchMethod(object): elif file.endswith('.tgz') or file.endswith('.tar.gz') or file.endswith('.tar.Z'): cmd = tar_cmd + ['-z', '-f', file] elif file.endswith('.tbz') or file.endswith('.tbz2') or file.endswith('.tar.bz2'): - cmd = 'bzip2 -dc %s | %s -f -' % (file, shlex.join(tar_cmd)) + cmd = tar_cmd + ['-j', '-f', file] elif file.endswith('.gz') or file.endswith('.Z') or file.endswith('.z'): cmd = 'gzip -dc %s > %s' % (file, efile) elif file.endswith('.bz2'): cmd = 'bzip2 -dc %s > %s' % (file, efile) elif file.endswith('.txz') or file.endswith('.tar.xz'): - cmd = 'xz -dc %s | %s -f -' % (file, shlex.join(tar_cmd)) + cmd = tar_cmd + ['-J', '-f', file] elif file.endswith('.xz'): cmd = 'xz -dc %s > %s' % (file, efile) elif file.endswith('.tar.lz'): - cmd = 'lzip -dc %s | %s -f -' % (file, shlex.join(tar_cmd)) + cmd = tar_cmd + ['--lzip', '-f', file] elif file.endswith('.lz'): cmd = 'lzip -dc %s > %s' % (file, efile) elif file.endswith('.tar.7z'):