From patchwork Tue May 14 01:53:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Antonin Godard X-Patchwork-Id: 43529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A196FC25B78 for ; Tue, 14 May 2024 01:53:18 +0000 (UTC) Received: from mail-4316.protonmail.ch (mail-4316.protonmail.ch [185.70.43.16]) by mx.groups.io with SMTP id smtpd.web11.5173.1715651593382705176 for ; Mon, 13 May 2024 18:53:13 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@pm.me header.s=protonmail3 header.b=IXwGz0qy; spf=pass (domain: pm.me, ip: 185.70.43.16, mailfrom: antoningodard@pm.me) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail3; t=1715651591; x=1715910791; bh=WbeF5GQcbqBqzjEyYWwOjthnka3HSaDB8SnLAw9dFbI=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=IXwGz0qyuqXDZ68vbDv2yG3xGFIQ6zf1F1F2OsjA4j+NIrCne9Hf9eTde77MLV+hK JxaaxCDsh8SsJi0pJxPTAd2JWCr7uy2Am8AEM9jB5fYWB4Kmm+YdZPm9SpcD/g1/Rf j+rkLVK0UTfDZ//Qzj6hKeAwv+vnee/ONoazcujVO5Oj9gxStgnqr6762jvceeheCH nTT00oNsF5trzEz7R29RCBm7zadhhGemwx7yEO7vkQGKVbHEYu1iJxhKQ8AtVuJpJQ Mmmwjbo060NuIrzeWTA77uPxyLWJlES4+eM5nJOEnz5F6I9xOY4gVePKKXDxFevy9q Rtv6Ep5wXtqPQ== Date: Tue, 14 May 2024 01:53:06 +0000 To: bitbake-devel@lists.openembedded.org From: Antonin Godard Cc: Antonin Godard Subject: [PATCH 1/3] codeparser: support shell substitutions in quotes Message-ID: <20240514015234.67318-2-antoningodard@pm.me> In-Reply-To: <20240514015234.67318-1-antoningodard@pm.me> References: <20240514015234.67318-1-antoningodard@pm.me> Feedback-ID: 17651652:user:proton X-Pm-Message-ID: 9edd3f9f6a3556c4388c095d1aedaa79b1f21fee MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 14 May 2024 01:53:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16212 The current shell substitution mechanism only works without quotes. For example: var1=$(cmd1 ...) Will work and add `cmd1` to the correspondind `run.do_*` file. However, although quite common, this syntax is not supported: var1="$(cmd1 ...)" This commit adds this feature by adding a step to process_words() to check whether we are dealing with quotes first, and by iterating on what's between them to detect new shell substitution candidates. These candidates are tested and parsed like before in the next step. The original `part` being part of the candidates means the syntax var1=$(cmd1 ...) is still valid. Signed-off-by: Antonin Godard --- lib/bb/codeparser.py | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py index 2e8b7ced3..c613806c8 100644 --- a/lib/bb/codeparser.py +++ b/lib/bb/codeparser.py @@ -490,13 +490,28 @@ class ShellParser(): if not isinstance(part, list): continue - if part[0] in ('`', '$('): - command = pyshlex.wordtree_as_string(part[1:-1]) - self._parse_shell(command) - - if word[0] in ("cmd_name", "cmd_word"): - if word in words: - words.remove(word) + candidates = [part] + + # If command is of type: + # + # var="... $(cmd [...]) ..." + # + # Then iterate on what's between the quotes and if we find a + # list, make that what we check for below. + if len(part) >= 3 and part[0] == '"': + for p in part[1:-1]: + if isinstance(p, list): + candidates.append(p) + + for candidate in candidates: + if len(candidate) >= 2: + if candidate[0] in ('`', '$('): + command = pyshlex.wordtree_as_string(candidate[1:-1]) + self._parse_shell(command) + + if word[0] in ("cmd_name", "cmd_word"): + if word in words: + words.remove(word) usetoken = False for word in words: From patchwork Tue May 14 01:53:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Antonin Godard X-Patchwork-Id: 43530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9978DC25B79 for ; Tue, 14 May 2024 01:53:28 +0000 (UTC) Received: from mail-4316.protonmail.ch (mail-4316.protonmail.ch [185.70.43.16]) by mx.groups.io with SMTP id smtpd.web10.5145.1715651599863402152 for ; Mon, 13 May 2024 18:53:20 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@pm.me header.s=protonmail3 header.b=FbFSLaVd; spf=pass (domain: pm.me, ip: 185.70.43.16, mailfrom: antoningodard@pm.me) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail3; t=1715651598; x=1715910798; bh=6G2tecOUMlA6iupAKncgNxcXujwvcn/b+8F6FU27I6w=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=FbFSLaVdnxaTqDV4eJOEtYgX7ntexd2jXkDnZqCNpbUMQvghad54EF7L1ccg8EboM BGjYHMbAlJ3lRuAqIHadR6LcTF0pDrFU8a1OIaICkql7KhrU9mWInAgVXCB3Jd6/qM nJlbb2qoNJXxTQxdAi6LUNMICn4P1kDiSDtTXPJ0zN4ZFqz9zk4yRHait7SP0OlJeF 7L5vVh5HqG6CUpKqPSgd5SyxjdC78wZodhHm2lzR/Llm9XDbXhMp2O14i8E59LKrpY SzVqVOCo96bpeXzvTqH9OxUv+ZgVYACTJ2yon+nTkPR696158tbyEWMlq1yYGeOIce xPPvOYTkZDoCw== Date: Tue, 14 May 2024 01:53:13 +0000 To: bitbake-devel@lists.openembedded.org From: Antonin Godard Cc: Antonin Godard Subject: [PATCH 2/3] codeparser: remove redundant list conversion Message-ID: <20240514015234.67318-3-antoningodard@pm.me> In-Reply-To: <20240514015234.67318-1-antoningodard@pm.me> References: <20240514015234.67318-1-antoningodard@pm.me> Feedback-ID: 17651652:user:proton X-Pm-Message-ID: 55658fdcdbdbf05119c8bca3f94066189cc0188f MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 14 May 2024 01:53:28 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16213 Signed-off-by: Antonin Godard --- lib/bb/codeparser.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py index c613806c8..691bdff75 100644 --- a/lib/bb/codeparser.py +++ b/lib/bb/codeparser.py @@ -484,7 +484,7 @@ class ShellParser(): """ words = list(words) - for word in list(words): + for word in words: wtree = pyshlex.make_wordtree(word[1]) for part in wtree: if not isinstance(part, list): From patchwork Tue May 14 01:53:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Antonin Godard X-Patchwork-Id: 43531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A8B1C25B78 for ; Tue, 14 May 2024 01:53:28 +0000 (UTC) Received: from mail-4322.protonmail.ch (mail-4322.protonmail.ch [185.70.43.22]) by mx.groups.io with SMTP id smtpd.web10.5147.1715651606498955040 for ; Mon, 13 May 2024 18:53:26 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@pm.me header.s=protonmail3 header.b=IokB8CO0; spf=pass (domain: pm.me, ip: 185.70.43.22, mailfrom: antoningodard@pm.me) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail3; t=1715651604; x=1715910804; bh=e4ID/A/jz5V+/0zM76Kx/KPz3oJql3KB3ClRo+Pmi4M=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=IokB8CO0AVEuajlPRWcx4WgXPWW+Xxil+mooe6aagijqB8PxRR+hFyEQPygKFTshU yF/YjgCyn41cHtsuxsSVSPu0FUo53w/cknsvdDeDWxrWnKDiN0eZZKEupncMfAsL52 i1Wq71reDpFckJVZxvi4xy4zZ9FN8xEj5dE+oGu8kaxPqJZ5vD29duvweuou3UmJmi vllCo5pGfgJI23iCX7OluAp9U0rF6RBHqxGzhtVlgq1yn1ZaAzspNNwIrjtAe7VOQz C+Rs8SLgPWIgAqcBKO4uSIS52M4woNmMtJ2Ez0lpugCr/azXj3DdDNfDQmMIs/Gx5o Zch3YLkbzNO2g== Date: Tue, 14 May 2024 01:53:21 +0000 To: bitbake-devel@lists.openembedded.org From: Antonin Godard Cc: Antonin Godard Subject: [PATCH 3/3] tests.codeparser: add tests for shell expansions Message-ID: <20240514015234.67318-4-antoningodard@pm.me> In-Reply-To: <20240514015234.67318-1-antoningodard@pm.me> References: <20240514015234.67318-1-antoningodard@pm.me> Feedback-ID: 17651652:user:proton X-Pm-Message-ID: 920e3077656712ea94219c6d7cb9ed27e251b181 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 14 May 2024 01:53:28 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/16214 Tests quotes around `` and $() expansions, nested and multiple expansions, and that escaped quotes are treated as characters by the parser. Signed-off-by: Antonin Godard --- lib/bb/tests/codeparser.py | 40 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/lib/bb/tests/codeparser.py b/lib/bb/tests/codeparser.py index f6585fb3a..c0d1362a0 100644 --- a/lib/bb/tests/codeparser.py +++ b/lib/bb/tests/codeparser.py @@ -106,6 +106,46 @@ ${D}${libdir}/pkgconfig/*.pc self.parseExpression("foo=$(echo bar)") self.assertExecs(set(["echo"])) + def test_assign_subshell_expansion_quotes(self): + self.parseExpression('foo="$(echo bar)"') + self.assertExecs(set(["echo"])) + + def test_assign_subshell_expansion_nested(self): + self.parseExpression('foo="$(func1 "$(func2 bar$(func3))")"') + self.assertExecs(set(["func1", "func2", "func3"])) + + def test_assign_subshell_expansion_multiple(self): + self.parseExpression('foo="$(func1 "$(func2)") $(func3)"') + self.assertExecs(set(["func1", "func2", "func3"])) + + def test_assign_subshell_expansion_escaped_quotes(self): + self.parseExpression('foo="\\"fo\\"o$(func1)"') + self.assertExecs(set(["func1"])) + + def test_assign_subshell_expansion_empty(self): + self.parseExpression('foo="bar$()foo"') + self.assertExecs(set()) + + def test_assign_subshell_backticks(self): + self.parseExpression("foo=`echo bar`") + self.assertExecs(set(["echo"])) + + def test_assign_subshell_backticks_quotes(self): + self.parseExpression('foo="`echo bar`"') + self.assertExecs(set(["echo"])) + + def test_assign_subshell_backticks_multiple(self): + self.parseExpression('foo="`func1 bar` `func2`"') + self.assertExecs(set(["func1", "func2"])) + + def test_assign_subshell_backticks_escaped_quotes(self): + self.parseExpression('foo="\\"fo\\"o`func1`"') + self.assertExecs(set(["func1"])) + + def test_assign_subshell_backticks_empty(self): + self.parseExpression('foo="bar``foo"') + self.assertExecs(set()) + def test_shell_unexpanded(self): self.setEmptyVars(["QT_BASE_NAME"]) self.parseExpression('echo "${QT_BASE_NAME}"')