From patchwork Thu Oct 10 16:06:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11A71CFB45C for ; Thu, 10 Oct 2024 16:06:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.47633.1728576390931892995 for ; Thu, 10 Oct 2024 09:06:31 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0320B497 for ; Thu, 10 Oct 2024 09:07:00 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 06F443F58B for ; Thu, 10 Oct 2024 09:06:29 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 01/11] ffmpeg: nasm is x86 only, so only DEPEND if x86 Date: Thu, 10 Oct 2024 17:06:13 +0100 Message-Id: <20241010160623.2880937-1-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Oct 2024 16:06:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205410 No need to depend on nasm if we're not going to use it. Signed-off-by: Ross Burton --- meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb b/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb index 9e60b5cd23e..b16bd51dbd7 100644 --- a/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb +++ b/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb @@ -46,7 +46,8 @@ ARM_INSTRUCTION_SET:armv6 = "arm" # libpostproc was previously packaged from a separate recipe PROVIDES = "libav libpostproc" -DEPENDS = "nasm-native" +DEPENDS:append:x86 = " nasm-native" +DEPENDS:append:x86-64 = " nasm-native" inherit autotools pkgconfig From patchwork Thu Oct 10 16:06:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55E3BCFC5F0 for ; Thu, 10 Oct 2024 16:06:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.47646.1728576391675715585 for ; Thu, 10 Oct 2024 09:06:31 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A81AA497 for ; Thu, 10 Oct 2024 09:07:00 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id ABB2C3F58B for ; Thu, 10 Oct 2024 09:06:30 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 02/11] ffmpeg: no need for textrel INSANE_SKIP Date: Thu, 10 Oct 2024 17:06:14 +0100 Message-Id: <20241010160623.2880937-2-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241010160623.2880937-1-ross.burton@arm.com> References: <20241010160623.2880937-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Oct 2024 16:06:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205411 It appears in testing that modern ffmpeg no longer needs to disable PIC, so there's no need to ignore textrel warnings. Signed-off-by: Ross Burton --- meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb b/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb index b16bd51dbd7..af66104ebf3 100644 --- a/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb +++ b/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb @@ -181,13 +181,3 @@ FILES:libpostproc = "${libdir}/libpostproc${SOLIBS}" FILES:libswresample = "${libdir}/libswresample${SOLIBS}" FILES:libswscale = "${libdir}/libswscale${SOLIBS}" FILES:${PN}-examples = "${datadir}/${PN}/examples" - -# ffmpeg disables PIC on some platforms (e.g. x86-32) -INSANE_SKIP:${MLPREFIX}libavcodec = "textrel" -INSANE_SKIP:${MLPREFIX}libavdevice = "textrel" -INSANE_SKIP:${MLPREFIX}libavfilter = "textrel" -INSANE_SKIP:${MLPREFIX}libavformat = "textrel" -INSANE_SKIP:${MLPREFIX}libavutil = "textrel" -INSANE_SKIP:${MLPREFIX}libswscale = "textrel" -INSANE_SKIP:${MLPREFIX}libswresample = "textrel" -INSANE_SKIP:${MLPREFIX}libpostproc = "textrel" From patchwork Thu Oct 10 16:06:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 144F6CFC5E1 for ; Thu, 10 Oct 2024 16:06:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.47519.1728576392226378976 for ; Thu, 10 Oct 2024 09:06:32 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 58D80DA7 for ; Thu, 10 Oct 2024 09:07:01 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5C7D53F58B for ; Thu, 10 Oct 2024 09:06:31 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 03/11] libjpeg-turbo: remove redundant RUNPATH entries Date: Thu, 10 Oct 2024 17:06:15 +0100 Message-Id: <20241010160623.2880937-3-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241010160623.2880937-1-ross.burton@arm.com> References: <20241010160623.2880937-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Oct 2024 16:06:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205412 Our RPATH sanity checks currently don't check RUNPATH (patch incoming), but the libjpeg objects are built with a RUNPATH set to ${libdir}. RUNPATH is less of a problem than RPATH (because it is used after the standard search), but redundant entries are still redundant so lets remove them anyway. Signed-off-by: Ross Burton --- meta/recipes-graphics/jpeg/libjpeg-turbo_3.0.1.bb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/meta/recipes-graphics/jpeg/libjpeg-turbo_3.0.1.bb b/meta/recipes-graphics/jpeg/libjpeg-turbo_3.0.1.bb index 132d4b375aa..4ff4cec721b 100644 --- a/meta/recipes-graphics/jpeg/libjpeg-turbo_3.0.1.bb +++ b/meta/recipes-graphics/jpeg/libjpeg-turbo_3.0.1.bb @@ -55,4 +55,9 @@ FILES:jpeg-tools = "${bindir}/*" DESCRIPTION:libturbojpeg = "A SIMD-accelerated JPEG codec which provides only TurboJPEG APIs" FILES:libturbojpeg = "${libdir}/libturbojpeg.so.*" +do_install:append() { + # The binaries have RUNPATH=$libdir, which is redundant + chrpath -d ${D}/${bindir}/* ${D}${libdir}/*${SOLIBS} +} + BBCLASSEXTEND = "native nativesdk" From patchwork Thu Oct 10 16:06:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50260 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 610DECFC5F1 for ; Thu, 10 Oct 2024 16:06:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.47521.1728576392692033621 for ; Thu, 10 Oct 2024 09:06:32 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0268B497 for ; Thu, 10 Oct 2024 09:07:02 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0E8303F58B for ; Thu, 10 Oct 2024 09:06:31 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 04/11] oe/elf: don't regenerate machine data on every call Date: Thu, 10 Oct 2024 17:06:16 +0100 Message-Id: <20241010160623.2880937-4-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241010160623.2880937-1-ross.burton@arm.com> References: <20241010160623.2880937-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Oct 2024 16:06:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205413 Every time oe.elf.machine_dict() is called a large dictionary is created and returned. However, the "arch" package_qa test will call this method for every binary in a package, which results in a lot of dictionary creation. Concrete exmaple: in running ltp:do_package_qa, the arch test takes 25% of the runtime, and opitimising the machine_dict() call to cache the generated dictionary reduces the runtime from 57s to 44s. Signed-off-by: Ross Burton --- meta/lib/oe/elf.py | 264 +++++++++++++++++++++++---------------------- 1 file changed, 133 insertions(+), 131 deletions(-) diff --git a/meta/lib/oe/elf.py b/meta/lib/oe/elf.py index eab2349a4fe..e1bc7b89846 100644 --- a/meta/lib/oe/elf.py +++ b/meta/lib/oe/elf.py @@ -5,141 +5,143 @@ # def machine_dict(d): -# TARGET_OS TARGET_ARCH MACHINE, OSABI, ABIVERSION, Little Endian, 32bit? - machdata = { - "darwin9" : { - "arm" : (40, 0, 0, True, 32), - }, - "eabi" : { - "arm" : (40, 0, 0, True, 32), - }, - "elf" : { - "aarch64" : (183, 0, 0, True, 64), - "aarch64_be" :(183, 0, 0, False, 64), - "i586" : (3, 0, 0, True, 32), - "i686" : (3, 0, 0, True, 32), - "x86_64": (62, 0, 0, True, 64), - "epiphany": (4643, 0, 0, True, 32), - "lm32": (138, 0, 0, False, 32), - "loongarch64":(258, 0, 0, True, 64), - "mips": ( 8, 0, 0, False, 32), - "mipsel": ( 8, 0, 0, True, 32), - "microblaze": (189, 0, 0, False, 32), - "microblazeel":(189, 0, 0, True, 32), - "powerpc": (20, 0, 0, False, 32), - "riscv32": (243, 0, 0, True, 32), - "riscv64": (243, 0, 0, True, 64), - }, - "linux" : { - "aarch64" : (183, 0, 0, True, 64), - "aarch64_be" :(183, 0, 0, False, 64), - "arm" : (40, 97, 0, True, 32), - "armeb": (40, 97, 0, False, 32), - "powerpc": (20, 0, 0, False, 32), - "powerpc64": (21, 0, 0, False, 64), - "powerpc64le": (21, 0, 0, True, 64), - "i386": ( 3, 0, 0, True, 32), - "i486": ( 3, 0, 0, True, 32), - "i586": ( 3, 0, 0, True, 32), - "i686": ( 3, 0, 0, True, 32), - "x86_64": (62, 0, 0, True, 64), - "ia64": (50, 0, 0, True, 64), - "alpha": (36902, 0, 0, True, 64), - "hppa": (15, 3, 0, False, 32), - "loongarch64":(258, 0, 0, True, 64), - "m68k": ( 4, 0, 0, False, 32), - "mips": ( 8, 0, 0, False, 32), - "mipsel": ( 8, 0, 0, True, 32), - "mips64": ( 8, 0, 0, False, 64), - "mips64el": ( 8, 0, 0, True, 64), - "mipsisa32r6": ( 8, 0, 0, False, 32), - "mipsisa32r6el": ( 8, 0, 0, True, 32), - "mipsisa64r6": ( 8, 0, 0, False, 64), - "mipsisa64r6el": ( 8, 0, 0, True, 64), - "nios2": (113, 0, 0, True, 32), - "riscv32": (243, 0, 0, True, 32), - "riscv64": (243, 0, 0, True, 64), - "s390": (22, 0, 0, False, 32), - "sh4": (42, 0, 0, True, 32), - "sparc": ( 2, 0, 0, False, 32), - "microblaze": (189, 0, 0, False, 32), - "microblazeel":(189, 0, 0, True, 32), - }, - "linux-android" : { - "aarch64" : (183, 0, 0, True, 64), - "i686": ( 3, 0, 0, True, 32), - "x86_64": (62, 0, 0, True, 64), - }, - "linux-androideabi" : { - "arm" : (40, 97, 0, True, 32), - }, - "linux-musl" : { - "aarch64" : (183, 0, 0, True, 64), - "aarch64_be" :(183, 0, 0, False, 64), - "arm" : ( 40, 97, 0, True, 32), - "armeb": ( 40, 97, 0, False, 32), - "powerpc": ( 20, 0, 0, False, 32), - "powerpc64": ( 21, 0, 0, False, 64), - "powerpc64le": (21, 0, 0, True, 64), - "i386": ( 3, 0, 0, True, 32), - "i486": ( 3, 0, 0, True, 32), - "i586": ( 3, 0, 0, True, 32), - "i686": ( 3, 0, 0, True, 32), - "x86_64": ( 62, 0, 0, True, 64), - "mips": ( 8, 0, 0, False, 32), - "mipsel": ( 8, 0, 0, True, 32), - "mips64": ( 8, 0, 0, False, 64), - "mips64el": ( 8, 0, 0, True, 64), - "microblaze": (189, 0, 0, False, 32), - "microblazeel":(189, 0, 0, True, 32), - "riscv32": (243, 0, 0, True, 32), - "riscv64": (243, 0, 0, True, 64), - "sh4": ( 42, 0, 0, True, 32), - }, - "uclinux-uclibc" : { - "bfin": ( 106, 0, 0, True, 32), - }, - "linux-gnueabi" : { - "arm" : (40, 0, 0, True, 32), - "armeb" : (40, 0, 0, False, 32), - }, - "linux-musleabi" : { - "arm" : (40, 0, 0, True, 32), - "armeb" : (40, 0, 0, False, 32), - }, - "linux-gnuspe" : { - "powerpc": (20, 0, 0, False, 32), - }, - "linux-muslspe" : { - "powerpc": (20, 0, 0, False, 32), - }, - "linux-gnu" : { - "powerpc": (20, 0, 0, False, 32), - "sh4": (42, 0, 0, True, 32), - }, - "linux-gnu_ilp32" : { - "aarch64" : (183, 0, 0, True, 32), - }, - "linux-gnux32" : { - "x86_64": (62, 0, 0, True, 32), - }, - "linux-muslx32" : { - "x86_64": (62, 0, 0, True, 32), - }, - "linux-gnun32" : { - "mips64": ( 8, 0, 0, False, 32), - "mips64el": ( 8, 0, 0, True, 32), - "mipsisa64r6": ( 8, 0, 0, False, 32), - "mipsisa64r6el":( 8, 0, 0, True, 32), - }, - } + # Generating this data is slow, so cache it + if not hasattr(machine_dict, "machdata"): + machine_dict.machdata = { + # TARGET_OS TARGET_ARCH MACHINE, OSABI, ABIVERSION, Little Endian, 32bit? + "darwin9" : { + "arm" : (40, 0, 0, True, 32), + }, + "eabi" : { + "arm" : (40, 0, 0, True, 32), + }, + "elf" : { + "aarch64" : (183, 0, 0, True, 64), + "aarch64_be" :(183, 0, 0, False, 64), + "i586" : (3, 0, 0, True, 32), + "i686" : (3, 0, 0, True, 32), + "x86_64": (62, 0, 0, True, 64), + "epiphany": (4643, 0, 0, True, 32), + "lm32": (138, 0, 0, False, 32), + "loongarch64":(258, 0, 0, True, 64), + "mips": ( 8, 0, 0, False, 32), + "mipsel": ( 8, 0, 0, True, 32), + "microblaze": (189, 0, 0, False, 32), + "microblazeel":(189, 0, 0, True, 32), + "powerpc": (20, 0, 0, False, 32), + "riscv32": (243, 0, 0, True, 32), + "riscv64": (243, 0, 0, True, 64), + }, + "linux" : { + "aarch64" : (183, 0, 0, True, 64), + "aarch64_be" :(183, 0, 0, False, 64), + "arm" : (40, 97, 0, True, 32), + "armeb": (40, 97, 0, False, 32), + "powerpc": (20, 0, 0, False, 32), + "powerpc64": (21, 0, 0, False, 64), + "powerpc64le": (21, 0, 0, True, 64), + "i386": ( 3, 0, 0, True, 32), + "i486": ( 3, 0, 0, True, 32), + "i586": ( 3, 0, 0, True, 32), + "i686": ( 3, 0, 0, True, 32), + "x86_64": (62, 0, 0, True, 64), + "ia64": (50, 0, 0, True, 64), + "alpha": (36902, 0, 0, True, 64), + "hppa": (15, 3, 0, False, 32), + "loongarch64":(258, 0, 0, True, 64), + "m68k": ( 4, 0, 0, False, 32), + "mips": ( 8, 0, 0, False, 32), + "mipsel": ( 8, 0, 0, True, 32), + "mips64": ( 8, 0, 0, False, 64), + "mips64el": ( 8, 0, 0, True, 64), + "mipsisa32r6": ( 8, 0, 0, False, 32), + "mipsisa32r6el": ( 8, 0, 0, True, 32), + "mipsisa64r6": ( 8, 0, 0, False, 64), + "mipsisa64r6el": ( 8, 0, 0, True, 64), + "nios2": (113, 0, 0, True, 32), + "riscv32": (243, 0, 0, True, 32), + "riscv64": (243, 0, 0, True, 64), + "s390": (22, 0, 0, False, 32), + "sh4": (42, 0, 0, True, 32), + "sparc": ( 2, 0, 0, False, 32), + "microblaze": (189, 0, 0, False, 32), + "microblazeel":(189, 0, 0, True, 32), + }, + "linux-android" : { + "aarch64" : (183, 0, 0, True, 64), + "i686": ( 3, 0, 0, True, 32), + "x86_64": (62, 0, 0, True, 64), + }, + "linux-androideabi" : { + "arm" : (40, 97, 0, True, 32), + }, + "linux-musl" : { + "aarch64" : (183, 0, 0, True, 64), + "aarch64_be" :(183, 0, 0, False, 64), + "arm" : ( 40, 97, 0, True, 32), + "armeb": ( 40, 97, 0, False, 32), + "powerpc": ( 20, 0, 0, False, 32), + "powerpc64": ( 21, 0, 0, False, 64), + "powerpc64le": (21, 0, 0, True, 64), + "i386": ( 3, 0, 0, True, 32), + "i486": ( 3, 0, 0, True, 32), + "i586": ( 3, 0, 0, True, 32), + "i686": ( 3, 0, 0, True, 32), + "x86_64": ( 62, 0, 0, True, 64), + "mips": ( 8, 0, 0, False, 32), + "mipsel": ( 8, 0, 0, True, 32), + "mips64": ( 8, 0, 0, False, 64), + "mips64el": ( 8, 0, 0, True, 64), + "microblaze": (189, 0, 0, False, 32), + "microblazeel":(189, 0, 0, True, 32), + "riscv32": (243, 0, 0, True, 32), + "riscv64": (243, 0, 0, True, 64), + "sh4": ( 42, 0, 0, True, 32), + }, + "uclinux-uclibc" : { + "bfin": ( 106, 0, 0, True, 32), + }, + "linux-gnueabi" : { + "arm" : (40, 0, 0, True, 32), + "armeb" : (40, 0, 0, False, 32), + }, + "linux-musleabi" : { + "arm" : (40, 0, 0, True, 32), + "armeb" : (40, 0, 0, False, 32), + }, + "linux-gnuspe" : { + "powerpc": (20, 0, 0, False, 32), + }, + "linux-muslspe" : { + "powerpc": (20, 0, 0, False, 32), + }, + "linux-gnu" : { + "powerpc": (20, 0, 0, False, 32), + "sh4": (42, 0, 0, True, 32), + }, + "linux-gnu_ilp32" : { + "aarch64" : (183, 0, 0, True, 32), + }, + "linux-gnux32" : { + "x86_64": (62, 0, 0, True, 32), + }, + "linux-muslx32" : { + "x86_64": (62, 0, 0, True, 32), + }, + "linux-gnun32" : { + "mips64": ( 8, 0, 0, False, 32), + "mips64el": ( 8, 0, 0, True, 32), + "mipsisa64r6": ( 8, 0, 0, False, 32), + "mipsisa64r6el":( 8, 0, 0, True, 32), + }, + } # Add in any extra user supplied data which may come from a BSP layer, removing the # need to always change this class directly extra_machdata = (d and d.getVar("PACKAGEQA_EXTRA_MACHDEFFUNCS" or None) or "").split() for m in extra_machdata: call = m + "(machdata, d)" - locs = { "machdata" : machdata, "d" : d} - machdata = bb.utils.better_eval(call, locs) + locs = { "machdata" : machine_dict.machdata, "d" : d} + machine_dict.machdata = bb.utils.better_eval(call, locs) - return machdata + return machine_dict.machdata From patchwork Thu Oct 10 16:06:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21DA8CFC5E4 for ; Thu, 10 Oct 2024 16:06:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.47647.1728576393519270397 for ; Thu, 10 Oct 2024 09:06:33 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9EA01497 for ; Thu, 10 Oct 2024 09:07:02 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AAD7A3F58B for ; Thu, 10 Oct 2024 09:06:32 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 05/11] insane: only parse ELFs if they're files, not symlinks Date: Thu, 10 Oct 2024 17:06:17 +0100 Message-Id: <20241010160623.2880937-5-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241010160623.2880937-1-ross.burton@arm.com> References: <20241010160623.2880937-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Oct 2024 16:06:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205414 This reduces the number of files that need to be swept by not scanning eg the library symlinks, and means we can remove the explicit islink() checks in many of the tests. Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index 1691d96b64a..7f01908e18e 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -122,9 +122,6 @@ def package_qa_check_rpath(file,name, d, elf): if not elf: return - if os.path.islink(file): - return - bad_dirs = [d.getVar('BASE_WORKDIR'), d.getVar('STAGING_DIR_TARGET')] phdrs = elf.run_objdump("-p", d) @@ -150,9 +147,6 @@ def package_qa_check_useless_rpaths(file, name, d, elf): if not elf: return - if os.path.islink(file): - return - libdir = d.getVar("libdir") base_libdir = d.getVar("base_libdir") @@ -337,11 +331,6 @@ def package_qa_check_arch(path,name,d, elf): oe.qa.handle_error("arch", pn + ": Recipe inherits the allarch class, but has packaged architecture-specific binaries", d) return - # avoid following links to /usr/bin (e.g. on udev builds) - # we will check the files pointed to anyway... - if os.path.islink(path): - return - #if this will throw an exception, then fix the dict above (machine, osabi, abiversion, littleendian, bits) \ = oe.elf.machine_dict(d)[host_os][host_arch] @@ -383,9 +372,6 @@ def package_qa_textrel(path, name, d, elf): if not elf: return - if os.path.islink(path): - return - phdrs = elf.run_objdump("-p", d) import re @@ -405,9 +391,6 @@ def package_qa_hash_style(path, name, d, elf): if not elf: return - if os.path.islink(path): - return - gnu_hash = "--hash-style=gnu" in d.getVar('LDFLAGS') if not gnu_hash: gnu_hash = "--hash-style=both" in d.getVar('LDFLAGS') @@ -601,7 +584,7 @@ def check_32bit_symbols(path, packagename, d, elf): ) # elf is a oe.qa.ELFFile object - if elf is not None: + if elf: phdrs = elf.run_objdump("-tw", d) syms = re.finditer(ptrn, phdrs) usedapis = {sym.group('notag') for sym in syms} @@ -789,7 +772,7 @@ def package_qa_walk(checkfuncs, package, d): elves = {} for path in pkgfiles[package]: elf = None - if os.path.isfile(path): + if os.path.isfile(path) and not os.path.islink(path): elf = oe.qa.ELFFile(path) try: elf.open() From patchwork Thu Oct 10 16:06:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E637CFC5E9 for ; Thu, 10 Oct 2024 16:06:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.47523.1728576394204847748 for ; Thu, 10 Oct 2024 09:06:34 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4710E497 for ; Thu, 10 Oct 2024 09:07:03 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 536853F58B for ; Thu, 10 Oct 2024 09:06:33 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 06/11] insane: check for RUNPATH as well as RPATH Date: Thu, 10 Oct 2024 17:06:18 +0100 Message-Id: <20241010160623.2880937-6-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241010160623.2880937-1-ross.burton@arm.com> References: <20241010160623.2880937-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Oct 2024 16:06:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205415 Since oe-core 66f8a7 merged in 2023[1], ld sets DT_RUNPATH instead of DT_RPATH when -rpath is specified, which we don't check for. Update the insane tests to look at both RPATH and RUNPATH. [1] oe-core 66f8a745668a067d8d763fa2af3e65f26c9c1ebe Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index 7f01908e18e..401807b2be3 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -115,7 +115,7 @@ def package_qa_check_libexec(path,name, d, elf): oe.qa.handle_error("libexec", "%s: %s is using libexec please relocate to %s" % (name, package_qa_clean_path(path, d, name), libexec), d) QAPATHTEST[rpaths] = "package_qa_check_rpath" -def package_qa_check_rpath(file,name, d, elf): +def package_qa_check_rpath(file, name, d, elf): """ Check for dangerous RPATHs """ @@ -127,14 +127,14 @@ def package_qa_check_rpath(file,name, d, elf): phdrs = elf.run_objdump("-p", d) import re - rpath_re = re.compile(r"\s+RPATH\s+(.*)") + rpath_re = re.compile(r"\s+(?:RPATH|RUNPATH)\s+(.*)") for line in phdrs.split("\n"): m = rpath_re.match(line) if m: rpath = m.group(1) for dir in bad_dirs: if dir in rpath: - oe.qa.handle_error("rpaths", "package %s contains bad RPATH %s in file %s" % (name, rpath, file), d) + oe.qa.handle_error("rpaths", "%s: %s contains bad RPATH %s" % (name, package_qa_clean_path(file, d, name), rpath), d) QAPATHTEST[useless-rpaths] = "package_qa_check_useless_rpaths" def package_qa_check_useless_rpaths(file, name, d, elf): @@ -153,7 +153,7 @@ def package_qa_check_useless_rpaths(file, name, d, elf): phdrs = elf.run_objdump("-p", d) import re - rpath_re = re.compile(r"\s+RPATH\s+(.*)") + rpath_re = re.compile(r"\s+(?:RPATH|RUNPATH)\s+(.*)") for line in phdrs.split("\n"): m = rpath_re.match(line) if m: From patchwork Thu Oct 10 16:06:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50258 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C81CCFC5EC for ; Thu, 10 Oct 2024 16:06:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.47524.1728576394606770716 for ; Thu, 10 Oct 2024 09:06:34 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E3608497 for ; Thu, 10 Oct 2024 09:07:03 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id EFCB33F58B for ; Thu, 10 Oct 2024 09:06:33 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 07/11] insane: tidy up objdump preloading in package_qa_walk() Date: Thu, 10 Oct 2024 17:06:19 +0100 Message-Id: <20241010160623.2880937-7-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241010160623.2880937-1-ross.burton@arm.com> References: <20241010160623.2880937-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Oct 2024 16:06:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205416 Move the prepopulate function out of global scope, and access the dictionary once instead of repeatedly. This still results in each ELF being opened twice, but this avoids opening all of the files at once and the ELFFile.open() call is fairly fast. Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index 401807b2be3..b165f111cec 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -763,10 +763,6 @@ def qa_check_staged(path,d): if not skip_shebang_size: package_qa_check_shebang_size(path, "", d, None) -def prepopulate_objdump_p(elf, d): - output = elf.run_objdump("-p", d) - return (elf.name, output) - # Walk over all files in a directory and call func def package_qa_walk(checkfuncs, package, d): elves = {} @@ -782,17 +778,22 @@ def package_qa_walk(checkfuncs, package, d): if elf: elves[path] = elf + def prepopulate_objdump_p(elf, d): + output = elf.run_objdump("-p", d) + return (elf.name, output) + results = oe.utils.multiprocess_launch(prepopulate_objdump_p, elves.values(), d, extraargs=(d,)) for item in results: elves[item[0]].set_objdump("-p", item[1]) for path in pkgfiles[package]: - if path in elves: - elves[path].open() - for func in checkfuncs: - func(path, package, d, elves.get(path)) - if path in elves: - elves[path].close() + elf = elves.get(path) + if elf: + elf.open() + for func in checkfuncs: + func(path, package, d, elf) + if elf: + elf.close() def package_qa_check_rdepends(pkg, pkgdest, skip, taskdeps, packages, d): # Don't do this check for kernel/module recipes, there aren't too many debug/development From patchwork Thu Oct 10 16:06:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50256 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D40ECFC5EB for ; Thu, 10 Oct 2024 16:06:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.47648.1728576395417135287 for ; Thu, 10 Oct 2024 09:06:35 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8BEEE497 for ; Thu, 10 Oct 2024 09:07:04 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9809D3F58B for ; Thu, 10 Oct 2024 09:06:34 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 08/11] insane: micro-optimise the sweep of pkgfiles Date: Thu, 10 Oct 2024 17:06:20 +0100 Message-Id: <20241010160623.2880937-8-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241010160623.2880937-1-ross.burton@arm.com> References: <20241010160623.2880937-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Oct 2024 16:06:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205417 Don't actively do more work: - Exit early if there are no packages being generated - Don't iterate repeatedly when removing CONTROL and DEBIAN - Extend a list with another list instead of appending item by item - Remove unused variables Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index b165f111cec..05b8538940d 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -1084,13 +1084,8 @@ parse_test_matrix[vardepsexclude] = "ERROR_QA WARN_QA" # The PACKAGE FUNC to scan each package python do_package_qa () { - import subprocess import oe.packagedata - bb.note("DO PACKAGE QA") - - main_lic = d.getVar('LICENSE') - # Check for obsolete license references in main LICENSE (packages are checked below for any changes) main_licenses = oe.license.list_licenses(d.getVar('LICENSE')) obsolete = set(oe.license.obsolete_license_list()) & main_licenses @@ -1106,27 +1101,27 @@ python do_package_qa () { pn = d.getVar('PN') # Scan the packages... - pkgdest = d.getVar('PKGDEST') packages = set((d.getVar('PACKAGES') or '').split()) + # no packages should be scanned + if not packages: + return global pkgfiles pkgfiles = {} + pkgdest = d.getVar('PKGDEST') for pkg in packages: - pkgfiles[pkg] = [] pkgdir = os.path.join(pkgdest, pkg) + pkgfiles[pkg] = [] for walkroot, dirs, files in os.walk(pkgdir): # Don't walk into top-level CONTROL or DEBIAN directories as these # are temporary directories created by do_package. if walkroot == pkgdir: - for control in ("CONTROL", "DEBIAN"): - if control in dirs: - dirs.remove(control) - for file in files: - pkgfiles[pkg].append(os.path.join(walkroot, file)) - - # no packages should be scanned - if not packages: - return + for removedir in ("CONTROL", "DEBIAN"): + try: + dirs.remove(removedir) + except ValueError: + pass + pkgfiles[pkg].extend((os.path.join(walkroot, f) for f in files)) import re # The package name matches the [a-z0-9.+-]+ regular expression From patchwork Thu Oct 10 16:06:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D843CFC5E8 for ; Thu, 10 Oct 2024 16:06:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.47525.1728576396139497526 for ; Thu, 10 Oct 2024 09:06:36 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4D6E7497 for ; Thu, 10 Oct 2024 09:07:05 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 412B33F58B for ; Thu, 10 Oct 2024 09:06:35 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 09/11] insane: optimise test in package_qa_check_shebang_size Date: Thu, 10 Oct 2024 17:06:21 +0100 Message-Id: <20241010160623.2880937-9-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241010160623.2880937-1-ross.burton@arm.com> References: <20241010160623.2880937-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Oct 2024 16:06:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205418 Check whether the elf is not None first, before doing os.stat() calls on disk. Also don't check anything that isn't a file, not just FIFOs. Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index 05b8538940d..bec349e97cc 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -82,8 +82,7 @@ def package_qa_clean_path(path, d, pkg=None): QAPATHTEST[shebang-size] = "package_qa_check_shebang_size" def package_qa_check_shebang_size(path, name, d, elf): - import stat - if os.path.islink(path) or stat.S_ISFIFO(os.stat(path).st_mode) or elf: + if elf or os.path.islink(path) or not os.path.isfile(path): return try: From patchwork Thu Oct 10 16:06:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50254 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C913CFC5E7 for ; Thu, 10 Oct 2024 16:06:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.47649.1728576396605338151 for ; Thu, 10 Oct 2024 09:06:36 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DA096DA7 for ; Thu, 10 Oct 2024 09:07:05 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E688B3F58B for ; Thu, 10 Oct 2024 09:06:35 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 10/11] insane: use oe.cachedpath.CachedPath instead of os.path Date: Thu, 10 Oct 2024 17:06:22 +0100 Message-Id: <20241010160623.2880937-10-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241010160623.2880937-1-ross.burton@arm.com> References: <20241010160623.2880937-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Oct 2024 16:06:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205419 The insane QAPATHTESTs make many os.stat() calls, the majority of which are redundant with caching as the initial sweep does a stat() on every entry to determine if it is a file or a directory, and from then on each test that does further stat()s is redundant as the tree doesn't change. Switch os.stat() and friends (os.path.isfile(), etc) to use a common oe.cachedpath.CachedPath() instance that is shared between all of the functions, meaning only one stat is done. In my test case of ltp:do_package_qa, this reduces the time taken from 44s to 37s. Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 42 ++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 14 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index bec349e97cc..2d69bdec8d3 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -82,7 +82,9 @@ def package_qa_clean_path(path, d, pkg=None): QAPATHTEST[shebang-size] = "package_qa_check_shebang_size" def package_qa_check_shebang_size(path, name, d, elf): - if elf or os.path.islink(path) or not os.path.isfile(path): + global cpath + + if elf or cpath.islink(path) or not cpath.isfile(path): return try: @@ -167,8 +169,8 @@ def package_qa_check_dev(path, name, d, elf): """ Check for ".so" library symlinks in non-dev packages """ - - if not name.endswith("-dev") and not name.endswith("-dbg") and not name.endswith("-ptest") and not name.startswith("nativesdk-") and path.endswith(".so") and os.path.islink(path): + global cpath + if not name.endswith("-dev") and not name.endswith("-dbg") and not name.endswith("-ptest") and not name.startswith("nativesdk-") and path.endswith(".so") and cpath.islink(path): oe.qa.handle_error("dev-so", "non -dev/-dbg/nativesdk- package %s contains symlink .so '%s'" % \ (name, package_qa_clean_path(path, d, name)), d) @@ -179,7 +181,8 @@ def package_qa_check_dev_elf(path, name, d, elf): check that the file is not a link and is an ELF object as some recipes install link-time .so files that are linker scripts. """ - if name.endswith("-dev") and path.endswith(".so") and not os.path.islink(path) and elf: + global cpath + if name.endswith("-dev") and path.endswith(".so") and not cpath.islink(path) and elf: oe.qa.handle_error("dev-elf", "-dev package %s contains non-symlink .so '%s'" % \ (name, package_qa_clean_path(path, d, name)), d) @@ -422,9 +425,9 @@ def package_qa_check_buildpaths(path, name, d, elf): explicitly ignored. """ import stat - + global cpath # Ignore symlinks/devs/fifos - mode = os.lstat(path).st_mode + mode = cpath.lstat(path).st_mode if stat.S_ISLNK(mode) or stat.S_ISBLK(mode) or stat.S_ISFIFO(mode) or stat.S_ISCHR(mode) or stat.S_ISSOCK(mode): return @@ -469,7 +472,8 @@ def package_qa_check_symlink_to_sysroot(path, name, d, elf): """ Check that the package doesn't contain any absolute symlinks to the sysroot. """ - if os.path.islink(path): + global cpath + if cpath.islink(path): target = os.readlink(path) if os.path.isabs(target): tmpdir = d.getVar('TMPDIR') @@ -760,14 +764,19 @@ def qa_check_staged(path,d): oe.qa.handle_error("pkgconfig", error_msg, d) if not skip_shebang_size: + global cpath + cpath = oe.cachedpath.CachedPath() package_qa_check_shebang_size(path, "", d, None) + cpath = None # Walk over all files in a directory and call func def package_qa_walk(checkfuncs, package, d): + global cpath + elves = {} for path in pkgfiles[package]: elf = None - if os.path.isfile(path) and not os.path.islink(path): + if cpath.isfile(path) and not cpath.islink(path): elf = oe.qa.ELFFile(path) try: elf.open() @@ -915,11 +924,12 @@ def package_qa_check_deps(pkg, pkgdest, d): QAPKGTEST[usrmerge] = "package_qa_check_usrmerge" def package_qa_check_usrmerge(pkg, d): + global cpath pkgdest = d.getVar('PKGDEST') pkg_dir = pkgdest + os.sep + pkg + os.sep merged_dirs = ['bin', 'sbin', 'lib'] + d.getVar('MULTILIB_VARIANTS').split() for f in merged_dirs: - if os.path.exists(pkg_dir + f) and not os.path.islink(pkg_dir + f): + if cpath.exists(pkg_dir + f) and not cpath.islink(pkg_dir + f): msg = "%s package is not obeying usrmerge distro feature. /%s should be relocated to /usr." % (pkg, f) oe.qa.handle_error("usrmerge", msg, d) return @@ -985,10 +995,11 @@ def package_qa_check_empty_dirs(pkg, d): empty. """ + global cpath pkgd = oe.path.join(d.getVar('PKGDEST'), pkg) for dir in (d.getVar('QA_EMPTY_DIRS') or "").split(): empty_dir = oe.path.join(pkgd, dir) - if os.path.exists(empty_dir) and os.listdir(empty_dir): + if cpath.exists(empty_dir) and os.listdir(empty_dir): recommendation = (d.getVar('QA_EMPTY_DIRS_RECOMMENDATION:' + dir) or "but it is expected to be empty") msg = "%s installs files in %s, %s" % (pkg, dir, recommendation) @@ -1018,8 +1029,9 @@ HOST_USER_GID := "${@os.getgid()}" QAPATHTEST[host-user-contaminated] = "package_qa_check_host_user" def package_qa_check_host_user(path, name, d, elf): """Check for paths outside of /home which are owned by the user running bitbake.""" + global cpath - if not os.path.lexists(path): + if not cpath.lexists(path): return dest = d.getVar('PKGDEST') @@ -1029,7 +1041,7 @@ def package_qa_check_host_user(path, name, d, elf): return try: - stat = os.lstat(path) + stat = cpath.lstat(path) except OSError as exc: import errno if exc.errno != errno.ENOENT: @@ -1105,13 +1117,14 @@ python do_package_qa () { if not packages: return - global pkgfiles + global pkgfiles, cpath pkgfiles = {} + cpath = oe.cachedpath.CachedPath() pkgdest = d.getVar('PKGDEST') for pkg in packages: pkgdir = os.path.join(pkgdest, pkg) pkgfiles[pkg] = [] - for walkroot, dirs, files in os.walk(pkgdir): + for walkroot, dirs, files in cpath.walk(pkgdir): # Don't walk into top-level CONTROL or DEBIAN directories as these # are temporary directories created by do_package. if walkroot == pkgdir: @@ -1159,6 +1172,7 @@ python do_package_qa () { package_qa_check_libdir(d) + cpath = None oe.qa.exit_if_errors(d) } From patchwork Thu Oct 10 16:06:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50253 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 171EECFC5E2 for ; Thu, 10 Oct 2024 16:06:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.47526.1728576397256351687 for ; Thu, 10 Oct 2024 09:06:37 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8192F497 for ; Thu, 10 Oct 2024 09:07:06 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8E4603F58B for ; Thu, 10 Oct 2024 09:06:36 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH v2 11/11] insane: rewrite package_qa_check_arch Date: Thu, 10 Oct 2024 17:06:23 +0100 Message-Id: <20241010160623.2880937-11-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241010160623.2880937-1-ross.burton@arm.com> References: <20241010160623.2880937-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Oct 2024 16:06:41 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205420 Reorder and comment the architecture checks to make it clearer what they are actually checking. Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 41 ++++++++++++++++++------------ 1 file changed, 25 insertions(+), 16 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index 2d69bdec8d3..36b139a20d0 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -325,32 +325,41 @@ def package_qa_check_arch(path,name,d, elf): host_os = d.getVar('HOST_OS') host_arch = d.getVar('HOST_ARCH') - provides = d.getVar('PROVIDES') - bpn = d.getVar('BPN') + provides = d.getVar('PROVIDES') if host_arch == "allarch": - pn = d.getVar('PN') - oe.qa.handle_error("arch", pn + ": Recipe inherits the allarch class, but has packaged architecture-specific binaries", d) + oe.qa.handle_error("arch", "%s: inherits the allarch class, but has architecture-specific binaries %s" % \ + (name, package_qa_clean_path(path, d, name)), d) return - #if this will throw an exception, then fix the dict above - (machine, osabi, abiversion, littleendian, bits) \ + # If this throws an exception, the machine_dict needs expanding + (expected_machine, expected_osabi, expected_abiversion, expected_littleendian, expected_bits) \ = oe.elf.machine_dict(d)[host_os][host_arch] + actual_machine = elf.machine() + actual_bits = elf.abiSize() + actual_littleendian = elf.isLittleEndian() + + # BPF don't match the target + if oe.qa.elf_machine_to_string(actual_machine) == "BPF": + return + + # These targets have 32-bit userspace but 64-bit kernel, so fudge the expected values + if (("virtual/kernel" in provides) or bb.data.inherits_class("module", d)) and (host_os in ("linux-gnux32", "linux-muslx32", "linux-gnu_ilp32") or re.match(r'mips64.*32', d.getVar('DEFAULTTUNE'))): + expected_bits = 64 + # Check the architecture and endiannes of the binary - is_32 = (("virtual/kernel" in provides) or bb.data.inherits_class("module", d)) and \ - (host_os == "linux-gnux32" or host_os == "linux-muslx32" or \ - host_os == "linux-gnu_ilp32" or re.match(r'mips64.*32', d.getVar('DEFAULTTUNE'))) - is_bpf = (oe.qa.elf_machine_to_string(elf.machine()) == "BPF") - if not ((machine == elf.machine()) or is_32 or is_bpf): + if expected_machine != actual_machine: oe.qa.handle_error("arch", "Architecture did not match (%s, expected %s) in %s" % \ - (oe.qa.elf_machine_to_string(elf.machine()), oe.qa.elf_machine_to_string(machine), package_qa_clean_path(path, d, name)), d) - elif not ((bits == elf.abiSize()) or is_32 or is_bpf): + (oe.qa.elf_machine_to_string(actual_machine), oe.qa.elf_machine_to_string(expected_machine), package_qa_clean_path(path, d, name)), d) + + if expected_bits != actual_bits: oe.qa.handle_error("arch", "Bit size did not match (%d, expected %d) in %s" % \ - (elf.abiSize(), bits, package_qa_clean_path(path, d, name)), d) - elif not ((littleendian == elf.isLittleEndian()) or is_bpf): + (actual_bits, expected_bits, package_qa_clean_path(path, d, name)), d) + + if expected_littleendian != actual_littleendian: oe.qa.handle_error("arch", "Endiannes did not match (%d, expected %d) in %s" % \ - (elf.isLittleEndian(), littleendian, package_qa_clean_path(path, d, name)), d) + (actual_littleendian, expected_littleendian, package_qa_clean_path(path, d, name)), d) package_qa_check_arch[vardepsexclude] = "DEFAULTTUNE" QAPATHTEST[desktop] = "package_qa_check_desktop"