From patchwork Tue Oct 8 19:30:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1294BCEFC4A for ; Tue, 8 Oct 2024 19:31:14 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.4652.1728415870265203034 for ; Tue, 08 Oct 2024 12:31:10 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CB4B3DA7 for ; Tue, 8 Oct 2024 12:31:38 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B03A13F73F for ; Tue, 8 Oct 2024 12:31:08 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 01/10] ffmpeg: nasm is x86 only, so only DEPEND if x86 Date: Tue, 8 Oct 2024 20:30:54 +0100 Message-Id: <20241008193103.2429829-1-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 19:31:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205301 No need to depend on nasm if we're not going to use it. Signed-off-by: Ross Burton --- meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb b/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb index 9e60b5cd23e..b16bd51dbd7 100644 --- a/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb +++ b/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb @@ -46,7 +46,8 @@ ARM_INSTRUCTION_SET:armv6 = "arm" # libpostproc was previously packaged from a separate recipe PROVIDES = "libav libpostproc" -DEPENDS = "nasm-native" +DEPENDS:append:x86 = " nasm-native" +DEPENDS:append:x86-64 = " nasm-native" inherit autotools pkgconfig From patchwork Tue Oct 8 19:30:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22F06CEFC4D for ; Tue, 8 Oct 2024 19:31:14 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.4697.1728415870562519378 for ; Tue, 08 Oct 2024 12:31:10 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AF810FEC for ; Tue, 8 Oct 2024 12:31:39 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 990DF3F73F for ; Tue, 8 Oct 2024 12:31:09 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 02/10] ffmpeg: no need for textrel INSANE_SKIP Date: Tue, 8 Oct 2024 20:30:55 +0100 Message-Id: <20241008193103.2429829-2-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241008193103.2429829-1-ross.burton@arm.com> References: <20241008193103.2429829-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 19:31:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205302 It appears in testing that modern ffmpeg no longer needs to disable PIC, so there's no need to ignore textrel warnings. Signed-off-by: Ross Burton --- meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb b/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb index b16bd51dbd7..af66104ebf3 100644 --- a/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb +++ b/meta/recipes-multimedia/ffmpeg/ffmpeg_7.0.2.bb @@ -181,13 +181,3 @@ FILES:libpostproc = "${libdir}/libpostproc${SOLIBS}" FILES:libswresample = "${libdir}/libswresample${SOLIBS}" FILES:libswscale = "${libdir}/libswscale${SOLIBS}" FILES:${PN}-examples = "${datadir}/${PN}/examples" - -# ffmpeg disables PIC on some platforms (e.g. x86-32) -INSANE_SKIP:${MLPREFIX}libavcodec = "textrel" -INSANE_SKIP:${MLPREFIX}libavdevice = "textrel" -INSANE_SKIP:${MLPREFIX}libavfilter = "textrel" -INSANE_SKIP:${MLPREFIX}libavformat = "textrel" -INSANE_SKIP:${MLPREFIX}libavutil = "textrel" -INSANE_SKIP:${MLPREFIX}libswscale = "textrel" -INSANE_SKIP:${MLPREFIX}libswresample = "textrel" -INSANE_SKIP:${MLPREFIX}libpostproc = "textrel" From patchwork Tue Oct 8 19:30:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50060 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 221E8CEFC49 for ; Tue, 8 Oct 2024 19:31:14 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.4698.1728415871418724833 for ; Tue, 08 Oct 2024 12:31:11 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A4297DA7 for ; Tue, 8 Oct 2024 12:31:40 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7DF1A3F73F for ; Tue, 8 Oct 2024 12:31:10 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 03/10] libjpeg-turbo: remove redundant RUNPATH entries Date: Tue, 8 Oct 2024 20:30:56 +0100 Message-Id: <20241008193103.2429829-3-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241008193103.2429829-1-ross.burton@arm.com> References: <20241008193103.2429829-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 19:31:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205303 Our RPATH sanity checks currently don't check RUNPATH (patch incoming), but the libjpeg objects are built with a RUNPATH set to ${libdir}. RUNPATH is less of a problem than RPATH (because it is used after the standard search), but redundant entries are still redundant so lets remove them anyway. Signed-off-by: Ross Burton --- meta/recipes-graphics/jpeg/libjpeg-turbo_3.0.1.bb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/meta/recipes-graphics/jpeg/libjpeg-turbo_3.0.1.bb b/meta/recipes-graphics/jpeg/libjpeg-turbo_3.0.1.bb index 132d4b375aa..4ff4cec721b 100644 --- a/meta/recipes-graphics/jpeg/libjpeg-turbo_3.0.1.bb +++ b/meta/recipes-graphics/jpeg/libjpeg-turbo_3.0.1.bb @@ -55,4 +55,9 @@ FILES:jpeg-tools = "${bindir}/*" DESCRIPTION:libturbojpeg = "A SIMD-accelerated JPEG codec which provides only TurboJPEG APIs" FILES:libturbojpeg = "${libdir}/libturbojpeg.so.*" +do_install:append() { + # The binaries have RUNPATH=$libdir, which is redundant + chrpath -d ${D}/${bindir}/* ${D}${libdir}/*${SOLIBS} +} + BBCLASSEXTEND = "native nativesdk" From patchwork Tue Oct 8 19:30:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50058 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F3D4CEFC48 for ; Tue, 8 Oct 2024 19:31:14 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.4699.1728415872475054963 for ; Tue, 08 Oct 2024 12:31:12 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A03D8DA7 for ; Tue, 8 Oct 2024 12:31:41 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6B34F3F73F for ; Tue, 8 Oct 2024 12:31:11 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 04/10] oe/elf: don't regenerate machine data on every call Date: Tue, 8 Oct 2024 20:30:57 +0100 Message-Id: <20241008193103.2429829-4-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241008193103.2429829-1-ross.burton@arm.com> References: <20241008193103.2429829-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 19:31:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205304 Every time oe.elf.machine_dict() is called a large dictionary is created and returned. However, the "arch" package_qa test will call this method for every binary in a package, which results in a lot of dictionary creation. Concrete exmaple: in running ltp:do_package_qa, the arch test takes 25% of the runtime, and opitimising the machine_dict() call to cache the generated dictionary reduces the runtime from 57s to 44s. Signed-off-by: Ross Burton --- meta/lib/oe/elf.py | 264 +++++++++++++++++++++++---------------------- 1 file changed, 133 insertions(+), 131 deletions(-) diff --git a/meta/lib/oe/elf.py b/meta/lib/oe/elf.py index eab2349a4fe..e1bc7b89846 100644 --- a/meta/lib/oe/elf.py +++ b/meta/lib/oe/elf.py @@ -5,141 +5,143 @@ # def machine_dict(d): -# TARGET_OS TARGET_ARCH MACHINE, OSABI, ABIVERSION, Little Endian, 32bit? - machdata = { - "darwin9" : { - "arm" : (40, 0, 0, True, 32), - }, - "eabi" : { - "arm" : (40, 0, 0, True, 32), - }, - "elf" : { - "aarch64" : (183, 0, 0, True, 64), - "aarch64_be" :(183, 0, 0, False, 64), - "i586" : (3, 0, 0, True, 32), - "i686" : (3, 0, 0, True, 32), - "x86_64": (62, 0, 0, True, 64), - "epiphany": (4643, 0, 0, True, 32), - "lm32": (138, 0, 0, False, 32), - "loongarch64":(258, 0, 0, True, 64), - "mips": ( 8, 0, 0, False, 32), - "mipsel": ( 8, 0, 0, True, 32), - "microblaze": (189, 0, 0, False, 32), - "microblazeel":(189, 0, 0, True, 32), - "powerpc": (20, 0, 0, False, 32), - "riscv32": (243, 0, 0, True, 32), - "riscv64": (243, 0, 0, True, 64), - }, - "linux" : { - "aarch64" : (183, 0, 0, True, 64), - "aarch64_be" :(183, 0, 0, False, 64), - "arm" : (40, 97, 0, True, 32), - "armeb": (40, 97, 0, False, 32), - "powerpc": (20, 0, 0, False, 32), - "powerpc64": (21, 0, 0, False, 64), - "powerpc64le": (21, 0, 0, True, 64), - "i386": ( 3, 0, 0, True, 32), - "i486": ( 3, 0, 0, True, 32), - "i586": ( 3, 0, 0, True, 32), - "i686": ( 3, 0, 0, True, 32), - "x86_64": (62, 0, 0, True, 64), - "ia64": (50, 0, 0, True, 64), - "alpha": (36902, 0, 0, True, 64), - "hppa": (15, 3, 0, False, 32), - "loongarch64":(258, 0, 0, True, 64), - "m68k": ( 4, 0, 0, False, 32), - "mips": ( 8, 0, 0, False, 32), - "mipsel": ( 8, 0, 0, True, 32), - "mips64": ( 8, 0, 0, False, 64), - "mips64el": ( 8, 0, 0, True, 64), - "mipsisa32r6": ( 8, 0, 0, False, 32), - "mipsisa32r6el": ( 8, 0, 0, True, 32), - "mipsisa64r6": ( 8, 0, 0, False, 64), - "mipsisa64r6el": ( 8, 0, 0, True, 64), - "nios2": (113, 0, 0, True, 32), - "riscv32": (243, 0, 0, True, 32), - "riscv64": (243, 0, 0, True, 64), - "s390": (22, 0, 0, False, 32), - "sh4": (42, 0, 0, True, 32), - "sparc": ( 2, 0, 0, False, 32), - "microblaze": (189, 0, 0, False, 32), - "microblazeel":(189, 0, 0, True, 32), - }, - "linux-android" : { - "aarch64" : (183, 0, 0, True, 64), - "i686": ( 3, 0, 0, True, 32), - "x86_64": (62, 0, 0, True, 64), - }, - "linux-androideabi" : { - "arm" : (40, 97, 0, True, 32), - }, - "linux-musl" : { - "aarch64" : (183, 0, 0, True, 64), - "aarch64_be" :(183, 0, 0, False, 64), - "arm" : ( 40, 97, 0, True, 32), - "armeb": ( 40, 97, 0, False, 32), - "powerpc": ( 20, 0, 0, False, 32), - "powerpc64": ( 21, 0, 0, False, 64), - "powerpc64le": (21, 0, 0, True, 64), - "i386": ( 3, 0, 0, True, 32), - "i486": ( 3, 0, 0, True, 32), - "i586": ( 3, 0, 0, True, 32), - "i686": ( 3, 0, 0, True, 32), - "x86_64": ( 62, 0, 0, True, 64), - "mips": ( 8, 0, 0, False, 32), - "mipsel": ( 8, 0, 0, True, 32), - "mips64": ( 8, 0, 0, False, 64), - "mips64el": ( 8, 0, 0, True, 64), - "microblaze": (189, 0, 0, False, 32), - "microblazeel":(189, 0, 0, True, 32), - "riscv32": (243, 0, 0, True, 32), - "riscv64": (243, 0, 0, True, 64), - "sh4": ( 42, 0, 0, True, 32), - }, - "uclinux-uclibc" : { - "bfin": ( 106, 0, 0, True, 32), - }, - "linux-gnueabi" : { - "arm" : (40, 0, 0, True, 32), - "armeb" : (40, 0, 0, False, 32), - }, - "linux-musleabi" : { - "arm" : (40, 0, 0, True, 32), - "armeb" : (40, 0, 0, False, 32), - }, - "linux-gnuspe" : { - "powerpc": (20, 0, 0, False, 32), - }, - "linux-muslspe" : { - "powerpc": (20, 0, 0, False, 32), - }, - "linux-gnu" : { - "powerpc": (20, 0, 0, False, 32), - "sh4": (42, 0, 0, True, 32), - }, - "linux-gnu_ilp32" : { - "aarch64" : (183, 0, 0, True, 32), - }, - "linux-gnux32" : { - "x86_64": (62, 0, 0, True, 32), - }, - "linux-muslx32" : { - "x86_64": (62, 0, 0, True, 32), - }, - "linux-gnun32" : { - "mips64": ( 8, 0, 0, False, 32), - "mips64el": ( 8, 0, 0, True, 32), - "mipsisa64r6": ( 8, 0, 0, False, 32), - "mipsisa64r6el":( 8, 0, 0, True, 32), - }, - } + # Generating this data is slow, so cache it + if not hasattr(machine_dict, "machdata"): + machine_dict.machdata = { + # TARGET_OS TARGET_ARCH MACHINE, OSABI, ABIVERSION, Little Endian, 32bit? + "darwin9" : { + "arm" : (40, 0, 0, True, 32), + }, + "eabi" : { + "arm" : (40, 0, 0, True, 32), + }, + "elf" : { + "aarch64" : (183, 0, 0, True, 64), + "aarch64_be" :(183, 0, 0, False, 64), + "i586" : (3, 0, 0, True, 32), + "i686" : (3, 0, 0, True, 32), + "x86_64": (62, 0, 0, True, 64), + "epiphany": (4643, 0, 0, True, 32), + "lm32": (138, 0, 0, False, 32), + "loongarch64":(258, 0, 0, True, 64), + "mips": ( 8, 0, 0, False, 32), + "mipsel": ( 8, 0, 0, True, 32), + "microblaze": (189, 0, 0, False, 32), + "microblazeel":(189, 0, 0, True, 32), + "powerpc": (20, 0, 0, False, 32), + "riscv32": (243, 0, 0, True, 32), + "riscv64": (243, 0, 0, True, 64), + }, + "linux" : { + "aarch64" : (183, 0, 0, True, 64), + "aarch64_be" :(183, 0, 0, False, 64), + "arm" : (40, 97, 0, True, 32), + "armeb": (40, 97, 0, False, 32), + "powerpc": (20, 0, 0, False, 32), + "powerpc64": (21, 0, 0, False, 64), + "powerpc64le": (21, 0, 0, True, 64), + "i386": ( 3, 0, 0, True, 32), + "i486": ( 3, 0, 0, True, 32), + "i586": ( 3, 0, 0, True, 32), + "i686": ( 3, 0, 0, True, 32), + "x86_64": (62, 0, 0, True, 64), + "ia64": (50, 0, 0, True, 64), + "alpha": (36902, 0, 0, True, 64), + "hppa": (15, 3, 0, False, 32), + "loongarch64":(258, 0, 0, True, 64), + "m68k": ( 4, 0, 0, False, 32), + "mips": ( 8, 0, 0, False, 32), + "mipsel": ( 8, 0, 0, True, 32), + "mips64": ( 8, 0, 0, False, 64), + "mips64el": ( 8, 0, 0, True, 64), + "mipsisa32r6": ( 8, 0, 0, False, 32), + "mipsisa32r6el": ( 8, 0, 0, True, 32), + "mipsisa64r6": ( 8, 0, 0, False, 64), + "mipsisa64r6el": ( 8, 0, 0, True, 64), + "nios2": (113, 0, 0, True, 32), + "riscv32": (243, 0, 0, True, 32), + "riscv64": (243, 0, 0, True, 64), + "s390": (22, 0, 0, False, 32), + "sh4": (42, 0, 0, True, 32), + "sparc": ( 2, 0, 0, False, 32), + "microblaze": (189, 0, 0, False, 32), + "microblazeel":(189, 0, 0, True, 32), + }, + "linux-android" : { + "aarch64" : (183, 0, 0, True, 64), + "i686": ( 3, 0, 0, True, 32), + "x86_64": (62, 0, 0, True, 64), + }, + "linux-androideabi" : { + "arm" : (40, 97, 0, True, 32), + }, + "linux-musl" : { + "aarch64" : (183, 0, 0, True, 64), + "aarch64_be" :(183, 0, 0, False, 64), + "arm" : ( 40, 97, 0, True, 32), + "armeb": ( 40, 97, 0, False, 32), + "powerpc": ( 20, 0, 0, False, 32), + "powerpc64": ( 21, 0, 0, False, 64), + "powerpc64le": (21, 0, 0, True, 64), + "i386": ( 3, 0, 0, True, 32), + "i486": ( 3, 0, 0, True, 32), + "i586": ( 3, 0, 0, True, 32), + "i686": ( 3, 0, 0, True, 32), + "x86_64": ( 62, 0, 0, True, 64), + "mips": ( 8, 0, 0, False, 32), + "mipsel": ( 8, 0, 0, True, 32), + "mips64": ( 8, 0, 0, False, 64), + "mips64el": ( 8, 0, 0, True, 64), + "microblaze": (189, 0, 0, False, 32), + "microblazeel":(189, 0, 0, True, 32), + "riscv32": (243, 0, 0, True, 32), + "riscv64": (243, 0, 0, True, 64), + "sh4": ( 42, 0, 0, True, 32), + }, + "uclinux-uclibc" : { + "bfin": ( 106, 0, 0, True, 32), + }, + "linux-gnueabi" : { + "arm" : (40, 0, 0, True, 32), + "armeb" : (40, 0, 0, False, 32), + }, + "linux-musleabi" : { + "arm" : (40, 0, 0, True, 32), + "armeb" : (40, 0, 0, False, 32), + }, + "linux-gnuspe" : { + "powerpc": (20, 0, 0, False, 32), + }, + "linux-muslspe" : { + "powerpc": (20, 0, 0, False, 32), + }, + "linux-gnu" : { + "powerpc": (20, 0, 0, False, 32), + "sh4": (42, 0, 0, True, 32), + }, + "linux-gnu_ilp32" : { + "aarch64" : (183, 0, 0, True, 32), + }, + "linux-gnux32" : { + "x86_64": (62, 0, 0, True, 32), + }, + "linux-muslx32" : { + "x86_64": (62, 0, 0, True, 32), + }, + "linux-gnun32" : { + "mips64": ( 8, 0, 0, False, 32), + "mips64el": ( 8, 0, 0, True, 32), + "mipsisa64r6": ( 8, 0, 0, False, 32), + "mipsisa64r6el":( 8, 0, 0, True, 32), + }, + } # Add in any extra user supplied data which may come from a BSP layer, removing the # need to always change this class directly extra_machdata = (d and d.getVar("PACKAGEQA_EXTRA_MACHDEFFUNCS" or None) or "").split() for m in extra_machdata: call = m + "(machdata, d)" - locs = { "machdata" : machdata, "d" : d} - machdata = bb.utils.better_eval(call, locs) + locs = { "machdata" : machine_dict.machdata, "d" : d} + machine_dict.machdata = bb.utils.better_eval(call, locs) - return machdata + return machine_dict.machdata From patchwork Tue Oct 8 19:30:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BA1ACEFC42 for ; Tue, 8 Oct 2024 19:31:14 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.4654.1728415873164008535 for ; Tue, 08 Oct 2024 12:31:13 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 67090DA7 for ; Tue, 8 Oct 2024 12:31:42 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4F5113F73F for ; Tue, 8 Oct 2024 12:31:12 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 05/10] insane: only parse ELFs if they're files, not symlinks Date: Tue, 8 Oct 2024 20:30:58 +0100 Message-Id: <20241008193103.2429829-5-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241008193103.2429829-1-ross.burton@arm.com> References: <20241008193103.2429829-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 19:31:14 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205305 This reduces the number of files that need to be swept by not scanning eg the library symlinks, and means we can remove the explicit islink() checks in many of the tests. Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index 1691d96b64a..7f01908e18e 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -122,9 +122,6 @@ def package_qa_check_rpath(file,name, d, elf): if not elf: return - if os.path.islink(file): - return - bad_dirs = [d.getVar('BASE_WORKDIR'), d.getVar('STAGING_DIR_TARGET')] phdrs = elf.run_objdump("-p", d) @@ -150,9 +147,6 @@ def package_qa_check_useless_rpaths(file, name, d, elf): if not elf: return - if os.path.islink(file): - return - libdir = d.getVar("libdir") base_libdir = d.getVar("base_libdir") @@ -337,11 +331,6 @@ def package_qa_check_arch(path,name,d, elf): oe.qa.handle_error("arch", pn + ": Recipe inherits the allarch class, but has packaged architecture-specific binaries", d) return - # avoid following links to /usr/bin (e.g. on udev builds) - # we will check the files pointed to anyway... - if os.path.islink(path): - return - #if this will throw an exception, then fix the dict above (machine, osabi, abiversion, littleendian, bits) \ = oe.elf.machine_dict(d)[host_os][host_arch] @@ -383,9 +372,6 @@ def package_qa_textrel(path, name, d, elf): if not elf: return - if os.path.islink(path): - return - phdrs = elf.run_objdump("-p", d) import re @@ -405,9 +391,6 @@ def package_qa_hash_style(path, name, d, elf): if not elf: return - if os.path.islink(path): - return - gnu_hash = "--hash-style=gnu" in d.getVar('LDFLAGS') if not gnu_hash: gnu_hash = "--hash-style=both" in d.getVar('LDFLAGS') @@ -601,7 +584,7 @@ def check_32bit_symbols(path, packagename, d, elf): ) # elf is a oe.qa.ELFFile object - if elf is not None: + if elf: phdrs = elf.run_objdump("-tw", d) syms = re.finditer(ptrn, phdrs) usedapis = {sym.group('notag') for sym in syms} @@ -789,7 +772,7 @@ def package_qa_walk(checkfuncs, package, d): elves = {} for path in pkgfiles[package]: elf = None - if os.path.isfile(path): + if os.path.isfile(path) and not os.path.islink(path): elf = oe.qa.ELFFile(path) try: elf.open() From patchwork Tue Oct 8 19:30:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50064 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EB93CEFC42 for ; Tue, 8 Oct 2024 19:31:24 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.4700.1728415874126736411 for ; Tue, 08 Oct 2024 12:31:14 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 50078DA7 for ; Tue, 8 Oct 2024 12:31:43 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 382893F73F for ; Tue, 8 Oct 2024 12:31:13 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 06/10] insane: check for RUNPATH as well as RPATH Date: Tue, 8 Oct 2024 20:30:59 +0100 Message-Id: <20241008193103.2429829-6-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241008193103.2429829-1-ross.burton@arm.com> References: <20241008193103.2429829-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 19:31:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205306 Since oe-core 66f8a7 merged in 2023[1], ld sets DT_RUNPATH instead of DT_RPATH when -rpath is specified, which we don't check for. Update the insane tests to look at both RPATH and RUNPATH. [1] oe-core 66f8a745668a067d8d763fa2af3e65f26c9c1ebe Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index 7f01908e18e..401807b2be3 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -115,7 +115,7 @@ def package_qa_check_libexec(path,name, d, elf): oe.qa.handle_error("libexec", "%s: %s is using libexec please relocate to %s" % (name, package_qa_clean_path(path, d, name), libexec), d) QAPATHTEST[rpaths] = "package_qa_check_rpath" -def package_qa_check_rpath(file,name, d, elf): +def package_qa_check_rpath(file, name, d, elf): """ Check for dangerous RPATHs """ @@ -127,14 +127,14 @@ def package_qa_check_rpath(file,name, d, elf): phdrs = elf.run_objdump("-p", d) import re - rpath_re = re.compile(r"\s+RPATH\s+(.*)") + rpath_re = re.compile(r"\s+(?:RPATH|RUNPATH)\s+(.*)") for line in phdrs.split("\n"): m = rpath_re.match(line) if m: rpath = m.group(1) for dir in bad_dirs: if dir in rpath: - oe.qa.handle_error("rpaths", "package %s contains bad RPATH %s in file %s" % (name, rpath, file), d) + oe.qa.handle_error("rpaths", "%s: %s contains bad RPATH %s" % (name, package_qa_clean_path(file, d, name), rpath), d) QAPATHTEST[useless-rpaths] = "package_qa_check_useless_rpaths" def package_qa_check_useless_rpaths(file, name, d, elf): @@ -153,7 +153,7 @@ def package_qa_check_useless_rpaths(file, name, d, elf): phdrs = elf.run_objdump("-p", d) import re - rpath_re = re.compile(r"\s+RPATH\s+(.*)") + rpath_re = re.compile(r"\s+(?:RPATH|RUNPATH)\s+(.*)") for line in phdrs.split("\n"): m = rpath_re.match(line) if m: From patchwork Tue Oct 8 19:31:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DC9ECEFC50 for ; Tue, 8 Oct 2024 19:31:24 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.4655.1728415875434379673 for ; Tue, 08 Oct 2024 12:31:15 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 58A77DA7 for ; Tue, 8 Oct 2024 12:31:44 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2045C3F73F for ; Tue, 8 Oct 2024 12:31:13 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 07/10] insane: tidy up objdump preloading in package_qa_walk() Date: Tue, 8 Oct 2024 20:31:00 +0100 Message-Id: <20241008193103.2429829-7-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241008193103.2429829-1-ross.burton@arm.com> References: <20241008193103.2429829-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 19:31:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205307 Move the prepopulate function out of global scope, and access the dictionary once instead of repeatedly. This still results in each ELF being opened twice, but this avoids opening all of the files at once and the ELFFile.open() call is fairly fast. Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index 401807b2be3..b165f111cec 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -763,10 +763,6 @@ def qa_check_staged(path,d): if not skip_shebang_size: package_qa_check_shebang_size(path, "", d, None) -def prepopulate_objdump_p(elf, d): - output = elf.run_objdump("-p", d) - return (elf.name, output) - # Walk over all files in a directory and call func def package_qa_walk(checkfuncs, package, d): elves = {} @@ -782,17 +778,22 @@ def package_qa_walk(checkfuncs, package, d): if elf: elves[path] = elf + def prepopulate_objdump_p(elf, d): + output = elf.run_objdump("-p", d) + return (elf.name, output) + results = oe.utils.multiprocess_launch(prepopulate_objdump_p, elves.values(), d, extraargs=(d,)) for item in results: elves[item[0]].set_objdump("-p", item[1]) for path in pkgfiles[package]: - if path in elves: - elves[path].open() - for func in checkfuncs: - func(path, package, d, elves.get(path)) - if path in elves: - elves[path].close() + elf = elves.get(path) + if elf: + elf.open() + for func in checkfuncs: + func(path, package, d, elf) + if elf: + elf.close() def package_qa_check_rdepends(pkg, pkgdest, skip, taskdeps, packages, d): # Don't do this check for kernel/module recipes, there aren't too many debug/development From patchwork Tue Oct 8 19:31:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50066 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30A10CEFC51 for ; Tue, 8 Oct 2024 19:31:24 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.4701.1728415875840450543 for ; Tue, 08 Oct 2024 12:31:15 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1D3B5FEC for ; Tue, 8 Oct 2024 12:31:45 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 082B83F73F for ; Tue, 8 Oct 2024 12:31:14 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 08/10] insane: micro-optimise the sweep of pkgfiles Date: Tue, 8 Oct 2024 20:31:01 +0100 Message-Id: <20241008193103.2429829-8-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241008193103.2429829-1-ross.burton@arm.com> References: <20241008193103.2429829-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 19:31:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205308 Don't actively do more work: - Exit early if there are no packages being generated - Don't iterate repeatedly when removing CONTROL and DEBIAN - Extend a list with another list instead of appending item by item - Remove unused variables Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index b165f111cec..686040f7a48 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -1084,13 +1084,8 @@ parse_test_matrix[vardepsexclude] = "ERROR_QA WARN_QA" # The PACKAGE FUNC to scan each package python do_package_qa () { - import subprocess import oe.packagedata - bb.note("DO PACKAGE QA") - - main_lic = d.getVar('LICENSE') - # Check for obsolete license references in main LICENSE (packages are checked below for any changes) main_licenses = oe.license.list_licenses(d.getVar('LICENSE')) obsolete = set(oe.license.obsolete_license_list()) & main_licenses @@ -1106,27 +1101,27 @@ python do_package_qa () { pn = d.getVar('PN') # Scan the packages... - pkgdest = d.getVar('PKGDEST') packages = set((d.getVar('PACKAGES') or '').split()) + # no packages should be scanned + if not packages: + return global pkgfiles pkgfiles = {} + pkgdest = d.getVar('PKGDEST') for pkg in packages: - pkgfiles[pkg] = [] pkgdir = os.path.join(pkgdest, pkg) + pkgfiles[pkg] = [] for walkroot, dirs, files in os.walk(pkgdir): # Don't walk into top-level CONTROL or DEBIAN directories as these # are temporary directories created by do_package. if walkroot == pkgdir: - for control in ("CONTROL", "DEBIAN"): - if control in dirs: - dirs.remove(control) - for file in files: - pkgfiles[pkg].append(os.path.join(walkroot, file)) - - # no packages should be scanned - if not packages: - return + try: + dirs.remove("CONTROL") + dirs.remove("DEBIAN") + except ValueError: + pass + pkgfiles[pkg].extend((os.path.join(walkroot, f) for f in files)) import re # The package name matches the [a-z0-9.+-]+ regular expression From patchwork Tue Oct 8 19:31:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50063 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2514FCEFC4D for ; Tue, 8 Oct 2024 19:31:24 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.4657.1728415876757632556 for ; Tue, 08 Oct 2024 12:31:16 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0D03ADA7 for ; Tue, 8 Oct 2024 12:31:46 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E234D3F73F for ; Tue, 8 Oct 2024 12:31:15 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 09/10] insane: optimise test in package_qa_check_shebang_size Date: Tue, 8 Oct 2024 20:31:02 +0100 Message-Id: <20241008193103.2429829-9-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241008193103.2429829-1-ross.burton@arm.com> References: <20241008193103.2429829-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 19:31:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205309 Check whether the elf is not None first, before doing os.stat() calls on disk. Also don't check anything that isn't a file, not just FIFOs. Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index 686040f7a48..be358f49993 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -82,8 +82,7 @@ def package_qa_clean_path(path, d, pkg=None): QAPATHTEST[shebang-size] = "package_qa_check_shebang_size" def package_qa_check_shebang_size(path, name, d, elf): - import stat - if os.path.islink(path) or stat.S_ISFIFO(os.stat(path).st_mode) or elf: + if elf or os.path.islink(path) or not os.path.isfile(path): return try: From patchwork Tue Oct 8 19:31:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 50067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24A44CEFC4C for ; Tue, 8 Oct 2024 19:31:24 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.4658.1728415877663184111 for ; Tue, 08 Oct 2024 12:31:17 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EDFEFDA7 for ; Tue, 8 Oct 2024 12:31:46 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D155D3F73F for ; Tue, 8 Oct 2024 12:31:16 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 10/10] insane: use oe.cachedpath.CachedPath instead of os.path Date: Tue, 8 Oct 2024 20:31:03 +0100 Message-Id: <20241008193103.2429829-10-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241008193103.2429829-1-ross.burton@arm.com> References: <20241008193103.2429829-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 08 Oct 2024 19:31:24 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/205310 The insane QAPATHTESTs make many os.stat() calls, the majority of which are redundant with caching as the initial sweep does a stat() on every entry to determine if it is a file or a directory, and from then on each test that does further stat()s is redundant as the tree doesn't change. Switch os.stat() and friends (os.path.isfile(), etc) to use a common oe.cachedpath.CachedPath() instance that is shared between all of the functions, meaning only one stat is done. In my test case of ltp:do_package_qa, this reduces the time taken from 44s to 37s. Signed-off-by: Ross Burton --- meta/classes-global/insane.bbclass | 42 ++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 14 deletions(-) diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass index be358f49993..2488be29f5d 100644 --- a/meta/classes-global/insane.bbclass +++ b/meta/classes-global/insane.bbclass @@ -82,7 +82,9 @@ def package_qa_clean_path(path, d, pkg=None): QAPATHTEST[shebang-size] = "package_qa_check_shebang_size" def package_qa_check_shebang_size(path, name, d, elf): - if elf or os.path.islink(path) or not os.path.isfile(path): + global cpath + + if elf or cpath.islink(path) or not cpath.isfile(path): return try: @@ -167,8 +169,8 @@ def package_qa_check_dev(path, name, d, elf): """ Check for ".so" library symlinks in non-dev packages """ - - if not name.endswith("-dev") and not name.endswith("-dbg") and not name.endswith("-ptest") and not name.startswith("nativesdk-") and path.endswith(".so") and os.path.islink(path): + global cpath + if not name.endswith("-dev") and not name.endswith("-dbg") and not name.endswith("-ptest") and not name.startswith("nativesdk-") and path.endswith(".so") and cpath.islink(path): oe.qa.handle_error("dev-so", "non -dev/-dbg/nativesdk- package %s contains symlink .so '%s'" % \ (name, package_qa_clean_path(path, d, name)), d) @@ -179,7 +181,8 @@ def package_qa_check_dev_elf(path, name, d, elf): check that the file is not a link and is an ELF object as some recipes install link-time .so files that are linker scripts. """ - if name.endswith("-dev") and path.endswith(".so") and not os.path.islink(path) and elf: + global cpath + if name.endswith("-dev") and path.endswith(".so") and not cpath.islink(path) and elf: oe.qa.handle_error("dev-elf", "-dev package %s contains non-symlink .so '%s'" % \ (name, package_qa_clean_path(path, d, name)), d) @@ -422,9 +425,9 @@ def package_qa_check_buildpaths(path, name, d, elf): explicitly ignored. """ import stat - + global cpath # Ignore symlinks/devs/fifos - mode = os.lstat(path).st_mode + mode = cpath.lstat(path).st_mode if stat.S_ISLNK(mode) or stat.S_ISBLK(mode) or stat.S_ISFIFO(mode) or stat.S_ISCHR(mode) or stat.S_ISSOCK(mode): return @@ -469,7 +472,8 @@ def package_qa_check_symlink_to_sysroot(path, name, d, elf): """ Check that the package doesn't contain any absolute symlinks to the sysroot. """ - if os.path.islink(path): + global cpath + if cpath.islink(path): target = os.readlink(path) if os.path.isabs(target): tmpdir = d.getVar('TMPDIR') @@ -760,14 +764,19 @@ def qa_check_staged(path,d): oe.qa.handle_error("pkgconfig", error_msg, d) if not skip_shebang_size: + global cpath + cpath = oe.cachedpath.CachedPath() package_qa_check_shebang_size(path, "", d, None) + cpath = None # Walk over all files in a directory and call func def package_qa_walk(checkfuncs, package, d): + global cpath + elves = {} for path in pkgfiles[package]: elf = None - if os.path.isfile(path) and not os.path.islink(path): + if cpath.isfile(path) and not cpath.islink(path): elf = oe.qa.ELFFile(path) try: elf.open() @@ -915,11 +924,12 @@ def package_qa_check_deps(pkg, pkgdest, d): QAPKGTEST[usrmerge] = "package_qa_check_usrmerge" def package_qa_check_usrmerge(pkg, d): + global cpath pkgdest = d.getVar('PKGDEST') pkg_dir = pkgdest + os.sep + pkg + os.sep merged_dirs = ['bin', 'sbin', 'lib'] + d.getVar('MULTILIB_VARIANTS').split() for f in merged_dirs: - if os.path.exists(pkg_dir + f) and not os.path.islink(pkg_dir + f): + if cpath.exists(pkg_dir + f) and not cpath.islink(pkg_dir + f): msg = "%s package is not obeying usrmerge distro feature. /%s should be relocated to /usr." % (pkg, f) oe.qa.handle_error("usrmerge", msg, d) return @@ -985,10 +995,11 @@ def package_qa_check_empty_dirs(pkg, d): empty. """ + global cpath pkgd = oe.path.join(d.getVar('PKGDEST'), pkg) for dir in (d.getVar('QA_EMPTY_DIRS') or "").split(): empty_dir = oe.path.join(pkgd, dir) - if os.path.exists(empty_dir) and os.listdir(empty_dir): + if cpath.exists(empty_dir) and os.listdir(empty_dir): recommendation = (d.getVar('QA_EMPTY_DIRS_RECOMMENDATION:' + dir) or "but it is expected to be empty") msg = "%s installs files in %s, %s" % (pkg, dir, recommendation) @@ -1018,8 +1029,9 @@ HOST_USER_GID := "${@os.getgid()}" QAPATHTEST[host-user-contaminated] = "package_qa_check_host_user" def package_qa_check_host_user(path, name, d, elf): """Check for paths outside of /home which are owned by the user running bitbake.""" + global cpath - if not os.path.lexists(path): + if not cpath.lexists(path): return dest = d.getVar('PKGDEST') @@ -1029,7 +1041,7 @@ def package_qa_check_host_user(path, name, d, elf): return try: - stat = os.lstat(path) + stat = cpath.lstat(path) except OSError as exc: import errno if exc.errno != errno.ENOENT: @@ -1105,13 +1117,14 @@ python do_package_qa () { if not packages: return - global pkgfiles + global pkgfiles, cpath pkgfiles = {} + cpath = oe.cachedpath.CachedPath() pkgdest = d.getVar('PKGDEST') for pkg in packages: pkgdir = os.path.join(pkgdest, pkg) pkgfiles[pkg] = [] - for walkroot, dirs, files in os.walk(pkgdir): + for walkroot, dirs, files in cpath.walk(pkgdir): # Don't walk into top-level CONTROL or DEBIAN directories as these # are temporary directories created by do_package. if walkroot == pkgdir: @@ -1159,6 +1172,7 @@ python do_package_qa () { package_qa_check_libdir(d) + cpath = None oe.qa.exit_if_errors(d) }