From patchwork Thu Jul 18 14:48:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 46616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3385FC3DA63 for ; Thu, 18 Jul 2024 14:48:42 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.16938.1721314115886587236 for ; Thu, 18 Jul 2024 07:48:36 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CA95E1063 for ; Thu, 18 Jul 2024 07:49:00 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 284163F762 for ; Thu, 18 Jul 2024 07:48:35 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 1/3] openssl: disable tests unless ptest is enabled Date: Thu, 18 Jul 2024 15:48:27 +0100 Message-Id: <20240718144829.2504384-1-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 18 Jul 2024 14:48:42 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/202226 The upstream Makefile always builds the tests unless they're explicitly disabled. Whilst this doesn't make a difference to the final package and sysroot output, disabling the tests for openssl-native reduces the size of the build tree from 659M to 78M and reduces the CPU time used by 30%. Signed-off-by: Ross Burton --- meta/recipes-connectivity/openssl/openssl_3.3.1.bb | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/meta/recipes-connectivity/openssl/openssl_3.3.1.bb b/meta/recipes-connectivity/openssl/openssl_3.3.1.bb index 0ad84951e65..f8f22541534 100644 --- a/meta/recipes-connectivity/openssl/openssl_3.3.1.bb +++ b/meta/recipes-connectivity/openssl/openssl_3.3.1.bb @@ -35,6 +35,8 @@ PACKAGECONFIG[manpages] = "" B = "${WORKDIR}/build" do_configure[cleandirs] = "${B}" +EXTRA_OECONF = "${@bb.utils.contains('PTEST_ENABLED', '1', '', 'no-tests', d)}" + #| ./libcrypto.so: undefined reference to `getcontext' #| ./libcrypto.so: undefined reference to `setcontext' #| ./libcrypto.so: undefined reference to `makecontext' @@ -43,8 +45,8 @@ EXTRA_OECONF:append:libc-musl:powerpc64 = " no-asm" # adding devrandom prevents openssl from using getrandom() which is not available on older glibc versions # (native versions can be built with newer glibc, but then relocated onto a system with older glibc) -EXTRA_OECONF:class-native = "--with-rand-seed=os,devrandom" -EXTRA_OECONF:class-nativesdk = "--with-rand-seed=os,devrandom" +EXTRA_OECONF:append:class-native = " --with-rand-seed=os,devrandom" +EXTRA_OECONF:append:class-nativesdk = " --with-rand-seed=os,devrandom" # Relying on hardcoded built-in paths causes openssl-native to not be relocateable from sstate. CFLAGS:append:class-native = " -DOPENSSLDIR=/not/builtin -DENGINESDIR=/not/builtin" From patchwork Thu Jul 18 14:48:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 46617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3449EC41513 for ; Thu, 18 Jul 2024 14:48:42 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.16871.1721314116384619052 for ; Thu, 18 Jul 2024 07:48:36 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5950E106F for ; Thu, 18 Jul 2024 07:49:01 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AC3E03F762 for ; Thu, 18 Jul 2024 07:48:35 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 2/3] openssl: strip the test suite Date: Thu, 18 Jul 2024 15:48:28 +0100 Message-Id: <20240718144829.2504384-2-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240718144829.2504384-1-ross.burton@arm.com> References: <20240718144829.2504384-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 18 Jul 2024 14:48:42 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/202227 The test suite is huge because every test binary is statically linked to libssl and/or libcrypto. This bloats the size of the -dbg package hugely, so strip the test suite before packaging. This reduces the size of openssl-dbg by 90% from ~1.2GB to ~120MB, and reduces the size of the build tree from ~1.9GB to ~800MB. Signed-off-by: Ross Burton --- meta/recipes-connectivity/openssl/openssl_3.3.1.bb | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/meta/recipes-connectivity/openssl/openssl_3.3.1.bb b/meta/recipes-connectivity/openssl/openssl_3.3.1.bb index f8f22541534..36d2f87c265 100644 --- a/meta/recipes-connectivity/openssl/openssl_3.3.1.bb +++ b/meta/recipes-connectivity/openssl/openssl_3.3.1.bb @@ -146,6 +146,13 @@ do_configure () { perl ${B}/configdata.pm --dump } +do_compile:append () { + # The test suite binaries are large and we don't need the debugging in them + if test -d ${B}/test; then + find ${B}/test -type f -executable -exec ${STRIP} {} \; + fi +} + do_install () { oe_runmake DESTDIR="${D}" MANDIR="${mandir}" MANSUFFIX=ssl install_sw install_ssldirs ${@bb.utils.contains('PACKAGECONFIG', 'manpages', 'install_docs', '', d)} @@ -252,6 +259,9 @@ RDEPENDS:${PN}-ptest += "openssl-bin perl perl-modules bash sed" RDEPENDS:${PN}-bin += "openssl-conf" +# The test suite is installed stripped +INSANE_SKIP:${PN} = "already-stripped" + BBCLASSEXTEND = "native nativesdk" CVE_PRODUCT = "openssl:openssl" From patchwork Thu Jul 18 14:48:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 46618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34E0FC3DA6F for ; Thu, 18 Jul 2024 14:48:42 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web11.16939.1721314116825043107 for ; Thu, 18 Jul 2024 07:48:36 -0700 Authentication-Results: mx.groups.io; dkim=none (message not signed); spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ross.burton@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DD4F91063 for ; Thu, 18 Jul 2024 07:49:01 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-04.oss.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3C1773F762 for ; Thu, 18 Jul 2024 07:48:36 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Subject: [PATCH 3/3] openssl: rewrite ptest installation Date: Thu, 18 Jul 2024 15:48:29 +0100 Message-Id: <20240718144829.2504384-3-ross.burton@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240718144829.2504384-1-ross.burton@arm.com> References: <20240718144829.2504384-1-ross.burton@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 18 Jul 2024 14:48:42 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/202228 Rewrite (again) the openssl test suite installation. Depend on and reuse already installed libraries and modules instead of installing them twice. Be more selective when installing from the build tree so we don't install intermediate .c .d .o files. This further reduces the size of openssl-dbg from ~120MB to ~18MB. Signed-off-by: Ross Burton --- .../openssl/openssl/run-ptest | 19 ++++-- .../openssl/openssl_3.3.1.bb | 67 +++++++++---------- 2 files changed, 46 insertions(+), 40 deletions(-) diff --git a/meta/recipes-connectivity/openssl/openssl/run-ptest b/meta/recipes-connectivity/openssl/openssl/run-ptest index c89ec5afa1b..cd29bb14461 100644 --- a/meta/recipes-connectivity/openssl/openssl/run-ptest +++ b/meta/recipes-connectivity/openssl/openssl/run-ptest @@ -1,12 +1,19 @@ #!/bin/sh -set -e +set -eu -# Optional arguments are 'list' to lists all tests, or the test name (base name -# ie test_evp, not 03_test_evp.t). +# Optional arguments are 'list' to lists the tests, or the test name (base name +# ie test_evp, not 03_test_evp.t). Without any arguments we run all tests. + +if test $# -gt 0; then + TESTS=$* +else + # Skip test_symbol_presence as this is for developers + TESTS="alltests -test_symbol_presence" +fi export TOP=. -# OPENSSL_ENGINES is relative from the test binaries -export OPENSSL_ENGINES=../engines +# Run four jobs in parallel +export HARNESS_JOBS=4 -{ HARNESS_JOBS=4 perl ./test/run_tests.pl $* || echo "FAIL: openssl" ; } | sed -u -r -e '/(.*) \.*.ok/ s/^/PASS: /g' -r -e '/Dubious(.*)/ s/^/FAIL: /g' -e '/(.*) \.*.skipped: (.*)/ s/^/SKIP: /g' +{ perl ./test/run_tests.pl $TESTS || echo "FAIL: openssl" ; } | sed -u -r -e '/(.*) \.*.ok/ s/^/PASS: /g' -r -e '/Dubious(.*)/ s/^/FAIL: /g' -e '/(.*) \.*.skipped: (.*)/ s/^/SKIP: /g' diff --git a/meta/recipes-connectivity/openssl/openssl_3.3.1.bb b/meta/recipes-connectivity/openssl/openssl_3.3.1.bb index 36d2f87c265..3bc0153429d 100644 --- a/meta/recipes-connectivity/openssl/openssl_3.3.1.bb +++ b/meta/recipes-connectivity/openssl/openssl_3.3.1.bb @@ -191,44 +191,43 @@ do_install:append:class-nativesdk () { PTEST_BUILD_HOST_FILES += "configdata.pm" PTEST_BUILD_HOST_PATTERN = "perl_version =" -do_install_ptest () { - install -d ${D}${PTEST_PATH}/test - install -m755 ${B}/test/p_test.so ${D}${PTEST_PATH}/test - install -m755 ${B}/test/p_minimal.so ${D}${PTEST_PATH}/test - install -m755 ${B}/test/provider_internal_test.cnf ${D}${PTEST_PATH}/test - - # Prune the build tree - rm -f ${B}/fuzz/*.* ${B}/test/*.* - - cp ${S}/Configure ${B}/configdata.pm ${D}${PTEST_PATH} - sed 's|${S}|${PTEST_PATH}|g' -i ${D}${PTEST_PATH}/configdata.pm - cp -r ${S}/external ${B}/test ${S}/test ${B}/fuzz ${S}/util ${B}/util ${D}${PTEST_PATH} - - # For test_shlibload - ln -s ${libdir}/libcrypto.so.1.1 ${D}${PTEST_PATH}/ - ln -s ${libdir}/libssl.so.1.1 ${D}${PTEST_PATH}/ +do_install_ptest() { + install -m644 ${S}/Configure ${B}/configdata.pm ${D}${PTEST_PATH} + cp -rf ${S}/Configurations ${S}/external ${D}${PTEST_PATH}/ install -d ${D}${PTEST_PATH}/apps ln -s ${bindir}/openssl ${D}${PTEST_PATH}/apps - install -m644 ${S}/apps/*.pem ${S}/apps/*.srl ${S}/apps/openssl.cnf ${D}${PTEST_PATH}/apps - install -m755 ${B}/apps/CA.pl ${D}${PTEST_PATH}/apps - - install -d ${D}${PTEST_PATH}/engines - install -m755 ${B}/engines/dasync.so ${D}${PTEST_PATH}/engines - install -m755 ${B}/engines/loader_attic.so ${D}${PTEST_PATH}/engines - install -m755 ${B}/engines/ossltest.so ${D}${PTEST_PATH}/engines - - install -d ${D}${PTEST_PATH}/providers - install -m755 ${B}/providers/legacy.so ${D}${PTEST_PATH}/providers - install -d ${D}${PTEST_PATH}/Configurations - cp -rf ${S}/Configurations/* ${D}${PTEST_PATH}/Configurations/ + cd ${S} + find test/certs test/ct test/d2i-tests test/recipes test/ocsp-tests test/ssl-tests test/smime-certs -type f -exec install -m644 -D {} ${D}${PTEST_PATH}/{} \; + find apps test -name \*.cnf -exec install -m644 -D {} ${D}${PTEST_PATH}/{} \; + find apps test -name \*.der -exec install -m644 -D {} ${D}${PTEST_PATH}/{} \; + find apps test -name \*.pem -exec install -m644 -D {} ${D}${PTEST_PATH}/{} \; + find util -name \*.p[lm] -exec install -m644 -D {} ${D}${PTEST_PATH}/{} \; + + cd ${B} + # Everything but .? (.o and .d) + find test -type f -name \*[^.]? -exec install -m755 -D {} ${D}${PTEST_PATH}/{} \; + find apps test -name \*.cnf -exec install -m644 -D {} ${D}${PTEST_PATH}/{} \; + find apps test -name \*.pem -exec install -m644 -D {} ${D}${PTEST_PATH}/{} \; + find apps test -name \*.srl -exec install -m644 -D {} ${D}${PTEST_PATH}/{} \; + install -m755 ${B}/util/*wrap.* ${D}${PTEST_PATH}/util/ + + install -m755 ${B}/apps/CA.pl ${D}${PTEST_PATH}/apps/ + install -m755 ${S}/test/*.pl ${D}${PTEST_PATH}/test/ + install -m755 ${S}/test/shibboleth.pfx ${D}${PTEST_PATH}/test/ + install -m755 ${S}/test/*.bin ${D}${PTEST_PATH}/test/ + install -m755 ${S}/test/dane*.in ${D}${PTEST_PATH}/test/ + install -m755 ${S}/test/smcont*.txt ${D}${PTEST_PATH}/test/ + install -m755 ${S}/test/ssl_test.tmpl ${D}${PTEST_PATH}/test/ + + sed 's|${S}|${PTEST_PATH}|g' -i ${D}${PTEST_PATH}/configdata.pm ${D}${PTEST_PATH}/util/wrap.pl - # seems to be needed with perl 5.32.1 - install -d ${D}${PTEST_PATH}/util/perl/recipes - cp ${D}${PTEST_PATH}/test/recipes/tconversion.pl ${D}${PTEST_PATH}/util/perl/recipes/ - - sed 's|${S}|${PTEST_PATH}|g' -i ${D}${PTEST_PATH}/util/wrap.pl + install -d ${D}${PTEST_PATH}/engines + install -m755 ${B}/engines/dasync.so ${D}${PTEST_PATH}/engines/ + install -m755 ${B}/engines/ossltest.so ${D}${PTEST_PATH}/engines/ + ln -s ${libdir}/engines-3/loader_attic.so ${D}${PTEST_PATH}/engines/ + ln -s ${libdir}/ossl-modules/ ${D}${PTEST_PATH}/providers } # Add the openssl.cnf file to the openssl-conf package. Make the libcrypto @@ -255,7 +254,7 @@ CONFFILES:openssl-conf = "${sysconfdir}/ssl/openssl.cnf" RRECOMMENDS:libcrypto += "openssl-conf ${PN}-ossl-module-legacy" RDEPENDS:${PN}-misc = "perl" -RDEPENDS:${PN}-ptest += "openssl-bin perl perl-modules bash sed" +RDEPENDS:${PN}-ptest += "openssl-bin perl perl-modules bash sed openssl-engines openssl-ossl-module-legacy" RDEPENDS:${PN}-bin += "openssl-conf"