diff mbox series

[meta-oe,6/7] imx-cst: fix remaining build failures with OpenSSL 4

Message ID 20260911141756.2275517-6-khem.raj@oss.qualcomm.com
State New
Headers show
Series [meta-oe,1/7] bit7z: Fix build with clang | expand

Commit Message

Khem Raj Sept. 11, 2026, 2:17 p.m. UTC 
Two more OpenSSL 4.0 breakages on top of the ASN.1 opacity already
handled by 0012-fix-openssl-4-asn1-opaque.patch.

The ENGINE API is gone. <openssl/engine.h> is still shipped as a
source-compatibility stub, so linking fails on ENGINE_free,
ENGINE_load_builtin_engines, ENGINE_by_id, ENGINE_init, ENGINE_finish,
ENGINE_ctrl_cmd and ENGINE_load_private_key. Define OPENSSL_ENGINE_STUBS,
which OpenSSL 4 provides for exactly this case: the declarations become
inline no-ops returning failure, so the file compiles and links unchanged.
OPENSSL_SUPPRESS_DEPRECATED is already defined there, so the stubs'
deprecation attributes do not trip -Werror. That leaves ENGINE_by_id()
returning NULL, which the existing code fed straight into ENGINE_init() -
a NULL dereference predating OpenSSL 4 - so check it and report what went
wrong, mentioning OpenSSL 4 so the failure is not mistaken for a missing
module. PKCS#11 backed signing genuinely is unavailable there, as no
provider exposes an equivalent of the pkcs11 engine's LOAD_CERT_CTRL.

X509_get_subject_name() now returns const X509_NAME *, so its result can
no longer be the destination of X509_NAME_add_entry_by_txt():

  src/tools/pki_tree/pki_helper.c:440:10: error: assigning to 'X509_NAME *' from 'const X509_NAME *' discards qualifiers

There is no mutable counterpart, so build the subject name standalone and
install it with X509_set_subject_name(); both setters copy it, so it is
freed once the issuer name has been set from it too.

AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <raj.khem@gmail.com>
---
 ...015-fix-openssl-4-engine-api-removal.patch | 67 +++++++++++++++++++
 ...016-fix-openssl-4-const-subject-name.patch | 48 +++++++++++++
 .../recipes-support/imx-cst/imx-cst_4.0.1.bb  |  2 +
 3 files changed, 117 insertions(+)
 create mode 100644 meta-oe/recipes-support/imx-cst/imx-cst/0015-fix-openssl-4-engine-api-removal.patch
 create mode 100644 meta-oe/recipes-support/imx-cst/imx-cst/0016-fix-openssl-4-const-subject-name.patch
diff mbox series

Patch

diff --git a/meta-oe/recipes-support/imx-cst/imx-cst/0015-fix-openssl-4-engine-api-removal.patch b/meta-oe/recipes-support/imx-cst/imx-cst/0015-fix-openssl-4-engine-api-removal.patch
new file mode 100644
index 0000000000..d4a3f0ff2e
--- /dev/null
+++ b/meta-oe/recipes-support/imx-cst/imx-cst/0015-fix-openssl-4-engine-api-removal.patch
@@ -0,0 +1,67 @@ 
+From: Khem Raj <raj.khem@gmail.com>
+Date: Wed, 3 Sep 2026 01:30:00 +0000
+Subject: Fix FTBFS with OpenSSL 4.0: ENGINE API removal
+
+OpenSSL 4.0 removed the ENGINE API. <openssl/engine.h> is still shipped,
+but only as a source-compatibility stub, so linking fails:
+
+  ld.lld: error: undefined symbol: ENGINE_free
+  ld.lld: error: undefined symbol: ENGINE_load_builtin_engines
+  ld.lld: error: undefined symbol: ENGINE_by_id
+  ld.lld: error: undefined symbol: ENGINE_init
+  ld.lld: error: undefined symbol: ENGINE_finish
+  ld.lld: error: undefined symbol: ENGINE_ctrl_cmd
+  ld.lld: error: undefined symbol: ENGINE_load_private_key
+
+Define OPENSSL_ENGINE_STUBS, which OpenSSL 4 offers exactly for this
+case: the ENGINE_* declarations become inline no-ops returning failure,
+so the file compiles and links unchanged. OPENSSL_SUPPRESS_DEPRECATED is
+already defined here, so the stubs' deprecation attributes do not trip
+-Werror either.
+
+That leaves ENGINE_by_id() returning NULL at runtime, which the existing
+code fed straight into ENGINE_init() - a NULL dereference that predates
+OpenSSL 4. Check the result and print what went wrong instead, noting the
+OpenSSL 4 situation so the failure is not mistaken for a missing module.
+
+Providers replace engines upstream, but nothing exposes an equivalent of
+the pkcs11 engine's LOAD_CERT_CTRL command yet, so PKCS#11 backed signing
+genuinely is unavailable with OpenSSL 4.
+
+Upstream-Status: Pending
+
+Signed-off-by: Khem Raj <raj.khem@gmail.com>
+---
+--- a/src/lib/back_end/engine.c
++++ b/src/lib/back_end/engine.c
+@@ -4,6 +4,13 @@
+  */
+
+ #define OPENSSL_SUPPRESS_DEPRECATED
++/*
++ * OpenSSL 4.0 removed the ENGINE API.  <openssl/engine.h> still declares
++ * it for source compatibility, and defining OPENSSL_ENGINE_STUBS turns the
++ * declarations into inline no-ops so this file keeps linking.  The pkcs11
++ * engine simply does not exist there, which engine_ctx_init() reports.
++ */
++#define OPENSSL_ENGINE_STUBS
+
+ #include "engine.h"
+ #include "err.h"
+@@ -63,6 +70,16 @@
+
+ 	ctx->engine = ENGINE_by_id("pkcs11");
+
++	if (!ctx->engine) {
++		fprintf(stderr, "ERROR: cannot load the pkcs11 OpenSSL engine\n");
++#if OPENSSL_VERSION_MAJOR >= 4
++		fprintf(stderr,
++			"ERROR: OpenSSL 4.0 removed ENGINE support, so PKCS#11 "
++			"backed signing is unavailable\n");
++#endif
++		return 0;
++	}
++
+ #ifdef DEBUG
+ 	ENGINE_ctrl_cmd_string(ctx->engine, "VERBOSE", NULL, 0);
+ #endif
diff --git a/meta-oe/recipes-support/imx-cst/imx-cst/0016-fix-openssl-4-const-subject-name.patch b/meta-oe/recipes-support/imx-cst/imx-cst/0016-fix-openssl-4-const-subject-name.patch
new file mode 100644
index 0000000000..3cea584983
--- /dev/null
+++ b/meta-oe/recipes-support/imx-cst/imx-cst/0016-fix-openssl-4-const-subject-name.patch
@@ -0,0 +1,48 @@ 
+From: Khem Raj <raj.khem@gmail.com>
+Date: Wed, 3 Sep 2026 01:30:00 +0000
+Subject: Fix FTBFS with OpenSSL 4.0: const X509_get_subject_name()
+
+OpenSSL 4.0 changed X509_get_subject_name() to return a const X509_NAME *,
+so using its result as the destination of X509_NAME_add_entry_by_txt() no
+longer compiles:
+
+  src/tools/pki_tree/pki_helper.c:440:10: error: assigning to 'X509_NAME *'
+    from 'const X509_NAME *' discards qualifiers
+    [-Werror,-Wincompatible-pointer-types-discards-qualifiers]
+
+There is no mutable counterpart to the getter (unlike X509_getm_notAfter),
+so build the subject name as a standalone X509_NAME and install it with
+X509_set_subject_name(). Both setters copy the name, so it is freed once
+the issuer name has been set from it too.
+
+Upstream-Status: Pending
+
+Signed-off-by: Khem Raj <raj.khem@gmail.com>
+---
+--- a/src/tools/pki_tree/pki_helper.c
++++ b/src/tools/pki_tree/pki_helper.c
+@@ -437,11 +437,15 @@
+     if (X509_set_pubkey(x509, pkey) != 1)
+         handle_errors();
+
+-    name = X509_get_subject_name(x509);
++    name = X509_NAME_new();
++    if (!name)
++        handle_errors();
+     if (X509_NAME_add_entry_by_txt(name, "CN", MBSTRING_ASC,
+                                    (const unsigned char *) subj, -1, -1,
+                                    0) != 1)
+         handle_errors();
++    if (X509_set_subject_name(x509, name) != 1)
++        handle_errors();
+     if (sign_cert)
+     {
+         if (X509_set_issuer_name(x509, X509_get_subject_name(sign_cert)) != 1)
+@@ -452,6 +456,7 @@
+         if (X509_set_issuer_name(x509, name) != 1)
+             handle_errors();
+     }
++    X509_NAME_free(name);
+
+     if (is_ca)
+     {
diff --git a/meta-oe/recipes-support/imx-cst/imx-cst_4.0.1.bb b/meta-oe/recipes-support/imx-cst/imx-cst_4.0.1.bb
index 03dfdbd6f7..ec6838e95d 100644
--- a/meta-oe/recipes-support/imx-cst/imx-cst_4.0.1.bb
+++ b/meta-oe/recipes-support/imx-cst/imx-cst_4.0.1.bb
@@ -27,6 +27,8 @@  SRC_URI = "\
     file://0012-fix-openssl-4-asn1-opaque.patch \
     file://0013-convlb-remove-redundant-NULL-definition.patch \
     file://0014-fix-pointer-sign-errors-with-clang.patch \
+    file://0015-fix-openssl-4-engine-api-removal.patch \
+    file://0016-fix-openssl-4-const-subject-name.patch \
 "
 SRC_URI[sha256sum] = "fd92a1a9faa10fb81bbf752c7ee1e257f17e1ec4c2964f8a47adf8a3eaa7df41"