package_rpm: allow arch overriding

From: Sergei Zhmylev <s.zhmylev@yadro.com>

Currently arch is being calculated from either MACHINE_ARCH or
TUNE_PKGARCH.  Some recipes do override PACKAGE_ARCH intentionally.
This commit makes possible to override ARCH for RPM packages
separately in order to simplify common rpm naming conformance.

Signed-off-by: Sergei Zhmylev <s.zhmylev@yadro.com>
---
 meta/classes-global/package_rpm.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

On Fri, 2023-01-13 at 00:33 +0300, Sergey Zhmylev wrote:
> From: Sergei Zhmylev <s.zhmylev@yadro.com>
> 
> Currently arch is being calculated from either MACHINE_ARCH or
> TUNE_PKGARCH.  Some recipes do override PACKAGE_ARCH intentionally.
> This commit makes possible to override ARCH for RPM packages
> separately in order to simplify common rpm naming conformance.
> 
> Signed-off-by: Sergei Zhmylev <s.zhmylev@yadro.com>
> ---
>  meta/classes-global/package_rpm.bbclass | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/meta/classes-global/package_rpm.bbclass b/meta/classes-global/package_rpm.bbclass
> index 85d0bd7fce..3fb7466778 100644
> --- a/meta/classes-global/package_rpm.bbclass
> +++ b/meta/classes-global/package_rpm.bbclass
> @@ -678,6 +678,10 @@ python do_package_rpm () {
>      bb.utils.mkdirhier(pkgwritedir)
>      os.chmod(pkgwritedir, 0o755)
>  
> +    rpm_arch_override = d.getVar("RPM_ARCH_OVERRIDE")
> +    if package_arch != "noarch" and rpm_arch_override:
> +        pkgarch = rpm_arch_override
> +
>      cmd = rpmbuild
>      cmd = cmd + " --noclean --nodeps --short-circuit --target " + pkgarch + " --buildroot " + pkgd
>      cmd = cmd + " --define '_topdir " + workdir + "' --define '_rpmdir " + pkgwritedir + "'"

This makes me very nervous. PACKAGE_ARCH can be overridden and I know
of BSPs which inject a whole new level of architecture into the system
but they don't need to make rpm backend specific changes to make it
work.

If we were to add such a thing we'd need a much clearer picture of when
it should be used and why it is needed. Wouldn't all package backends
need to adjust this?

Cheers,

Richard

Exactly! Even more: when I build for "genericx86_64", I get mix of
packages with architecture one of: "x86_64", "genericx86_64",
"core2_64", and "noarch". Our audit team highlited this as this is not
compliant with common RPM naming convention.

In that case override architecture string only for packaging would be,
from my perspective, the best workaround. 

At the same time I doubt we have to change other package backends as
they're less strict while architecture overriding effectively looses
(probably) important information.

-- 
With best wishes,
Sergei Zhmylev
Engineering consultant
OS development department




On Thu, 2023-01-12 at 22:32 +0000, Richard Purdie wrote:
> «Внимание! Данное письмо от внешнего адресата!»
> 
> On Fri, 2023-01-13 at 00:33 +0300, Sergey Zhmylev wrote:
> > From: Sergei Zhmylev <s.zhmylev@yadro.com>
> > 
> > Currently arch is being calculated from either MACHINE_ARCH or
> > TUNE_PKGARCH.  Some recipes do override PACKAGE_ARCH intentionally.
> > This commit makes possible to override ARCH for RPM packages
> > separately in order to simplify common rpm naming conformance.
> > 
> > Signed-off-by: Sergei Zhmylev <s.zhmylev@yadro.com>
> > ---
> >  meta/classes-global/package_rpm.bbclass | 4 ++++
> >  1 file changed, 4 insertions(+)
> > 
> > diff --git a/meta/classes-global/package_rpm.bbclass
> > b/meta/classes-global/package_rpm.bbclass
> > index 85d0bd7fce..3fb7466778 100644
> > --- a/meta/classes-global/package_rpm.bbclass
> > +++ b/meta/classes-global/package_rpm.bbclass
> > @@ -678,6 +678,10 @@ python do_package_rpm () {
> >      bb.utils.mkdirhier(pkgwritedir)
> >      os.chmod(pkgwritedir, 0o755)
> > 
> > +    rpm_arch_override = d.getVar("RPM_ARCH_OVERRIDE")
> > +    if package_arch != "noarch" and rpm_arch_override:
> > +        pkgarch = rpm_arch_override
> > +
> >      cmd = rpmbuild
> >      cmd = cmd + " --noclean --nodeps --short-circuit --target " +
> > pkgarch + " --buildroot " + pkgd
> >      cmd = cmd + " --define '_topdir " + workdir + "' --define
> > '_rpmdir " + pkgwritedir + "'"
> 
> This makes me very nervous. PACKAGE_ARCH can be overridden and I know
> of BSPs which inject a whole new level of architecture into the
> system
> but they don't need to make rpm backend specific changes to make it
> work.
> 
> If we were to add such a thing we'd need a much clearer picture of
> when
> it should be used and why it is needed. Wouldn't all package backends
> need to adjust this?
> 
> Cheers,
> 
> Richard
>

On Fri, 13 Jan 2023 at 09:25, Sergey Zhmylev <s.zhmylev@yadro.com> wrote:
>
> Exactly! Even more: when I build for "genericx86_64", I get mix of
> packages with architecture one of: "x86_64", "genericx86_64",
> "core2_64", and "noarch". Our audit team highlited this as this is not
> compliant with common RPM naming convention.

Can you please link to (or describe with references) the common naming
convention?

Alex

Sure, RPM naming convention from rpm.org:
http://ftp.rpm.org/max-rpm/ch-rpm-file-format.html
As you can see, there are nothing about "core2_64" architecture.

-- 
With best wishes,
Sergei Zhmylev
Engineering consultant
OS development department








On Fri, 2023-01-13 at 09:34 +0100, Alexander Kanavin wrote:
> «Внимание! Данное письмо от внешнего адресата!»
> 
> On Fri, 13 Jan 2023 at 09:25, Sergey Zhmylev <s.zhmylev@yadro.com>
> wrote:
> > 
> > Exactly! Even more: when I build for "genericx86_64", I get mix of
> > packages with architecture one of: "x86_64", "genericx86_64",
> > "core2_64", and "noarch". Our audit team highlited this as this is
> > not
> > compliant with common RPM naming convention.
> 
> Can you please link to (or describe with references) the common
> naming
> convention?
> 
> Alex
>

This document hasn't been maintained in decades, and is not in any way official.

Alex

On Fri, 13 Jan 2023 at 10:14, Sergey Zhmylev <s.zhmylev@yadro.com> wrote:
>
> Sure, RPM naming convention from rpm.org:
> http://ftp.rpm.org/max-rpm/ch-rpm-file-format.html
> As you can see, there are nothing about "core2_64" architecture.
>
> --
> With best wishes,
> Sergei Zhmylev
> Engineering consultant
> OS development department
>
>
>
>
>
>
>
>
> On Fri, 2023-01-13 at 09:34 +0100, Alexander Kanavin wrote:
> > «Внимание! Данное письмо от внешнего адресата!»
> >
> > On Fri, 13 Jan 2023 at 09:25, Sergey Zhmylev <s.zhmylev@yadro.com>
> > wrote:
> > >
> > > Exactly! Even more: when I build for "genericx86_64", I get mix of
> > > packages with architecture one of: "x86_64", "genericx86_64",
> > > "core2_64", and "noarch". Our audit team highlited this as this is
> > > not
> > > compliant with common RPM naming convention.
> >
> > Can you please link to (or describe with references) the common
> > naming
> > convention?
> >
> > Alex
> >

On Fri, 2023-01-13 at 09:14 +0000, Sergey Zhmylev wrote:
> Sure, RPM naming convention from rpm.org:
> http://ftp.rpm.org/max-rpm/ch-rpm-file-format.html
> As you can see, there are nothing about "core2_64" architecture.

We build target binaries to different levels of tuning, there are
packages which run everywhere (noarch), there are binaries which are
optimised to a specific tune and there is machine specific output.
People can inject other levels of packages as well. Giving a hint of
the kind of thing a given rpm contains seems like a perfectly
reasonable thing for us to be doing.

If you come from the rpm world, yes it is slightly unusual but as the
document above says, it works fine. We're not "in-compliant", we're
just adapting rpm usage to the environment it is being used in.

So no, I don't think we should be taking a patch which would just
confuse things for what seems to be a cosmetic reason.

Can you perhaps convince your audit team this is a sensible usage given
how the project builds and uses the rpms?

Cheers,

Richard

On Fri, 13 Jan 2023 at 10:50, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Can you perhaps convince your audit team this is a sensible usage given
> how the project builds and uses the rpms?

One more thing I want to mention is that if you want to align the
architectures so that you can install RPMs from Fedora, RHEL or any
other RPM-based binary distro, don't. It's not going to work at all,
or will cause you lots of pain and suffering. We use rpm for its
packaging and dependency resolution functions, but no one ever
promised any kind of 'compatibility' with Red Hat products.

Alex

> One more thing I want to mention is that if you want to align the
> architectures so that you can install RPMs from Fedora, RHEL or any
> other RPM-based binary distro, don't. It's not going to work at all,
> or will cause you lots of pain and suffering. We use rpm for its
> packaging and dependency resolution functions, but no one ever
> promised any kind of 'compatibility' with Red Hat products.
> 

Indeed, I'm not going to align those names for this.  Moreover, for
this particular case rpm(1) supports several parameters to disable
arch/os/etc checks.

> So no, I don't think we should be taking a patch which would just
> confuse things for what seems to be a cosmetic reason.
> 

The patch proposed is not about a cosmetic change, it's about precise
control over what packaging system produces.  Given the recipes can
override variables with a priority over any confs/classes, it looks
like there is no proper way to overload the values (in particular
PACKAGE_ARCH) used during packaging process.  Some variables could be
used to adjust TUNE flags and their values should stay intact prior to
packaging.

Anyways, your word will be the last but I am still sure that "more
control" is better than "no conrol". 

Kind wishes, Sergei

> On 13 Jan 2023, at 12:29, Sergey Zhmylev via lists.openembedded.org <s.zhmylev=yadro.com@lists.openembedded.org> wrote:
>> So no, I don't think we should be taking a patch which would just
>> confuse things for what seems to be a cosmetic reason.
>> 
> 
> The patch proposed is not about a cosmetic change, it's about precise
> control over what packaging system produces.  Given the recipes can
> override variables with a priority over any confs/classes, it looks
> like there is no proper way to overload the values (in particular
> PACKAGE_ARCH) used during packaging process.  Some variables could be
> used to adjust TUNE flags and their values should stay intact prior to
> packaging.

What is the use-case for this level of control then, and why would it be specific to RPM?

Ross

> 
> What is the use-case for this level of control then, and why would it
> be specific to RPM?
> 

Despite Alexander's words about this document, most of (I suppose,
actually all) wide-used rpm-based distros use the same well-known rpm
packages naming convention: red hat, opensuse, centos, oracle
enterprise linux, fedora, ...

There are a lot of useful tools in this huge world of enterprise-level
distros, especially all those staff related to static security
scans/audit and around SBOM collection and processing.  Those tools
rely on some basic rules of rpm naming for their analysis and do not
handle architectures like "genericx86_64".

I.e. this level of control opens for us a tremendous window of
possibilities when our build pretends to be a mature distro.

BR, Sergei

On Fri, Jan 13, 2023 at 11:01 AM Sergey Zhmylev <s.zhmylev@yadro.com> wrote:
>
> >
> > What is the use-case for this level of control then, and why would it
> > be specific to RPM?
> >
>
> Despite Alexander's words about this document, most of (I suppose,
> actually all) wide-used rpm-based distros use the same well-known rpm
> packages naming convention: red hat, opensuse, centos, oracle
> enterprise linux, fedora, ...
>
> There are a lot of useful tools in this huge world of enterprise-level
> distros, especially all those staff related to static security
> scans/audit and around SBOM collection and processing.  Those tools
> rely on some basic rules of rpm naming for their analysis and do not
> handle architectures like "genericx86_64".

You mean things that the Yocto Project does itself ? :)

My humorous answer is on purpose. If there are gaps in the tooling,
the project is interested in filing those gaps, and doing it in a way
that anyone using the project could leverage. Not just the users of
a specific packaging format.

It isn't that we wouldn't want to enable that sort of tools / packaging
capability, but if it opens us up to complexity or other issues down
the road, it needs to be carefully considered. Which is what the questions
in the thread are trying to determine.

>
> I.e. this level of control opens for us a tremendous window of
> possibilities when our build pretends to be a mature distro.
>

FWIW. I would not characterize a distro created with the
Yocto project as "immature".

Cheers,

Bruce

> BR, Sergei
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#175836): https://lists.openembedded.org/g/openembedded-core/message/175836
> Mute This Topic: https://lists.openembedded.org/mt/96233100/1050810
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

On Fri, 13 Jan 2023 at 17:01, Sergey Zhmylev <s.zhmylev@yadro.com> wrote:
> There are a lot of useful tools in this huge world of enterprise-level
> distros, especially all those staff related to static security
> scans/audit and around SBOM collection and processing.  Those tools
> rely on some basic rules of rpm naming for their analysis and do not
> handle architectures like "genericx86_64".

Can you list the tools please? Are they under open source licenses?
What do they do that Yocto doesn't yet offer? Yocto does CVE scans,
and SBOM generation as well, so what is it that is missing?

It also goes both ways: you can figure out why the tools do not handle
architectures in a generic way and submit a patch to respective
upstreams so that they do.

'Enterprise-level distros' means vendor lock-in and expensive
subscription fees. Precisely the thing that most or all of us here do
not want. Yes, we are not mature.

Alex
-