diff mbox series

reproducible: allow each layer to define its known non-repro packages

Message ID 20250603215916.2429422-1-yoann.congal@smile.fr
State New
Headers show
Series reproducible: allow each layer to define its known non-repro packages | expand

Commit Message

Yoann Congal June 3, 2025, 9:59 p.m. UTC 
From: Guillaume Swaenepoel <guillaume.swaenepoel@smile.fr>

With this change, layers can use a conf/include/known_non_repro.inc file
to extend the KNOWN_NON_REPRO_PACKAGES_LIST variable with its own known
non-reproducible packages to make them automatically excluded from the
reproducible test.

Example of usage, in conf/include/known_non_repro.inc:
  KNOWN_NON_REPRO_PACKAGES_LIST += " \
        this-package-is-not-reproducible \
  "

This uses the new "include_all" in the reproducible test and will move
the maintenance burden from AB config[0] to the layers themselves.

Fixes [YOCTO #15699]

[0]: https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json#n343

Co-Developed-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Guillaume Swaenepoel <guillaume.swaenepoel@smile.fr>
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

Comments

Yoann Congal June 4, 2025, 7:53 a.m. UTC | #1 
Le mer. 4 juin 2025 à 00:02, Yoann Congal <yoann.congal@smile.fr> a écrit :

> From: Guillaume Swaenepoel <guillaume.swaenepoel@smile.fr>
>
> With this change, layers can use a conf/include/known_non_repro.inc file
> to extend the KNOWN_NON_REPRO_PACKAGES_LIST variable with its own known
> non-reproducible packages to make them automatically excluded from the
> reproducible test.
>
> Example of usage, in conf/include/known_non_repro.inc:
>   KNOWN_NON_REPRO_PACKAGES_LIST += " \
>         this-package-is-not-reproducible \
>   "
>
> This uses the new "include_all" in the reproducible test and will move
> the maintenance burden from AB config[0] to the layers themselves.
>
> Fixes [YOCTO #15699]
>
> [0]:
> https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json#n343
>
> Co-Developed-by: Yoann Congal <yoann.congal@smile.fr>
> Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
> Signed-off-by: Guillaume Swaenepoel <guillaume.swaenepoel@smile.fr>
> ---
>

I sent the meta-oe application/example as RFC in
[RFC PATCH] reproducibility: move known non-repro list to the new common
file
https://lists.openembedded.org/g/openembedded-devel/message/117731


>  meta/lib/oeqa/selftest/cases/reproducible.py | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
>
> diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py
> b/meta/lib/oeqa/selftest/cases/reproducible.py
> index f06027cb03..e2c3a10f06 100644
> --- a/meta/lib/oeqa/selftest/cases/reproducible.py
> +++ b/meta/lib/oeqa/selftest/cases/reproducible.py
> @@ -14,6 +14,7 @@ import shutil
>  import stat
>  import os
>  import datetime
> +import oeqa.utils.ftools as ftools
>
>  exclude_packages = [
>         ]
> @@ -158,6 +159,12 @@ class ReproducibleTests(OESelftestTestCase):
>      # reproducible, disable this in your derived test class
>      build_from_sstate = True
>
> +    @classmethod
> +    def setUpClass(cls):
> +        super().setUpClass()
> +        ftools.append_file(os.path.join(cls.builddir, "conf/local.conf"),
> \
> +             "\ninclude_all conf/include/known_non_repro.inc\n")
> +
>      def setUpLocal(self):
>          super().setUpLocal()
>          needed_vars = [
> @@ -170,6 +177,7 @@ class ReproducibleTests(OESelftestTestCase):
>              'OEQA_REPRODUCIBLE_TEST_SSTATE_TARGETS',
>              'OEQA_REPRODUCIBLE_EXCLUDED_PACKAGES',
>              'OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS',
> +            'KNOWN_NON_REPRO_PACKAGES_LIST',
>          ]
>          bb_vars = get_bb_vars(needed_vars)
>          for v in needed_vars:
> @@ -189,6 +197,10 @@ class ReproducibleTests(OESelftestTestCase):
>              for leaf_recipe in
> bb_vars['OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS'].split():
>                  self.sstate_targets.extend(get_bb_var('DEPENDS',
> leaf_recipe).split())
>
> +        # Apply exclusion from layers (conf/include/known_non_repro.inc
> files)
> +        if bb_vars['KNOWN_NON_REPRO_PACKAGES_LIST']:
> +            self.oeqa_reproducible_excluded_packages =
> (self.oeqa_reproducible_excluded_packages or "") +
> bb_vars['KNOWN_NON_REPRO_PACKAGES_LIST']
> +
>          self.extraresults = {}
>          self.extraresults.setdefault('reproducible',
> {}).setdefault('files', {})
>
>
Peter Kjellerstedt June 4, 2025, 9:14 a.m. UTC | #2 
> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Yoann Congal via lists.openembedded.org
> Sent: den 3 juni 2025 23:59
> To: openembedded-core@lists.openembedded.org
> Cc: Yoann Congal <yoann.congal@smile.fr>; Guillaume Swaenepoel <guillaume.swaenepoel@smile.fr>
> Subject: [OE-core] [PATCH] reproducible: allow each layer to define its known non-repro packages
> 
> From: Guillaume Swaenepoel <guillaume.swaenepoel@smile.fr>
> 
> With this change, layers can use a conf/include/known_non_repro.inc file
> to extend the KNOWN_NON_REPRO_PACKAGES_LIST variable with its own known
> non-reproducible packages to make them automatically excluded from the
> reproducible test.
> 
> Example of usage, in conf/include/known_non_repro.inc:
>   KNOWN_NON_REPRO_PACKAGES_LIST += " \
>         this-package-is-not-reproducible \
>   "
> 
> This uses the new "include_all" in the reproducible test and will move
> the maintenance burden from AB config[0] to the layers themselves.
> 
> Fixes [YOCTO #15699]
> 
> [0]: https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json#n343
> 
> Co-Developed-by: Yoann Congal <yoann.congal@smile.fr>
> Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
> Signed-off-by: Guillaume Swaenepoel <guillaume.swaenepoel@smile.fr>
> ---
>  meta/lib/oeqa/selftest/cases/reproducible.py | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
> index f06027cb03..e2c3a10f06 100644
> --- a/meta/lib/oeqa/selftest/cases/reproducible.py
> +++ b/meta/lib/oeqa/selftest/cases/reproducible.py
> @@ -14,6 +14,7 @@ import shutil
>  import stat
>  import os
>  import datetime
> +import oeqa.utils.ftools as ftools
> 
>  exclude_packages = [
>  	]
> @@ -158,6 +159,12 @@ class ReproducibleTests(OESelftestTestCase):
>      # reproducible, disable this in your derived test class
>      build_from_sstate = True
> 
> +    @classmethod
> +    def setUpClass(cls):
> +        super().setUpClass()
> +        ftools.append_file(os.path.join(cls.builddir, "conf/local.conf"), \
> +             "\ninclude_all conf/include/known_non_repro.inc\n")
> +
>      def setUpLocal(self):
>          super().setUpLocal()
>          needed_vars = [
> @@ -170,6 +177,7 @@ class ReproducibleTests(OESelftestTestCase):
>              'OEQA_REPRODUCIBLE_TEST_SSTATE_TARGETS',
>              'OEQA_REPRODUCIBLE_EXCLUDED_PACKAGES',
>              'OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS',
> +            'KNOWN_NON_REPRO_PACKAGES_LIST',

Why do you need a new variable? Isn't this what 
OEQA_REPRODUCIBLE_EXCLUDED_PACKAGES is used for?

>          ]
>          bb_vars = get_bb_vars(needed_vars)
>          for v in needed_vars:
> @@ -189,6 +197,10 @@ class ReproducibleTests(OESelftestTestCase):
>              for leaf_recipe in bb_vars['OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS'].split():
>                  self.sstate_targets.extend(get_bb_var('DEPENDS', leaf_recipe).split())
> 
> +        # Apply exclusion from layers (conf/include/known_non_repro.inc files)
> +        if bb_vars['KNOWN_NON_REPRO_PACKAGES_LIST']:
> +            self.oeqa_reproducible_excluded_packages = (self.oeqa_reproducible_excluded_packages or "") + bb_vars['KNOWN_NON_REPRO_PACKAGES_LIST']
> +
>          self.extraresults = {}
>          self.extraresults.setdefault('reproducible', {}).setdefault('files', {})
> 

//Peter
Yoann Congal June 4, 2025, 9:20 a.m. UTC | #3 
Le mer. 4 juin 2025 à 11:14, Peter Kjellerstedt <peter.kjellerstedt@axis.com>
a écrit :

> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <
> openembedded-core@lists.openembedded.org> On Behalf Of Yoann Congal via
> lists.openembedded.org
> > Sent: den 3 juni 2025 23:59
> > To: openembedded-core@lists.openembedded.org
> > Cc: Yoann Congal <yoann.congal@smile.fr>; Guillaume Swaenepoel <
> guillaume.swaenepoel@smile.fr>
> > Subject: [OE-core] [PATCH] reproducible: allow each layer to define its
> known non-repro packages
> >
> > From: Guillaume Swaenepoel <guillaume.swaenepoel@smile.fr>
> >
> > With this change, layers can use a conf/include/known_non_repro.inc file
> > to extend the KNOWN_NON_REPRO_PACKAGES_LIST variable with its own known
> > non-reproducible packages to make them automatically excluded from the
> > reproducible test.
> >
> > Example of usage, in conf/include/known_non_repro.inc:
> >   KNOWN_NON_REPRO_PACKAGES_LIST += " \
> >         this-package-is-not-reproducible \
> >   "
> >
> > This uses the new "include_all" in the reproducible test and will move
> > the maintenance burden from AB config[0] to the layers themselves.
> >
> > Fixes [YOCTO #15699]
> >
> > [0]:
> https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json#n343
> >
> > Co-Developed-by: Yoann Congal <yoann.congal@smile.fr>
> > Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
> > Signed-off-by: Guillaume Swaenepoel <guillaume.swaenepoel@smile.fr>
> > ---
> >  meta/lib/oeqa/selftest/cases/reproducible.py | 12 ++++++++++++
> >  1 file changed, 12 insertions(+)
> >
> > diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py
> b/meta/lib/oeqa/selftest/cases/reproducible.py
> > index f06027cb03..e2c3a10f06 100644
> > --- a/meta/lib/oeqa/selftest/cases/reproducible.py
> > +++ b/meta/lib/oeqa/selftest/cases/reproducible.py
> > @@ -14,6 +14,7 @@ import shutil
> >  import stat
> >  import os
> >  import datetime
> > +import oeqa.utils.ftools as ftools
> >
> >  exclude_packages = [
> >       ]
> > @@ -158,6 +159,12 @@ class ReproducibleTests(OESelftestTestCase):
> >      # reproducible, disable this in your derived test class
> >      build_from_sstate = True
> >
> > +    @classmethod
> > +    def setUpClass(cls):
> > +        super().setUpClass()
> > +        ftools.append_file(os.path.join(cls.builddir,
> "conf/local.conf"), \
> > +             "\ninclude_all conf/include/known_non_repro.inc\n")
> > +
> >      def setUpLocal(self):
> >          super().setUpLocal()
> >          needed_vars = [
> > @@ -170,6 +177,7 @@ class ReproducibleTests(OESelftestTestCase):
> >              'OEQA_REPRODUCIBLE_TEST_SSTATE_TARGETS',
> >              'OEQA_REPRODUCIBLE_EXCLUDED_PACKAGES',
> >              'OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS',
> > +            'KNOWN_NON_REPRO_PACKAGES_LIST',
>
> Why do you need a new variable? Isn't this what
> OEQA_REPRODUCIBLE_EXCLUDED_PACKAGES is used for?
>

The idea was at some point to de-couple the two to allow to run the test
without the exclusion coming automatically from the layer files.
I may have over-thought this one.


> >          ]
> >          bb_vars = get_bb_vars(needed_vars)
> >          for v in needed_vars:
> > @@ -189,6 +197,10 @@ class ReproducibleTests(OESelftestTestCase):
> >              for leaf_recipe in
> bb_vars['OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS'].split():
> >                  self.sstate_targets.extend(get_bb_var('DEPENDS',
> leaf_recipe).split())
> >
> > +        # Apply exclusion from layers (conf/include/known_non_repro.inc
> files)
> > +        if bb_vars['KNOWN_NON_REPRO_PACKAGES_LIST']:
> > +            self.oeqa_reproducible_excluded_packages =
> (self.oeqa_reproducible_excluded_packages or "") +
> bb_vars['KNOWN_NON_REPRO_PACKAGES_LIST']
> > +
> >          self.extraresults = {}
> >          self.extraresults.setdefault('reproducible',
> {}).setdefault('files', {})
> >
>
> //Peter
>
>
Quentin Schulz June 4, 2025, 9:22 a.m. UTC | #4 
Hi Yoann,

On 6/3/25 11:59 PM, Yoann Congal via lists.openembedded.org wrote:
> From: Guillaume Swaenepoel <guillaume.swaenepoel@smile.fr>
> 
> With this change, layers can use a conf/include/known_non_repro.inc file
> to extend the KNOWN_NON_REPRO_PACKAGES_LIST variable with its own known
> non-reproducible packages to make them automatically excluded from the
> reproducible test.
> 
> Example of usage, in conf/include/known_non_repro.inc:
>    KNOWN_NON_REPRO_PACKAGES_LIST += " \
>          this-package-is-not-reproducible \
>    "
> 
> This uses the new "include_all" in the reproducible test and will move
> the maintenance burden from AB config[0] to the layers themselves.
> 

A few points unrelated to implementation:
- maybe we can shorten the variable name? NON_REPRO_PACKAGES would be 
enough no? If a package is listed in there, it's necessarily that we 
know it's non reproducible, would we have a variable for "unknown" 
non-reproducible packages? Additionally, we already have a PACKAGES 
variable which contains a space-separated list of packages to build, so 
maybe we can drop the _LIST as the plural of PACKAGES kind of imply it's 
a list?
- once this gets merged, please provide documentation in yocto-docs for 
it :)
- why packages and not recipes? Do we need a per-package granularity? 
Isn't the recipe broken if any package it generates is non-reproducible? 
This would also allow recipes to add packages without having to also 
list them to the NON_REPRO_PACKAGES list?
- what happens if a layer has a version X of a recipe that is not 
reproducible, but another layer has version Y that is (or vice-versa)?

Cheers,
Quentin
Yoann Congal June 4, 2025, 9:48 a.m. UTC | #5 
Le mer. 4 juin 2025 à 11:22, Quentin Schulz <quentin.schulz@cherry.de> a
écrit :

> Hi Yoann,
>
> On 6/3/25 11:59 PM, Yoann Congal via lists.openembedded.org wrote:
> > From: Guillaume Swaenepoel <guillaume.swaenepoel@smile.fr>
> >
> > With this change, layers can use a conf/include/known_non_repro.inc file
> > to extend the KNOWN_NON_REPRO_PACKAGES_LIST variable with its own known
> > non-reproducible packages to make them automatically excluded from the
> > reproducible test.
> >
> > Example of usage, in conf/include/known_non_repro.inc:
> >    KNOWN_NON_REPRO_PACKAGES_LIST += " \
> >          this-package-is-not-reproducible \
> >    "
> >
> > This uses the new "include_all" in the reproducible test and will move
> > the maintenance burden from AB config[0] to the layers themselves.
> >
>
> A few points unrelated to implementation:
> - maybe we can shorten the variable name? NON_REPRO_PACKAGES would be
> enough no? If a package is listed in there, it's necessarily that we
> know it's non reproducible, would we have a variable for "unknown"
> non-reproducible packages? Additionally, we already have a PACKAGES
> variable which contains a space-separated list of packages to build, so
> maybe we can drop the _LIST as the plural of PACKAGES kind of imply it's
> a list?
>

Yes, I can shorten it a lot (I was kind of afraid to clash with existing
variables).

I agree on NON_REPRO_PACKAGES (if we don't switch to the more
direct OEQA_REPRODUCIBLE_EXCLUDED_PACKAGES)


> - once this gets merged, please provide documentation in yocto-docs for
> it :)
>

That was planned :)


> - why packages and not recipes? Do we need a per-package granularity?
>

The reproducibility test works at package level. (I don't know if that's by
choice or because it's easier)
For example, the main package of a recipe may be reproducible but not the
-ptest one.
Changing this to recipe level is a bit more fundamental work than I
anticipated but I can do it if necessary.


> Isn't the recipe broken if any package it generates is non-reproducible?
> This would also allow recipes to add packages without having to also
> list them to the NON_REPRO_PACKAGES list?
>

I don't dislike the idea of requiring more work for
non-reproducible recipes.
At some point, it might even be easier to make the recipe reproducible ;)

Joking aside, reproducibility was for a long time handled asynchronously
(non-repro got merged, fixed/excluded later)


> - what happens if a layer has a version X of a recipe that is not
> reproducible, but another layer has version Y that is (or vice-versa)?
>

Right now, this is not handled very well : version is ignored by the
reproducible test. If layer X ignores the package, it will be ignored for
layer Y.

Taking a step back, if we are redesigning the whole thing, why not add a
variable into the known-non reproducible recipes? (Like INSANE_SKIPs)

Cheers,
> Quentin
>
Yoann Congal June 5, 2025, 10:24 a.m. UTC | #6 
Le mer. 4 juin 2025 à 11:48, Yoann Congal <yoann.congal@smile.fr> a écrit :

>
>
> Le mer. 4 juin 2025 à 11:22, Quentin Schulz <quentin.schulz@cherry.de> a
> écrit :
>
>> Hi Yoann,
>>
>> On 6/3/25 11:59 PM, Yoann Congal via lists.openembedded.org wrote:
>> > From: Guillaume Swaenepoel <guillaume.swaenepoel@smile.fr>
>> >
>> > With this change, layers can use a conf/include/known_non_repro.inc file
>> > to extend the KNOWN_NON_REPRO_PACKAGES_LIST variable with its own known
>> > non-reproducible packages to make them automatically excluded from the
>> > reproducible test.
>> >
>> > Example of usage, in conf/include/known_non_repro.inc:
>> >    KNOWN_NON_REPRO_PACKAGES_LIST += " \
>> >          this-package-is-not-reproducible \
>> >    "
>> >
>> > This uses the new "include_all" in the reproducible test and will move
>> > the maintenance burden from AB config[0] to the layers themselves.
>> >
>>
>> A few points unrelated to implementation:
>> - maybe we can shorten the variable name? NON_REPRO_PACKAGES would be
>> enough no? If a package is listed in there, it's necessarily that we
>> know it's non reproducible, would we have a variable for "unknown"
>> non-reproducible packages? Additionally, we already have a PACKAGES
>> variable which contains a space-separated list of packages to build, so
>> maybe we can drop the _LIST as the plural of PACKAGES kind of imply it's
>> a list?
>>
>
> Yes, I can shorten it a lot (I was kind of afraid to clash with existing
> variables).
>
> I agree on NON_REPRO_PACKAGES (if we don't switch to the more
> direct OEQA_REPRODUCIBLE_EXCLUDED_PACKAGES)
>
>
>> - once this gets merged, please provide documentation in yocto-docs for
>> it :)
>>
>
> That was planned :)
>
>
>> - why packages and not recipes? Do we need a per-package granularity?
>>
>
> The reproducibility test works at package level. (I don't know if that's
> by choice or because it's easier)
> For example, the main package of a recipe may be reproducible but not the
> -ptest one.
> Changing this to recipe level is a bit more fundamental work than I
> anticipated but I can do it if necessary.
>
>
>> Isn't the recipe broken if any package it generates is non-reproducible?
>> This would also allow recipes to add packages without having to also
>> list them to the NON_REPRO_PACKAGES list?
>>
>
> I don't dislike the idea of requiring more work for
> non-reproducible recipes.
> At some point, it might even be easier to make the recipe reproducible ;)
>
> Joking aside, reproducibility was for a long time handled asynchronously
> (non-repro got merged, fixed/excluded later)
>
>
>> - what happens if a layer has a version X of a recipe that is not
>> reproducible, but another layer has version Y that is (or vice-versa)?
>>
>
> Right now, this is not handled very well : version is ignored by the
> reproducible test. If layer X ignores the package, it will be ignored for
> layer Y.
>
> Taking a step back, if we are redesigning the whole thing, why not add a
> variable into the known-non reproducible recipes? (Like INSANE_SKIPs)
>

 Summarizing the discussion that happened at today's patch review call:
* RP agrees with the approach in general
* OEQA_REPRODUCIBLE_EXCLUDED_PACKAGES should be used instead (I'll send a
v2)
* We should stay at package level:
  * switching to recipe level may over complicate things for something that
is (hopefully) transient (all recipes/packages should be reproducible one
day)
  * Also, in case of a non-reproducible subpackage, we don't want to skip
the test on the main package (e.g ruby-doc was for a long time non-repro
but the test on the main ruby package was still valuable)

Regards,
diff mbox series

Patch

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index f06027cb03..e2c3a10f06 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -14,6 +14,7 @@  import shutil
 import stat
 import os
 import datetime
+import oeqa.utils.ftools as ftools
 
 exclude_packages = [
 	]
@@ -158,6 +159,12 @@  class ReproducibleTests(OESelftestTestCase):
     # reproducible, disable this in your derived test class
     build_from_sstate = True
 
+    @classmethod
+    def setUpClass(cls):
+        super().setUpClass()
+        ftools.append_file(os.path.join(cls.builddir, "conf/local.conf"), \
+             "\ninclude_all conf/include/known_non_repro.inc\n")
+
     def setUpLocal(self):
         super().setUpLocal()
         needed_vars = [
@@ -170,6 +177,7 @@  class ReproducibleTests(OESelftestTestCase):
             'OEQA_REPRODUCIBLE_TEST_SSTATE_TARGETS',
             'OEQA_REPRODUCIBLE_EXCLUDED_PACKAGES',
             'OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS',
+            'KNOWN_NON_REPRO_PACKAGES_LIST',
         ]
         bb_vars = get_bb_vars(needed_vars)
         for v in needed_vars:
@@ -189,6 +197,10 @@  class ReproducibleTests(OESelftestTestCase):
             for leaf_recipe in bb_vars['OEQA_REPRODUCIBLE_TEST_LEAF_TARGETS'].split():
                 self.sstate_targets.extend(get_bb_var('DEPENDS', leaf_recipe).split())
 
+        # Apply exclusion from layers (conf/include/known_non_repro.inc files)
+        if bb_vars['KNOWN_NON_REPRO_PACKAGES_LIST']:
+            self.oeqa_reproducible_excluded_packages = (self.oeqa_reproducible_excluded_packages or "") + bb_vars['KNOWN_NON_REPRO_PACKAGES_LIST']
+
         self.extraresults = {}
         self.extraresults.setdefault('reproducible', {}).setdefault('files', {})