diff mbox series

[1/3] oe/utils: extract method for parallel_make without d context

Message ID 20250808055059.8851-1-peter.marko@siemens.com
State New
Headers show
Series [1/3] oe/utils: extract method for parallel_make without d context | expand

Commit Message

Marko, Peter Aug. 8, 2025, 5:50 a.m. UTC 
From: Peter Marko <peter.marko@siemens.com>

oeqa does not have general access to d variable context and needs to
determine parallel make settings.
Extract the code from parallel_make into reusable parallel_make_value.

Also correct function description of return value from None to empty
string.

Signed-off-by: Peter Marko <peter.marko@siemens.com>
---
 meta/lib/oe/utils.py | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)

Comments

Marko, Peter Aug. 8, 2025, 6 a.m. UTC | #1 
Note that this change rebuilds from scratch (with rn_work or empty tmpdir).
So the patch verification took me most of the day.
Although the value of PARALLEL_MAKE does not change and hashes of configure/compile stay the same,
hash of populate_sysroot changes although it does not evaluate parallel_make at all.
Is this a bug or a feature?

[poky]/home/projects/build/work-ccp7$ bitbake-diffsigs {/tmp,projects/poky/tmp}/stamps/x86_64-linux/quilt-native/0.69.do_populate_sysroot.sigdata.*
NOTE: Starting bitbake server...
basehash changed from 3e355b49fe002160af197c4418d78fe5e6e3f5ee2bbd294bc5ebe2f6cd62c93a to 13cd423267171f8ba581c9bf6c59a2380690daa9fa5dba3f14f33852d0c138fc
Variable oe.utils.get_bb_number_threads value changed from '7789d42c55d3455966829336c3431581e91ea7f90daff00c90ba0bbc747625ac' to '3c2db6724598a1d5fdcef41fbb3e0971256afed3a3e1f891cd4b360864802a3f'
Variable oe.utils.multiprocess_launch_mp value changed from '6fcecddf5f515afff9d1970920f983978d2e68da9ef052ec5e434062795d61ff' to '3a7bc5ba90a4b7dcaae29af22611269610cad86f9ad8f3db6a27234d046239c3'

Peter

> -----Original Message-----
> From: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
> Sent: Friday, August 8, 2025 7:51
> To: openembedded-core@lists.openembedded.org
> Cc: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
> Subject: [OE-core][PATCH 1/3] oe/utils: extract method for parallel_make without d
> context
> 
> From: Peter Marko <peter.marko@siemens.com>
> 
> oeqa does not have general access to d variable context and needs to
> determine parallel make settings.
> Extract the code from parallel_make into reusable parallel_make_value.
> 
> Also correct function description of return value from None to empty
> string.
> 
> Signed-off-by: Peter Marko <peter.marko@siemens.com>
> ---
>  meta/lib/oe/utils.py | 26 ++++++++++++++++++--------
>  1 file changed, 18 insertions(+), 8 deletions(-)
> 
> diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
> index 779c5e593f..8aa15373f1 100644
> --- a/meta/lib/oe/utils.py
> +++ b/meta/lib/oe/utils.py
> @@ -174,18 +174,14 @@ def any_distro_features(d, features, truevalue="1",
> falsevalue=""):
>      """
>      return bb.utils.contains_any("DISTRO_FEATURES", features, truevalue,
> falsevalue, d)
> 
> -def parallel_make(d, makeinst=False):
> +def parallel_make_value(pm):
>      """
>      Return the integer value for the number of parallel threads to use when
> -    building, scraped out of PARALLEL_MAKE. If no parallelization option is
> -    found, returns None
> +    building, scraped out of given string. If no parallelization option is
> +    found, returns empty string
> 
> -    e.g. if PARALLEL_MAKE = "-j 10", this will return 10 as an integer.
> +    e.g. if string is "-j 10", this will return 10 as an integer.
>      """
> -    if makeinst:
> -        pm = (d.getVar('PARALLEL_MAKEINST') or '').split()
> -    else:
> -        pm = (d.getVar('PARALLEL_MAKE') or '').split()
>      # look for '-j' and throw other options (e.g. '-l') away
>      while pm:
>          opt = pm.pop(0)
> @@ -200,6 +196,20 @@ def parallel_make(d, makeinst=False):
> 
>      return ''
> 
> +def parallel_make(d, makeinst=False):
> +    """
> +    Return the integer value for the number of parallel threads to use when
> +    building, scraped out of PARALLEL_MAKE. If no parallelization option is
> +    found, returns empty string
> +
> +    e.g. if PARALLEL_MAKE = "-j 10", this will return 10 as an integer.
> +    """
> +    if makeinst:
> +        pm = (d.getVar('PARALLEL_MAKEINST') or '').split()
> +    else:
> +        pm = (d.getVar('PARALLEL_MAKE') or '').split()
> +    return parallel_make_value(pm)
> +
>  def parallel_make_argument(d, fmt, limit=None, makeinst=False):
>      """
>      Helper utility to construct a parallel make argument from the number of
Richard Purdie Aug. 8, 2025, 6:38 a.m. UTC | #2 
On Fri, 2025-08-08 at 06:00 +0000, Marko, Peter wrote:
> Note that this change rebuilds from scratch (with rn_work or empty tmpdir).
> So the patch verification took me most of the day.
> Although the value of PARALLEL_MAKE does not change and hashes of configure/compile stay the same,
> hash of populate_sysroot changes although it does not evaluate parallel_make at all.
> Is this a bug or a feature?
> 
> [poky]/home/projects/build/work-ccp7$ bitbake-diffsigs {/tmp,projects/poky/tmp}/stamps/x86_64-linux/quilt-native/0.69.do_populate_sysroot.sigdata.*
> NOTE: Starting bitbake server...
> basehash changed from 3e355b49fe002160af197c4418d78fe5e6e3f5ee2bbd294bc5ebe2f6cd62c93a to 13cd423267171f8ba581c9bf6c59a2380690daa9fa5dba3f14f33852d0c138fc
> Variable oe.utils.get_bb_number_threads value changed from '7789d42c55d3455966829336c3431581e91ea7f90daff00c90ba0bbc747625ac' to '3c2db6724598a1d5fdcef41fbb3e0971256afed3a3e1f891cd4b360864802a3f'
> Variable oe.utils.multiprocess_launch_mp value changed from '6fcecddf5f515afff9d1970920f983978d2e68da9ef052ec5e434062795d61ff' to '3a7bc5ba90a4b7dcaae29af22611269610cad86f9ad8f3db6a27234d046239c3'

Feature. It is telling you the code changed and hence the task
checksums using that code changed and it rebuilt.

Cheers,

Richard
Marko, Peter Aug. 8, 2025, 10:20 a.m. UTC | #3 
Please ignore this series, somehow a broken testing version got out.
I'll send a v2.

Peter

> -----Original Message-----
> From: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
> Sent: Friday, August 8, 2025 7:51
> To: openembedded-core@lists.openembedded.org
> Cc: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>
> Subject: [OE-core][PATCH 1/3] oe/utils: extract method for parallel_make without d
> context
> 
> From: Peter Marko <peter.marko@siemens.com>
> 
> oeqa does not have general access to d variable context and needs to
> determine parallel make settings.
> Extract the code from parallel_make into reusable parallel_make_value.
> 
> Also correct function description of return value from None to empty
> string.
> 
> Signed-off-by: Peter Marko <peter.marko@siemens.com>
> ---
>  meta/lib/oe/utils.py | 26 ++++++++++++++++++--------
>  1 file changed, 18 insertions(+), 8 deletions(-)
> 
> diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
> index 779c5e593f..8aa15373f1 100644
> --- a/meta/lib/oe/utils.py
> +++ b/meta/lib/oe/utils.py
> @@ -174,18 +174,14 @@ def any_distro_features(d, features, truevalue="1",
> falsevalue=""):
>      """
>      return bb.utils.contains_any("DISTRO_FEATURES", features, truevalue,
> falsevalue, d)
> 
> -def parallel_make(d, makeinst=False):
> +def parallel_make_value(pm):
>      """
>      Return the integer value for the number of parallel threads to use when
> -    building, scraped out of PARALLEL_MAKE. If no parallelization option is
> -    found, returns None
> +    building, scraped out of given string. If no parallelization option is
> +    found, returns empty string
> 
> -    e.g. if PARALLEL_MAKE = "-j 10", this will return 10 as an integer.
> +    e.g. if string is "-j 10", this will return 10 as an integer.
>      """
> -    if makeinst:
> -        pm = (d.getVar('PARALLEL_MAKEINST') or '').split()
> -    else:
> -        pm = (d.getVar('PARALLEL_MAKE') or '').split()
>      # look for '-j' and throw other options (e.g. '-l') away
>      while pm:
>          opt = pm.pop(0)
> @@ -200,6 +196,20 @@ def parallel_make(d, makeinst=False):
> 
>      return ''
> 
> +def parallel_make(d, makeinst=False):
> +    """
> +    Return the integer value for the number of parallel threads to use when
> +    building, scraped out of PARALLEL_MAKE. If no parallelization option is
> +    found, returns empty string
> +
> +    e.g. if PARALLEL_MAKE = "-j 10", this will return 10 as an integer.
> +    """
> +    if makeinst:
> +        pm = (d.getVar('PARALLEL_MAKEINST') or '').split()
> +    else:
> +        pm = (d.getVar('PARALLEL_MAKE') or '').split()
> +    return parallel_make_value(pm)
> +
>  def parallel_make_argument(d, fmt, limit=None, makeinst=False):
>      """
>      Helper utility to construct a parallel make argument from the number of
diff mbox series

Patch

diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
index 779c5e593f..8aa15373f1 100644
--- a/meta/lib/oe/utils.py
+++ b/meta/lib/oe/utils.py
@@ -174,18 +174,14 @@  def any_distro_features(d, features, truevalue="1", falsevalue=""):
     """
     return bb.utils.contains_any("DISTRO_FEATURES", features, truevalue, falsevalue, d)
 
-def parallel_make(d, makeinst=False):
+def parallel_make_value(pm):
     """
     Return the integer value for the number of parallel threads to use when
-    building, scraped out of PARALLEL_MAKE. If no parallelization option is
-    found, returns None
+    building, scraped out of given string. If no parallelization option is
+    found, returns empty string
 
-    e.g. if PARALLEL_MAKE = "-j 10", this will return 10 as an integer.
+    e.g. if string is "-j 10", this will return 10 as an integer.
     """
-    if makeinst:
-        pm = (d.getVar('PARALLEL_MAKEINST') or '').split()
-    else:
-        pm = (d.getVar('PARALLEL_MAKE') or '').split()
     # look for '-j' and throw other options (e.g. '-l') away
     while pm:
         opt = pm.pop(0)
@@ -200,6 +196,20 @@  def parallel_make(d, makeinst=False):
 
     return ''
 
+def parallel_make(d, makeinst=False):
+    """
+    Return the integer value for the number of parallel threads to use when
+    building, scraped out of PARALLEL_MAKE. If no parallelization option is
+    found, returns empty string
+
+    e.g. if PARALLEL_MAKE = "-j 10", this will return 10 as an integer.
+    """
+    if makeinst:
+        pm = (d.getVar('PARALLEL_MAKEINST') or '').split()
+    else:
+        pm = (d.getVar('PARALLEL_MAKE') or '').split()
+    return parallel_make_value(pm)
+
 def parallel_make_argument(d, fmt, limit=None, makeinst=False):
     """
     Helper utility to construct a parallel make argument from the number of