| Message ID | 7c42e340baae4038324e250583e9395fb8d90463.1732725209.git.joerg.sommer@navimatix.de |
|---|---|
| State | New |
| Headers | show |
| Series | lib/oe/utils.cpu_count: Raise maximum count from 64 to 192 | expand |
Please read the original rationale for picking 64, and provide benchmarks that actually show better completion times with the new setting. Alex On Wed 27. Nov 2024 at 17.33, Jörg Sommer via lists.openembedded.org <joerg.sommer=navimatix.de@lists.openembedded.org> wrote: > From: Jörg Sommer <joerg.sommer@navimatix.de> > > We have a system with 96 CPUs and 128 are not uncommon. The border of 64 > limits the number of parallel tasks make or ninja spawns, because the value > goes into `PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"`. If only a > single > recipe gets build (e.g. rust, due to dependencies) this leaves one third of > our CPUs idle. > > Signed-off-by: Jörg Sommer <joerg.sommer@navimatix.de> > --- > meta/lib/oe/utils.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py > index c9c7a47041..354237b643 100644 > --- a/meta/lib/oe/utils.py > +++ b/meta/lib/oe/utils.py > @@ -251,7 +251,7 @@ def trim_version(version, num_parts=2): > trimmed = ".".join(parts[:num_parts]) > return trimmed > > -def cpu_count(at_least=1, at_most=64): > +def cpu_count(at_least=1, at_most=192): > cpus = len(os.sched_getaffinity(0)) > return max(min(cpus, at_most), at_least) > > -- > 2.45.2 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#207936): > https://lists.openembedded.org/g/openembedded-core/message/207936 > Mute This Topic: https://lists.openembedded.org/mt/109808873/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information: --- Testing patch /home/patchtest/share/mboxes/lib-oe-utils.cpu_count-Raise-maximum-count-from-64-to-192.patch FAIL: test commit message user tags: Mbox includes one or more GitHub-style username tags. Ensure that any "@" symbols are stripped out of usernames (test_mbox.TestMbox.test_commit_message_user_tags) PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint) PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence) PASS: test author valid (test_mbox.TestMbox.test_author_valid) PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence) PASS: test max line length (test_metadata.TestMetadata.test_max_line_length) PASS: test mbox format (test_mbox.TestMbox.test_mbox_format) PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade) PASS: test pylint (test_python_pylint.PyLint.test_pylint) PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format) PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length) PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list) SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files) SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore) SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format) SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence) SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format) SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format) SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned) SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence) SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence) SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head) SKIP: test src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.test_src_uri_left_files) SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence) --- Please address the issues identified and submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!
On Wed, 27 Nov 2024 at 17:43, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > > Please read the original rationale for picking 64, and provide benchmarks that actually show better completion times with the new setting. This was the rationale: https://git.yoctoproject.org/poky/commit/?id=c6f23f1f0fad29da4dee27a9cb8219ae05a8bfd5 Alex
diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py index c9c7a47041..354237b643 100644 --- a/meta/lib/oe/utils.py +++ b/meta/lib/oe/utils.py @@ -251,7 +251,7 @@ def trim_version(version, num_parts=2): trimmed = ".".join(parts[:num_parts]) return trimmed -def cpu_count(at_least=1, at_most=64): +def cpu_count(at_least=1, at_most=192): cpus = len(os.sched_getaffinity(0)) return max(min(cpus, at_most), at_least)