| Message ID | 20240930091015.21148-1-yoann.congal@smile.fr |
|---|---|
| State | New |
| Headers | show |
| Series | reproducibility: continue testing in case of build failure | expand |
On Mon, 2024-09-30 at 11:10 +0200, Yoann Congal via lists.openembedded.org wrote: > From: Yoann Congal <yoann.congal@smile.fr> > > The current reproducibility test stops all build tasks when a single > task fails (default BitBake behavior). This means that a single build > failure prevents the reproducibility of other packages from being > tested, which is not ideal. > > To address this, we now use the --continue option of BitBake during test > builds, allowing the build process to proceed even when some tasks fail. > The failure cases are handled as gracefully as possible. > > In the event of a build failure: > - The entire reproducibility test will be considered a failure. > - The complete BitBake log will be saved in the "saved output" directory > to facilitate debugging. On the autobuilder, this log should be > available at https://autobuilder.yocto.io/pub/repro-fail/. > - The last 20 lines of the log, which should show the failing recipes, > will be displayed in the oe-selftest console. > - If BitBake fails to create the deployment directory, it will be > manually created to allow the comparison process to proceed. > > Here is what the output looks like when testing reproducibility of bash, > a failing recipe (hello-fail) and a non-reproducible recipe > (hello-norepro): > $ rm -rf ../build-master-st <snip>/yocto/repro_output/* ; OEQA_DEBUGGING_SAVED_OUTPUT=<snip>/yocto/repro_output oe-selftest -r reproducible.ReproducibleTests > <snip> > 2024-09-30 00:04:27,485 - oe-selftest - INFO - test_reproducible_builds (reproducible.ReproducibleTests.test_reproducible_builds) > 2024-09-30 00:05:31,676 - oe-selftest - INFO - Non-reproducible packages will be copied to <snip>/yocto/repro_output/oe-reproducible-20240930-aukc_3o6 > 2024-09-30 00:05:31,676 - oe-selftest - INFO - Building reproducibleA (sstate allowed)... > 2024-09-30 00:13:08,009 - oe-selftest - INFO - Bitbake failed! but keep going... Log: > 2024-09-30 00:13:08,009 - oe-selftest - INFO - Command 'bitbake --continue bash hello-norepro hello-fail' returned non-zero exit status 1: > 2024-09-30 00:13:08,009 - oe-selftest - INFO - > 2024-09-30 00:13:08,009 - oe-selftest - INFO - ... (last 20 lines of output) > 2024-09-30 00:13:08,009 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_deb: Started > 2024-09-30 00:13:08,009 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_qa: Started > 2024-09-30 00:13:08,009 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_populate_sysroot: Succeeded > 2024-09-30 00:13:08,009 - oe-selftest - INFO - NOTE: Running setscene task 342 of 343 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_create_package_spdx_setscene) > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_package_spdx_setscene: Started > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_package_spdx_setscene: Succeeded > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: Running setscene task 343 of 343 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_create_spdx_setscene) > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_spdx_setscene: Started > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_spdx_setscene: Succeeded > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_qa: Succeeded > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_deb: Succeeded > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: Running noexec task 979 of 990 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_build) > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: Tasks Summary: Attempted 979 tasks of which 833 didn't need to be rerun and 1 failed. > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: Generating JSON CVE summary > 2024-09-30 00:13:08,010 - oe-selftest - INFO - Complete CVE JSON report summary created at: <snip>/yocto/poky/build-master-st/reproducibleA/tmp/log/cve/cve-summary.json > 2024-09-30 00:13:08,010 - oe-selftest - INFO - > 2024-09-30 00:13:08,010 - oe-selftest - INFO - Summary: 1 task failed: > 2024-09-30 00:13:08,010 - oe-selftest - INFO - <snip>/yocto/poky/meta/recipes-core/hello-single/hello-fail_1.0.bb:do_compile > 2024-09-30 00:13:08,010 - oe-selftest - INFO - log: <snip>/yocto/poky/build-master-st/reproducibleA/tmp/work/core2-64-poky-linux/hello-fail/1.0/temp/log.do_compile.2116040 > 2024-09-30 00:13:08,010 - oe-selftest - INFO - Summary: There was 1 ERROR message, returning a non-zero exit code. > 2024-09-30 00:13:08,011 - oe-selftest - INFO - reproducibleA build failed. Trying to compute built packages differences but the test will fail. > 2024-09-30 00:13:08,011 - oe-selftest - INFO - Building reproducibleB-extended (sstate NOT allowed)... > 2024-09-30 00:40:12,933 - oe-selftest - INFO - Bitbake failed! but keep going... Log: > 2024-09-30 00:40:12,933 - oe-selftest - INFO - Command 'bitbake --continue bash hello-norepro hello-fail' returned non-zero exit status 1: > 2024-09-30 00:40:12,933 - oe-selftest - INFO - > 2024-09-30 00:40:12,933 - oe-selftest - INFO - ... (last 20 lines of output) > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_deb: Started > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_qa: Started > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_spdx: Succeeded > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: Running task 978 of 990 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_create_package_spdx) > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_package_spdx: Started > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_package_spdx: Succeeded > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_ipk: Succeeded > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_qa: Succeeded > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_deb: Succeeded > 2024-09-30 00:40:12,934 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_rpm: Succeeded > 2024-09-30 00:40:12,934 - oe-selftest - INFO - NOTE: Running noexec task 979 of 990 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_build) > 2024-09-30 00:40:12,934 - oe-selftest - INFO - NOTE: Tasks Summary: Attempted 979 tasks of which 2 didn't need to be rerun and 1 failed. > 2024-09-30 00:40:12,934 - oe-selftest - INFO - NOTE: Generating JSON CVE summary > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Complete CVE JSON report summary created at: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/log/cve/cve-summary.json > 2024-09-30 00:40:12,934 - oe-selftest - INFO - > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Summary: 1 task failed: > 2024-09-30 00:40:12,934 - oe-selftest - INFO - <snip>/yocto/poky/meta/recipes-core/hello-single/hello-fail_1.0.bb:do_compile > 2024-09-30 00:40:12,934 - oe-selftest - INFO - log: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/work/core2-64-poky-linux/hello-fail/1.0/temp/log.do_compile.3007134 > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Summary: There were 5 WARNING messages. > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Summary: There was 1 ERROR message, returning a non-zero exit code. > 2024-09-30 00:40:12,934 - oe-selftest - INFO - reproducibleB-extended build failed. Trying to compute built packages differences but the test will fail. > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Checking deb packages for differences... > 2024-09-30 00:40:12,974 - oe-selftest - INFO - Reproducibility summary for deb: same=52 different=1 different_excluded=0 missing=0 total=53 > unused_exclusions=[] > 2024-09-30 00:40:12,976 - oe-selftest - INFO - Checking ipk packages for differences... > 2024-09-30 00:40:13,014 - oe-selftest - INFO - Reproducibility summary for ipk: same=52 different=1 different_excluded=0 missing=0 total=53 > unused_exclusions=[] > 2024-09-30 00:40:13,015 - oe-selftest - INFO - Checking rpm packages for differences... > 2024-09-30 00:40:13,047 - oe-selftest - INFO - Reproducibility summary for rpm: same=52 different=1 different_excluded=0 missing=0 total=53 > unused_exclusions=[] > 2024-09-30 00:40:13,048 - oe-selftest - INFO - Running diffoscope > 2024-09-30 00:40:14,051 - oe-selftest - INFO - ... FAIL > 2024-09-30 00:40:14,052 - oe-selftest - INFO - Traceback (most recent call last): > File "<snip>/yocto/poky/meta/lib/oeqa/selftest/cases/reproducible.py", line 380, in test_reproducible_builds > self.fail('\n'.join(fails)) > AssertionError: Bitbake reproducibleA failure > Bitbake reproducibleB-extended failure > The following deb packages are different and not in exclusion list: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb > The following ipk packages are different and not in exclusion list: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/hello-norepro_1.0-r0_core2-64.ipk > The following rpm packages are different and not in exclusion list: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/hello-norepro-1.0-r0.core2_64.rpm > > 2024-09-30 00:40:14,052 - oe-selftest - INFO - ====================================================================== > 2024-09-30 00:40:14,052 - oe-selftest - INFO - FAIL: test_reproducible_builds (reproducible.ReproducibleTests.test_reproducible_builds) > 2024-09-30 00:40:14,052 - oe-selftest - INFO - ---------------------------------------------------------------------- > 2024-09-30 00:40:14,052 - oe-selftest - INFO - Traceback (most recent call last): > File "<snip>/yocto/poky/meta/lib/oeqa/selftest/cases/reproducible.py", line 380, in test_reproducible_builds > self.fail('\n'.join(fails)) > AssertionError: Bitbake reproducibleA failure > Bitbake reproducibleB-extended failure > The following deb packages are different and not in exclusion list: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb > The following ipk packages are different and not in exclusion list: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/hello-norepro_1.0-r0_core2-64.ipk > The following rpm packages are different and not in exclusion list: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/hello-norepro-1.0-r0.core2_64.rpm > > 2024-09-30 00:40:14,053 - oe-selftest - INFO - ---------------------------------------------------------------------- > 2024-09-30 00:40:14,053 - oe-selftest - INFO - Ran 1 test in 2146.797s > 2024-09-30 00:40:14,053 - oe-selftest - INFO - FAILED > 2024-09-30 00:40:14,053 - oe-selftest - INFO - (failures=1) > 2024-09-30 00:40:17,620 - oe-selftest - INFO - RESULTS: > 2024-09-30 00:40:17,621 - oe-selftest - INFO - RESULTS - reproducible.ReproducibleTests.test_reproducible_builds: FAILED (2146.57s) > 2024-09-30 00:40:17,623 - oe-selftest - INFO - SUMMARY: > 2024-09-30 00:40:17,623 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 2146.798s > 2024-09-30 00:40:17,623 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=1, errors=0) > > => Test failed but hello-norepro is displayed as non-reproducible. > > The testresult.json contains: > { > "oeselftest_debian-12_qemux86-64_20240930000424": { > "configuration": { <snip> }, > "result": { > "reproducible": { > "files": { > "package_deb": { > "different": [ > { > "reference": "<snip>/yocto/poky/build-master-st/reproducibleA/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb", > "test": "<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb" > } > ], > "different_excluded": [], > "missing": [], > "same": [ > <snip> > { > "reference": "<snip>/yocto/poky/build-master-st/reproducibleA/tmp/deploy/deb/./core2-64/bash_5.2.32-r0_amd64.deb", > "test": "<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/bash_5.2.32-r0_amd64.deb" > }, > <snip> > ] > }, > "package_ipk": { <same as deb> }, > "package_rpm": { <same as deb> } > } > }, > "reproducible.ReproducibleTests.test_reproducible_builds": { > "duration": 2146.5671875476837, > "log": "Traceback (most recent call last):\n File \"<snip>/yocto/poky/meta/lib/oeqa/selftest/cases/reproducible.py\", line 380, in test_reproducible_builds\n self.fail('\\n'.join(fails))\nAssertionError: Bitbake reproducibleA failure\nBitbake reproducibleB-extended failure\nThe following deb packages are different and not in exclusion list:\n<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb\nThe following ipk packages are different and not in exclusion list:\n<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/hello-norepro_1.0-r0_core2-64.ipk\nThe following rpm packages are different and not in exclusion list:\n<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/hello-norepro-1.0-r0.core2_64.rpm\n", > "status": "FAILED" > }, > "reproducible.rawlogs": { > "log": "DIFFERENT: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb\nSAME: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/bash_5.2.32-r0_amd64.deb\n<snip>" > } > } > } > } > > => "reproducible.ReproducibleTests.test_reproducible_builds".status is > correctly "FAILED" but the reproducibility of bash and hello-norepro is > tested. > > Signed-off-by: Yoann Congal <yoann.congal@smile.fr> > --- > meta/lib/oeqa/selftest/cases/reproducible.py | 57 ++++++++++++++++---- > 1 file changed, 48 insertions(+), 9 deletions(-) > > diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py > index 3d3f30eebc..af4e53af2f 100644 > --- a/meta/lib/oeqa/selftest/cases/reproducible.py > +++ b/meta/lib/oeqa/selftest/cases/reproducible.py > @@ -215,7 +215,7 @@ class ReproducibleTests(OESelftestTestCase): > bb.utils.mkdirhier(os.path.dirname(dest)) > shutil.copyfile(source, dest) > > - def do_test_build(self, name, use_sstate): > + def do_test_build(self, name, use_sstate, output_log=None): > capture_vars = ['DEPLOY_DIR_' + c.upper() for c in self.package_classes] > > tmpdir = os.path.join(self.topdir, name, 'tmp') > @@ -234,11 +234,19 @@ class ReproducibleTests(OESelftestTestCase): > ''').format(package_classes=' '.join('package_%s' % c for c in self.package_classes), > tmpdir=tmpdir) > > + bitbake_failure_count = 0 > if not use_sstate: > if self.sstate_targets: > self.logger.info("Building prebuild for %s (sstate allowed)..." % (name)) > self.write_config(config) > - bitbake(' '.join(self.sstate_targets)) > + try: > + bitbake("--continue "+' '.join(self.sstate_targets), > + output_log=output_log, limit_exc_output=20) > + except AssertionError as e: > + bitbake_failure_count += 1 > + self.logger.info("Bitbake failed! but keep going... Log:") > + for line in str(e).split("\n"): > + self.logger.info(" "+line) > > # This config fragment will disable using shared and the sstate > # mirror, forcing a complete build from scratch > @@ -251,8 +259,21 @@ class ReproducibleTests(OESelftestTestCase): > self.write_config(config) > d = get_bb_vars(capture_vars) > # targets used to be called images > - bitbake(' '.join(getattr(self, 'images', self.targets))) > - return d > + try: > + bitbake("--continue "+' '.join(getattr(self, 'images', self.targets)), > + output_log=output_log, limit_exc_output=20) > + except AssertionError as e: > + bitbake_failure_count += 1 > + > + self.logger.info("Bitbake failed! but keep going... Log:") > + for line in str(e).split("\n"): > + self.logger.info(" "+line) > + for c in self.package_classes: > + deploy = d['DEPLOY_DIR_' + c.upper()] > + if not os.path.exists(deploy): > + self.logger.info("Manually creating %s" % deploy) > + bb.utils.mkdirhier(deploy) > + return (d, bitbake_failure_count) > > def test_reproducible_builds(self): > def strip_topdir(s): > @@ -278,15 +299,33 @@ class ReproducibleTests(OESelftestTestCase): > # https://bugzilla.yoctoproject.org/show_bug.cgi?id=15554 > # So, the reproducibleA & reproducibleB directories are changed to reproducibleA & reproducibleB-extended to have different size. > > - vars_A = self.do_test_build('reproducibleA', self.build_from_sstate) > - > - vars_B = self.do_test_build('reproducibleB-extended', False) > + fails = [] > + vars_list = [None, None] > + > + for i, (name, use_sstate) in enumerate( > + (('reproducibleA', self.build_from_sstate), > + ('reproducibleB-extended', False))): > + if self.save_results: > + output_log_path = os.path.join(save_dir, "bitbake_%s.log" % name) > + output_log = bb.msg.logger_create("bitbake_%s" % name, > + output=open(output_log_path, mode="w", encoding="utf-8"), > + preserve_handlers=True) > + (variables, bitbake_failure_count) = self.do_test_build( > + name, use_sstate, output_log=output_log) > + if bitbake_failure_count > 0: > + self.logger.info('%s build failed. Trying to compute built packages differences but the test will fail.' % name) > + fails.append("Bitbake %s failure" % name) > + else: > + if self.save_results: > + # Remove full logs if bitbake was successful > + del output_log > + os.remove(output_log_path) > + vars_list[i] = variables > > + vars_A, vars_B = vars_list > # NOTE: The temp directories from the reproducible build are purposely > # kept after the build so it can be diffed for debugging. > > - fails = [] > - > for c in self.package_classes: > with self.subTest(package_class=c): > package_class = 'package_' + c > I think the change makes sense but the code above feels a bit ugly and hard to read. Rather than having to create directories and remove directories/logs and change output paths, can we not just copy the log in case of failure? Cheers, Richard
I'm not sure I can support this. It adds a lot of difficult to understand code with the sole purpose of dancing around the build failures, I'd rather just fail the test with no attempt to salvage reproducibility comparisons and keep implmenetation simple and short. Alex On Mon, 30 Sept 2024 at 11:11, Yoann Congal via lists.openembedded.org <yoann.congal=smile.fr@lists.openembedded.org> wrote: > > From: Yoann Congal <yoann.congal@smile.fr> > > The current reproducibility test stops all build tasks when a single > task fails (default BitBake behavior). This means that a single build > failure prevents the reproducibility of other packages from being > tested, which is not ideal. > > To address this, we now use the --continue option of BitBake during test > builds, allowing the build process to proceed even when some tasks fail. > The failure cases are handled as gracefully as possible. > > In the event of a build failure: > - The entire reproducibility test will be considered a failure. > - The complete BitBake log will be saved in the "saved output" directory > to facilitate debugging. On the autobuilder, this log should be > available at https://autobuilder.yocto.io/pub/repro-fail/. > - The last 20 lines of the log, which should show the failing recipes, > will be displayed in the oe-selftest console. > - If BitBake fails to create the deployment directory, it will be > manually created to allow the comparison process to proceed. > > Here is what the output looks like when testing reproducibility of bash, > a failing recipe (hello-fail) and a non-reproducible recipe > (hello-norepro): > $ rm -rf ../build-master-st <snip>/yocto/repro_output/* ; OEQA_DEBUGGING_SAVED_OUTPUT=<snip>/yocto/repro_output oe-selftest -r reproducible.ReproducibleTests > <snip> > 2024-09-30 00:04:27,485 - oe-selftest - INFO - test_reproducible_builds (reproducible.ReproducibleTests.test_reproducible_builds) > 2024-09-30 00:05:31,676 - oe-selftest - INFO - Non-reproducible packages will be copied to <snip>/yocto/repro_output/oe-reproducible-20240930-aukc_3o6 > 2024-09-30 00:05:31,676 - oe-selftest - INFO - Building reproducibleA (sstate allowed)... > 2024-09-30 00:13:08,009 - oe-selftest - INFO - Bitbake failed! but keep going... Log: > 2024-09-30 00:13:08,009 - oe-selftest - INFO - Command 'bitbake --continue bash hello-norepro hello-fail' returned non-zero exit status 1: > 2024-09-30 00:13:08,009 - oe-selftest - INFO - > 2024-09-30 00:13:08,009 - oe-selftest - INFO - ... (last 20 lines of output) > 2024-09-30 00:13:08,009 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_deb: Started > 2024-09-30 00:13:08,009 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_qa: Started > 2024-09-30 00:13:08,009 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_populate_sysroot: Succeeded > 2024-09-30 00:13:08,009 - oe-selftest - INFO - NOTE: Running setscene task 342 of 343 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_create_package_spdx_setscene) > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_package_spdx_setscene: Started > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_package_spdx_setscene: Succeeded > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: Running setscene task 343 of 343 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_create_spdx_setscene) > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_spdx_setscene: Started > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_spdx_setscene: Succeeded > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_qa: Succeeded > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_deb: Succeeded > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: Running noexec task 979 of 990 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_build) > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: Tasks Summary: Attempted 979 tasks of which 833 didn't need to be rerun and 1 failed. > 2024-09-30 00:13:08,010 - oe-selftest - INFO - NOTE: Generating JSON CVE summary > 2024-09-30 00:13:08,010 - oe-selftest - INFO - Complete CVE JSON report summary created at: <snip>/yocto/poky/build-master-st/reproducibleA/tmp/log/cve/cve-summary.json > 2024-09-30 00:13:08,010 - oe-selftest - INFO - > 2024-09-30 00:13:08,010 - oe-selftest - INFO - Summary: 1 task failed: > 2024-09-30 00:13:08,010 - oe-selftest - INFO - <snip>/yocto/poky/meta/recipes-core/hello-single/hello-fail_1.0.bb:do_compile > 2024-09-30 00:13:08,010 - oe-selftest - INFO - log: <snip>/yocto/poky/build-master-st/reproducibleA/tmp/work/core2-64-poky-linux/hello-fail/1.0/temp/log.do_compile.2116040 > 2024-09-30 00:13:08,010 - oe-selftest - INFO - Summary: There was 1 ERROR message, returning a non-zero exit code. > 2024-09-30 00:13:08,011 - oe-selftest - INFO - reproducibleA build failed. Trying to compute built packages differences but the test will fail. > 2024-09-30 00:13:08,011 - oe-selftest - INFO - Building reproducibleB-extended (sstate NOT allowed)... > 2024-09-30 00:40:12,933 - oe-selftest - INFO - Bitbake failed! but keep going... Log: > 2024-09-30 00:40:12,933 - oe-selftest - INFO - Command 'bitbake --continue bash hello-norepro hello-fail' returned non-zero exit status 1: > 2024-09-30 00:40:12,933 - oe-selftest - INFO - > 2024-09-30 00:40:12,933 - oe-selftest - INFO - ... (last 20 lines of output) > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_deb: Started > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_qa: Started > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_spdx: Succeeded > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: Running task 978 of 990 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_create_package_spdx) > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_package_spdx: Started > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_create_package_spdx: Succeeded > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_ipk: Succeeded > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_qa: Succeeded > 2024-09-30 00:40:12,933 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_deb: Succeeded > 2024-09-30 00:40:12,934 - oe-selftest - INFO - NOTE: recipe bash-5.2.32-r0: task do_package_write_rpm: Succeeded > 2024-09-30 00:40:12,934 - oe-selftest - INFO - NOTE: Running noexec task 979 of 990 (<snip>/yocto/poky/meta/recipes-extended/bash/bash_5.2.32.bb:do_build) > 2024-09-30 00:40:12,934 - oe-selftest - INFO - NOTE: Tasks Summary: Attempted 979 tasks of which 2 didn't need to be rerun and 1 failed. > 2024-09-30 00:40:12,934 - oe-selftest - INFO - NOTE: Generating JSON CVE summary > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Complete CVE JSON report summary created at: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/log/cve/cve-summary.json > 2024-09-30 00:40:12,934 - oe-selftest - INFO - > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Summary: 1 task failed: > 2024-09-30 00:40:12,934 - oe-selftest - INFO - <snip>/yocto/poky/meta/recipes-core/hello-single/hello-fail_1.0.bb:do_compile > 2024-09-30 00:40:12,934 - oe-selftest - INFO - log: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/work/core2-64-poky-linux/hello-fail/1.0/temp/log.do_compile.3007134 > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Summary: There were 5 WARNING messages. > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Summary: There was 1 ERROR message, returning a non-zero exit code. > 2024-09-30 00:40:12,934 - oe-selftest - INFO - reproducibleB-extended build failed. Trying to compute built packages differences but the test will fail. > 2024-09-30 00:40:12,934 - oe-selftest - INFO - Checking deb packages for differences... > 2024-09-30 00:40:12,974 - oe-selftest - INFO - Reproducibility summary for deb: same=52 different=1 different_excluded=0 missing=0 total=53 > unused_exclusions=[] > 2024-09-30 00:40:12,976 - oe-selftest - INFO - Checking ipk packages for differences... > 2024-09-30 00:40:13,014 - oe-selftest - INFO - Reproducibility summary for ipk: same=52 different=1 different_excluded=0 missing=0 total=53 > unused_exclusions=[] > 2024-09-30 00:40:13,015 - oe-selftest - INFO - Checking rpm packages for differences... > 2024-09-30 00:40:13,047 - oe-selftest - INFO - Reproducibility summary for rpm: same=52 different=1 different_excluded=0 missing=0 total=53 > unused_exclusions=[] > 2024-09-30 00:40:13,048 - oe-selftest - INFO - Running diffoscope > 2024-09-30 00:40:14,051 - oe-selftest - INFO - ... FAIL > 2024-09-30 00:40:14,052 - oe-selftest - INFO - Traceback (most recent call last): > File "<snip>/yocto/poky/meta/lib/oeqa/selftest/cases/reproducible.py", line 380, in test_reproducible_builds > self.fail('\n'.join(fails)) > AssertionError: Bitbake reproducibleA failure > Bitbake reproducibleB-extended failure > The following deb packages are different and not in exclusion list: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb > The following ipk packages are different and not in exclusion list: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/hello-norepro_1.0-r0_core2-64.ipk > The following rpm packages are different and not in exclusion list: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/hello-norepro-1.0-r0.core2_64.rpm > > 2024-09-30 00:40:14,052 - oe-selftest - INFO - ====================================================================== > 2024-09-30 00:40:14,052 - oe-selftest - INFO - FAIL: test_reproducible_builds (reproducible.ReproducibleTests.test_reproducible_builds) > 2024-09-30 00:40:14,052 - oe-selftest - INFO - ---------------------------------------------------------------------- > 2024-09-30 00:40:14,052 - oe-selftest - INFO - Traceback (most recent call last): > File "<snip>/yocto/poky/meta/lib/oeqa/selftest/cases/reproducible.py", line 380, in test_reproducible_builds > self.fail('\n'.join(fails)) > AssertionError: Bitbake reproducibleA failure > Bitbake reproducibleB-extended failure > The following deb packages are different and not in exclusion list: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb > The following ipk packages are different and not in exclusion list: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/hello-norepro_1.0-r0_core2-64.ipk > The following rpm packages are different and not in exclusion list: > <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/hello-norepro-1.0-r0.core2_64.rpm > > 2024-09-30 00:40:14,053 - oe-selftest - INFO - ---------------------------------------------------------------------- > 2024-09-30 00:40:14,053 - oe-selftest - INFO - Ran 1 test in 2146.797s > 2024-09-30 00:40:14,053 - oe-selftest - INFO - FAILED > 2024-09-30 00:40:14,053 - oe-selftest - INFO - (failures=1) > 2024-09-30 00:40:17,620 - oe-selftest - INFO - RESULTS: > 2024-09-30 00:40:17,621 - oe-selftest - INFO - RESULTS - reproducible.ReproducibleTests.test_reproducible_builds: FAILED (2146.57s) > 2024-09-30 00:40:17,623 - oe-selftest - INFO - SUMMARY: > 2024-09-30 00:40:17,623 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 2146.798s > 2024-09-30 00:40:17,623 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=1, errors=0) > > => Test failed but hello-norepro is displayed as non-reproducible. > > The testresult.json contains: > { > "oeselftest_debian-12_qemux86-64_20240930000424": { > "configuration": { <snip> }, > "result": { > "reproducible": { > "files": { > "package_deb": { > "different": [ > { > "reference": "<snip>/yocto/poky/build-master-st/reproducibleA/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb", > "test": "<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb" > } > ], > "different_excluded": [], > "missing": [], > "same": [ > <snip> > { > "reference": "<snip>/yocto/poky/build-master-st/reproducibleA/tmp/deploy/deb/./core2-64/bash_5.2.32-r0_amd64.deb", > "test": "<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/bash_5.2.32-r0_amd64.deb" > }, > <snip> > ] > }, > "package_ipk": { <same as deb> }, > "package_rpm": { <same as deb> } > } > }, > "reproducible.ReproducibleTests.test_reproducible_builds": { > "duration": 2146.5671875476837, > "log": "Traceback (most recent call last):\n File \"<snip>/yocto/poky/meta/lib/oeqa/selftest/cases/reproducible.py\", line 380, in test_reproducible_builds\n self.fail('\\n'.join(fails))\nAssertionError: Bitbake reproducibleA failure\nBitbake reproducibleB-extended failure\nThe following deb packages are different and not in exclusion list:\n<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb\nThe following ipk packages are different and not in exclusion list:\n<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/hello-norepro_1.0-r0_core2-64.ipk\nThe following rpm packages are different and not in exclusion list:\n<snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/hello-norepro-1.0-r0.core2_64.rpm\n", > "status": "FAILED" > }, > "reproducible.rawlogs": { > "log": "DIFFERENT: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/hello-norepro_1.0-r0_amd64.deb\nSAME: <snip>/yocto/poky/build-master-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/bash_5.2.32-r0_amd64.deb\n<snip>" > } > } > } > } > > => "reproducible.ReproducibleTests.test_reproducible_builds".status is > correctly "FAILED" but the reproducibility of bash and hello-norepro is > tested. > > Signed-off-by: Yoann Congal <yoann.congal@smile.fr> > --- > meta/lib/oeqa/selftest/cases/reproducible.py | 57 ++++++++++++++++---- > 1 file changed, 48 insertions(+), 9 deletions(-) > > diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py > index 3d3f30eebc..af4e53af2f 100644 > --- a/meta/lib/oeqa/selftest/cases/reproducible.py > +++ b/meta/lib/oeqa/selftest/cases/reproducible.py > @@ -215,7 +215,7 @@ class ReproducibleTests(OESelftestTestCase): > bb.utils.mkdirhier(os.path.dirname(dest)) > shutil.copyfile(source, dest) > > - def do_test_build(self, name, use_sstate): > + def do_test_build(self, name, use_sstate, output_log=None): > capture_vars = ['DEPLOY_DIR_' + c.upper() for c in self.package_classes] > > tmpdir = os.path.join(self.topdir, name, 'tmp') > @@ -234,11 +234,19 @@ class ReproducibleTests(OESelftestTestCase): > ''').format(package_classes=' '.join('package_%s' % c for c in self.package_classes), > tmpdir=tmpdir) > > + bitbake_failure_count = 0 > if not use_sstate: > if self.sstate_targets: > self.logger.info("Building prebuild for %s (sstate allowed)..." % (name)) > self.write_config(config) > - bitbake(' '.join(self.sstate_targets)) > + try: > + bitbake("--continue "+' '.join(self.sstate_targets), > + output_log=output_log, limit_exc_output=20) > + except AssertionError as e: > + bitbake_failure_count += 1 > + self.logger.info("Bitbake failed! but keep going... Log:") > + for line in str(e).split("\n"): > + self.logger.info(" "+line) > > # This config fragment will disable using shared and the sstate > # mirror, forcing a complete build from scratch > @@ -251,8 +259,21 @@ class ReproducibleTests(OESelftestTestCase): > self.write_config(config) > d = get_bb_vars(capture_vars) > # targets used to be called images > - bitbake(' '.join(getattr(self, 'images', self.targets))) > - return d > + try: > + bitbake("--continue "+' '.join(getattr(self, 'images', self.targets)), > + output_log=output_log, limit_exc_output=20) > + except AssertionError as e: > + bitbake_failure_count += 1 > + > + self.logger.info("Bitbake failed! but keep going... Log:") > + for line in str(e).split("\n"): > + self.logger.info(" "+line) > + for c in self.package_classes: > + deploy = d['DEPLOY_DIR_' + c.upper()] > + if not os.path.exists(deploy): > + self.logger.info("Manually creating %s" % deploy) > + bb.utils.mkdirhier(deploy) > + return (d, bitbake_failure_count) > > def test_reproducible_builds(self): > def strip_topdir(s): > @@ -278,15 +299,33 @@ class ReproducibleTests(OESelftestTestCase): > # https://bugzilla.yoctoproject.org/show_bug.cgi?id=15554 > # So, the reproducibleA & reproducibleB directories are changed to reproducibleA & reproducibleB-extended to have different size. > > - vars_A = self.do_test_build('reproducibleA', self.build_from_sstate) > - > - vars_B = self.do_test_build('reproducibleB-extended', False) > + fails = [] > + vars_list = [None, None] > + > + for i, (name, use_sstate) in enumerate( > + (('reproducibleA', self.build_from_sstate), > + ('reproducibleB-extended', False))): > + if self.save_results: > + output_log_path = os.path.join(save_dir, "bitbake_%s.log" % name) > + output_log = bb.msg.logger_create("bitbake_%s" % name, > + output=open(output_log_path, mode="w", encoding="utf-8"), > + preserve_handlers=True) > + (variables, bitbake_failure_count) = self.do_test_build( > + name, use_sstate, output_log=output_log) > + if bitbake_failure_count > 0: > + self.logger.info('%s build failed. Trying to compute built packages differences but the test will fail.' % name) > + fails.append("Bitbake %s failure" % name) > + else: > + if self.save_results: > + # Remove full logs if bitbake was successful > + del output_log > + os.remove(output_log_path) > + vars_list[i] = variables > > + vars_A, vars_B = vars_list > # NOTE: The temp directories from the reproducible build are purposely > # kept after the build so it can be diffed for debugging. > > - fails = [] > - > for c in self.package_classes: > with self.subTest(package_class=c): > package_class = 'package_' + c > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#205101): https://lists.openembedded.org/g/openembedded-core/message/205101 > Mute This Topic: https://lists.openembedded.org/mt/108732482/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Le 30/09/2024 à 11:20, Alexander Kanavin a écrit : > I'm not sure I can support this. It adds a lot of difficult to > understand code with the sole purpose of dancing around the build > failures, I'd rather just fail the test with no attempt to salvage > reproducibility comparisons and keep implmenetation simple and short. The problem I'm trying to solve here is the reproducibility of meta-openembedded. While oe-core world is always buildable, this is, sadly, not the case for meta-openembedded layers. In order to move forward with its reproducibility, we need to know the reproducibility state of as much packages as possible. Do you think that goal is worth making the code more complex? Do you have a suggestion on how to approach this? Making (and keeping) meta-oe build every time is the goal, but history shows that it is hard [0]. [0]: https://autobuilder.yoctoproject.org/typhoon/#/builders/155 Thanks!
On Mon, 30 Sept 2024 at 16:49, Yoann Congal <yoann.congal@smile.fr> wrote: > The problem I'm trying to solve here is the reproducibility of meta-openembedded. While oe-core world is always buildable, this is, sadly, not the case for meta-openembedded layers. In order to move forward with its reproducibility, we need to know the reproducibility state of as much packages as possible. > > Do you think that goal is worth making the code more complex? Do you have a suggestion on how to approach this? Making (and keeping) meta-oe build every time is the goal, but history shows that it is hard [0]. > [0]: https://autobuilder.yoctoproject.org/typhoon/#/builders/155 The real problem is that meta-oe isn't built in lockstep with oe-core changes, and regressions slip in all the time, and are fixed after the fact through heroic efforts of Khem and others. Maybe the new AB setup will allow including meta-oe builds into a-full? Alex
On Tue, 2024-10-01 at 11:36 +0200, Alexander Kanavin wrote: > On Mon, 30 Sept 2024 at 16:49, Yoann Congal <yoann.congal@smile.fr> > wrote: > > The problem I'm trying to solve here is the reproducibility of > > meta-openembedded. While oe-core world is always buildable, this > > is, sadly, not the case for meta-openembedded layers. In order to > > move forward with its reproducibility, we need to know the > > reproducibility state of as much packages as possible. > > > > Do you think that goal is worth making the code more complex? Do > > you have a suggestion on how to approach this? Making (and keeping) > > meta-oe build every time is the goal, but history shows that it is > > hard [0]. > > [0]: https://autobuilder.yoctoproject.org/typhoon/#/builders/155 > > The real problem is that meta-oe isn't built in lockstep with oe-core > changes, and regressions slip in all the time, and are fixed after > the > fact through heroic efforts of Khem and others. > > Maybe the new AB setup will allow including meta-oe builds into a- > full? That pushes the problem to me and the core maintainers as we then have to block until meta-oe is fixed as well with all the tracking and push back on patches that involves. Is that really what we want to do? Cheers, Richard
On Tue, 1 Oct 2024 at 11:38, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > > The problem I'm trying to solve here is the reproducibility of > > > meta-openembedded. While oe-core world is always buildable, this > > > is, sadly, not the case for meta-openembedded layers. In order to > > > move forward with its reproducibility, we need to know the > > > reproducibility state of as much packages as possible. > > > > > > Do you think that goal is worth making the code more complex? Do > > > you have a suggestion on how to approach this? Making (and keeping) > > > meta-oe build every time is the goal, but history shows that it is > > > hard [0]. > > > [0]: https://autobuilder.yoctoproject.org/typhoon/#/builders/155 > > > > The real problem is that meta-oe isn't built in lockstep with oe-core > > changes, and regressions slip in all the time, and are fixed after > > the > > fact through heroic efforts of Khem and others. > > > > Maybe the new AB setup will allow including meta-oe builds into a- > > full? > > That pushes the problem to me and the core maintainers as we then have > to block until meta-oe is fixed as well with all the tracking and push > back on patches that involves. Is that really what we want to do? I'm ok with handling a failing build, if v2 is possible and looks more maintainable than v1. Alex
Le 01/10/2024 à 11:46, Alexander Kanavin a écrit : > On Tue, 1 Oct 2024 at 11:38, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: >>>> The problem I'm trying to solve here is the reproducibility of >>>> meta-openembedded. While oe-core world is always buildable, this >>>> is, sadly, not the case for meta-openembedded layers. In order to >>>> move forward with its reproducibility, we need to know the >>>> reproducibility state of as much packages as possible. >>>> >>>> Do you think that goal is worth making the code more complex? Do >>>> you have a suggestion on how to approach this? Making (and keeping) >>>> meta-oe build every time is the goal, but history shows that it is >>>> hard [0]. >>>> [0]: https://autobuilder.yoctoproject.org/typhoon/#/builders/155 >>> >>> The real problem is that meta-oe isn't built in lockstep with oe-core >>> changes, and regressions slip in all the time, and are fixed after >>> the >>> fact through heroic efforts of Khem and others. >>> >>> Maybe the new AB setup will allow including meta-oe builds into a- >>> full? >> >> That pushes the problem to me and the core maintainers as we then have >> to block until meta-oe is fixed as well with all the tracking and push >> back on patches that involves. Is that really what we want to do? > > I'm ok with handling a failing build, if v2 is possible and looks more > maintainable than v1. I'm working on a v2 with Richard's suggestion about the log file handling. I have no problem in taking any other suggestion on how to improve maintainability and keep improving this patch. Thanks!
diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py index 3d3f30eebc..af4e53af2f 100644 --- a/meta/lib/oeqa/selftest/cases/reproducible.py +++ b/meta/lib/oeqa/selftest/cases/reproducible.py @@ -215,7 +215,7 @@ class ReproducibleTests(OESelftestTestCase): bb.utils.mkdirhier(os.path.dirname(dest)) shutil.copyfile(source, dest) - def do_test_build(self, name, use_sstate): + def do_test_build(self, name, use_sstate, output_log=None): capture_vars = ['DEPLOY_DIR_' + c.upper() for c in self.package_classes] tmpdir = os.path.join(self.topdir, name, 'tmp') @@ -234,11 +234,19 @@ class ReproducibleTests(OESelftestTestCase): ''').format(package_classes=' '.join('package_%s' % c for c in self.package_classes), tmpdir=tmpdir) + bitbake_failure_count = 0 if not use_sstate: if self.sstate_targets: self.logger.info("Building prebuild for %s (sstate allowed)..." % (name)) self.write_config(config) - bitbake(' '.join(self.sstate_targets)) + try: + bitbake("--continue "+' '.join(self.sstate_targets), + output_log=output_log, limit_exc_output=20) + except AssertionError as e: + bitbake_failure_count += 1 + self.logger.info("Bitbake failed! but keep going... Log:") + for line in str(e).split("\n"): + self.logger.info(" "+line) # This config fragment will disable using shared and the sstate # mirror, forcing a complete build from scratch @@ -251,8 +259,21 @@ class ReproducibleTests(OESelftestTestCase): self.write_config(config) d = get_bb_vars(capture_vars) # targets used to be called images - bitbake(' '.join(getattr(self, 'images', self.targets))) - return d + try: + bitbake("--continue "+' '.join(getattr(self, 'images', self.targets)), + output_log=output_log, limit_exc_output=20) + except AssertionError as e: + bitbake_failure_count += 1 + + self.logger.info("Bitbake failed! but keep going... Log:") + for line in str(e).split("\n"): + self.logger.info(" "+line) + for c in self.package_classes: + deploy = d['DEPLOY_DIR_' + c.upper()] + if not os.path.exists(deploy): + self.logger.info("Manually creating %s" % deploy) + bb.utils.mkdirhier(deploy) + return (d, bitbake_failure_count) def test_reproducible_builds(self): def strip_topdir(s): @@ -278,15 +299,33 @@ class ReproducibleTests(OESelftestTestCase): # https://bugzilla.yoctoproject.org/show_bug.cgi?id=15554 # So, the reproducibleA & reproducibleB directories are changed to reproducibleA & reproducibleB-extended to have different size. - vars_A = self.do_test_build('reproducibleA', self.build_from_sstate) - - vars_B = self.do_test_build('reproducibleB-extended', False) + fails = [] + vars_list = [None, None] + + for i, (name, use_sstate) in enumerate( + (('reproducibleA', self.build_from_sstate), + ('reproducibleB-extended', False))): + if self.save_results: + output_log_path = os.path.join(save_dir, "bitbake_%s.log" % name) + output_log = bb.msg.logger_create("bitbake_%s" % name, + output=open(output_log_path, mode="w", encoding="utf-8"), + preserve_handlers=True) + (variables, bitbake_failure_count) = self.do_test_build( + name, use_sstate, output_log=output_log) + if bitbake_failure_count > 0: + self.logger.info('%s build failed. Trying to compute built packages differences but the test will fail.' % name) + fails.append("Bitbake %s failure" % name) + else: + if self.save_results: + # Remove full logs if bitbake was successful + del output_log + os.remove(output_log_path) + vars_list[i] = variables + vars_A, vars_B = vars_list # NOTE: The temp directories from the reproducible build are purposely # kept after the build so it can be diffed for debugging. - fails = [] - for c in self.package_classes: with self.subTest(package_class=c): package_class = 'package_' + c