| Message ID | 20251208223118.374387-2-t.f.g.geelen@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [auh,1/2] testimage: update package installation logic to use groups | expand |
Hi Alex, This one I would like your opinion on. I have enabled a testimage in my local AUH configuration but when upgrading something the message is always that 'qmp' cannot be found. I adjusted auh now such that this is detected from the image passed via 'testimage' in the conf file. However it could also be that I'm overlooking something, in that case apologies. Kind regards, Tom On Mon, Dec 8, 2025 at 11:31 PM Tom Geelen <t.f.g.geelen@gmail.com> wrote: > > When defining a testimage it will try to run an image via Qemu and need qmp. QMP is assumed to be provided. However this throws an error. Use the RECIPE_SYSROOT_NATIVE of the testimage and set up the paths using this. > > Signed-off-by: Tom Geelen <t.f.g.geelen@gmail.com> > --- > modules/testimage.py | 38 +++++++++++++++++++++++++++++++++++++- > 1 file changed, 37 insertions(+), 1 deletion(-) > > diff --git a/modules/testimage.py b/modules/testimage.py > index 849a9f1..0a0f924 100644 > --- a/modules/testimage.py > +++ b/modules/testimage.py > @@ -73,10 +73,46 @@ class TestImage(): > return ' '.join(pkgs_out) > > def testimage(self, groups, machine, image): > + # Minimal: derive native sysroot from the image's environment and expose it via PYTHONPATH. > + try: > + # Use the image-specific environment; this matches what succeeds outside AUH > + bb_env = self.bb.env(image) > + rsn = bb_env.get('RECIPE_SYSROOT_NATIVE') > + if rsn: > + I(f"AUH: RECIPE_SYSROOT_NATIVE (image)={rsn}") > + pythonpaths = [] > + libdir = os.path.join(rsn, 'usr', 'lib') > + if os.path.isdir(libdir): > + for entry in os.listdir(libdir): > + if entry.startswith('python'): > + sp = os.path.join(libdir, entry, 'site-packages') > + if os.path.isdir(sp): > + pythonpaths.append(sp) > + qp = os.path.join(libdir, 'qemu-python') > + if os.path.isdir(qp): > + pythonpaths.append(qp) > + > + if pythonpaths: > + new_py = os.pathsep.join(pythonpaths) > + os.environ['OEQA_PYTHONPATH_NATIVE'] = new_py > + os.environ['PYTHONPATH'] = new_py + os.pathsep + os.environ.get('PYTHONPATH', '') > + os.environ['BB_ENV_PASSTHROUGH_ADDITIONS'] = os.environ.get('BB_ENV_PASSTHROUGH_ADDITIONS', '') + ' OEQA_PYTHONPATH_NATIVE PYTHONPATH' > + # Ensure oeqa picks up changes during task execution > + os.environ['TESTIMAGE_UPDATE_VARS'] = (os.environ.get('TESTIMAGE_UPDATE_VARS', '') + ' OEQA_PYTHONPATH_NATIVE PYTHONPATH').strip() > + I(f"AUH: OEQA_PYTHONPATH_NATIVE set to: {new_py}") > + else: > + W('Native sysroot found but no python paths; qmp import may fail') > + else: > + W('RECIPE_SYSROOT_NATIVE not present; ensure qemu-native/qemu-system-native are available') > + except Exception as ex: > + W('Failed to configure PYTHONPATH for qmp: {}'.format(ex)) > + > os.environ['CORE_IMAGE_EXTRA_INSTALL'] = \ > self._get_pkgs_to_install(groups) > os.environ['TEST_LOG_DIR'] = self.logdir > - os.environ['TESTIMAGE_UPDATE_VARS'] = 'TEST_LOG_DIR' > + # Keep TEST_LOG_DIR propagating alongside any previously set vars > + prev_update = os.environ.get('TESTIMAGE_UPDATE_VARS', '') > + os.environ['TESTIMAGE_UPDATE_VARS'] = (prev_update + ' TEST_LOG_DIR').strip() > I( " Installing additional packages to the image: {}".format(os.environ['CORE_IMAGE_EXTRA_INSTALL'])) > > I( " building %s for %s ..." % (image, machine)) > -- > 2.43.0 >
On Mon, 8 Dec 2025 at 23:40, Tom Geelen <t.f.g.geelen@gmail.com> wrote: > This one I would like your opinion on. I have enabled a testimage in > my local AUH configuration but when upgrading something the message is > always that 'qmp' cannot be found. I adjusted auh now such that this > is detected from the image passed via 'testimage' in the conf file. > However it could also be that I'm overlooking something, in that case apologies. Can you provide a simple way to reproduce and observe this? It looks like a complicated workaround for an issue that is not fully understood. This particular feature of AUH is not used in the autobuilder jobs, and AUH has no tests of its own, so it's prone to regressions. And if I didn't inherit the code from somewhere else, I would just throw it all away and do a clean rewrite (possibly as a devtool plugin/command). It barely works :) Alex
Hi Alex,
Certainly, here is an example which triggers the problem on my end.
My upgrade-helper.conf file:
[maintainer_override]
[settings]
from=t.f.g.geelen@gmail.com
maintainers_whitelist=t.f.g.geelen@gmail.com tim.orling@konsulko.com
machines=qemux86-64
commit_revert_policy=failed_to_build
testimage=yes
testimage_name=core-image-test
The core-image-test which is referenced is here:
https://github.com/Gerry546/meta-sanctum/blob/main/meta-sanctum/recipes-images/images/core-image-test.bb
It is a minimal image to run ptest on a qemu target.
I noticed this issue when trying to upgrade python3-attrs currently as
I need the latest version. You will need to remove this from the
recipe though as the recipe itself fails to compile after the upgrade
as this file is no longer needed. I use master on all layers
do_install_ptest:append() {
install ${S}/conftest.py ${D}${PTEST_PATH}/
}
This is the output I see in my testimage.log
Summary: 1 task failed:
/home/tom/projects/meta-sanctum/build-homeassistant/layers/../../meta-sanctum/recipes-images/images/core-image-test.bb:do_testimage
log: /home/tom/projects/meta-sanctum/build-homeassistant/build/tmp/work/qemux86_64-oe-linux/core-image-test/1.0/temp/log.do_testimage.624190
Summary: There were 2 ERROR messages, returning a non-zero exit code.
ERROR: core-image-test-1.0-r0 do_testimage: qemurunner: qmp module
missing, please ensure it's installed in
/home/tom/projects/meta-sanctum/build-homeassistant/build/upgrade-helper/20251209222910/recipe-sysroot-native/usr/lib/qemu-python
(No module named 'qmp')
ERROR: core-image-test-1.0-r0 do_testimage: Error executing a python
function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_testimage(d)
0003:
File: '/home/tom/projects/meta-sanctum/build-homeassistant/layers/../../sources/openembedded-core/meta/classes-recipe/testimage.bbclass',
lineno: 124, function: do_testimage
0120: dump-guest-memory {"paging":false,"protocol":"file:%s.img"}
0121:}
0122:
0123:python do_testimage() {
*** 0124: testimage_main(d)
0125:}
0126:
0127:addtask testimage
0128:do_testimage[nostamp] = "1"
File: '/home/tom/projects/meta-sanctum/build-homeassistant/layers/../../sources/openembedded-core/meta/classes-recipe/testimage.bbclass',
lineno: 376, function: testimage_main
0372: orig_sigterm_handler = signal.signal(signal.SIGTERM,
sigterm_exception)
0373: try:
0374: # We need to check if runqemu ends unexpectedly
0375: # or if the worker send us a SIGTERM
*** 0376: tc.target.start(params=d.getVar("TEST_QEMUPARAMS"),
runqemuparams=d.getVar("TEST_RUNQEMUPARAMS"))
0377: import threading
0378: try:
0379:
threading.Timer(int(d.getVar("TEST_OVERALL_TIMEOUT")),
handle_test_timeout, (int(d.getVar("TEST_OVERALL_TIMEOUT")),)).start()
0380: except ValueError:
File: '/home/tom/projects/meta-sanctum/build-homeassistant/layers/../../sources/openembedded-core/meta/lib/oeqa/core/target/qemu.py',
lineno: 91, function: start
0087: except (subprocess.CalledProcessError,
subprocess.TimeoutExpired, FileNotFoundError) as err:
0088: msg += "Error running command: %s\n%s\n" %
(blcmd, err)
0089: msg += "\n\n===== end: snippet =====\n"
0090:
*** 0091: raise RuntimeError("FAILED to start qemu - check
the task log and the boot log %s" % (msg))
0092:
0093: def stop(self):
0094: self.runner.stop()
Exception: RuntimeError: FAILED to start qemu - check the task log and
the boot log
And this is the log of the upgrade-helper:
Reconnecting to bitbake server...
Retrying server connection (#1)... (22:29:11.267662)
Reconnecting to bitbake server...
Retrying server connection (#1)... (22:29:11.267662)
Starting bitbake server...
########### The list of recipes to be upgraded #############
, python3-attrs, 25.3.0, 25.4.0, Tim Orling <tim.orling@konsulko.com>, N/A
############################################################
Building gcc runtimes ...
building gcc runtime for qemux86-64
ATTEMPT PACKAGE GROUP 1/1
python3-attrs: Upgrading to 25.4.0
python3-attrs: Loading environment ...
python3-attrs: Running 'devtool upgrade' ...
python3-attrs: Running 'devtool finish' ...
python3-attrs: compiling upgraded version for qemux86-64 ...
python3-attrs: Upgrade SUCCESSFUL! Please test!
python3-attrs: Auto commit changes ...
python3-attrs: Save patch in directory:
/home/tom/projects/meta-sanctum/build-homeassistant/build/upgrade-helper/20251209222910/all/python3-attrs.
Testing image for qemux86-64 ...
Checking if package python3-attrs has ptests...
...yes
Installing additional packages to the image: python3-attrs python3-attrs-ptest
building core-image-test for qemux86-64 ...
building the testimage failed! Collecting logs...
All done! Testimage/ptest/qemu logs are collected to
/home/tom/projects/meta-sanctum/build-homeassistant/build/upgrade-helper/20251209222910/testimage-logs
Generating work tarball in
/home/tom/projects/meta-sanctum/build-homeassistant/build/upgrade-helper/20251209222910.tar.gz
...
Recipe upgrade statistics:
* Succeeded: 1
python3-attrs, 25.4.0, Tim Orling <tim.orling@konsulko.com>
TOTAL: attempted=1 succeeded=1(100.00%) failed=0(0.00%)
Recipe upgrade statistics per Maintainer:
Tim Orling <tim.orling: attempted=1 succeeded=1(100.00%) failed=0(0.00%)
Let me know whenever you are planning on doing an AUH upgrade.. I use
AUH actually quite heavily for my work in meta-homeassistant as it
saves me a tremendous amount of time and I would love to help with
this tool. It has a lot of potential.
Regards,
Tom
On Tue, Dec 9, 2025 at 10:02 AM Alexander Kanavin
<alex.kanavin@gmail.com> wrote:
>
> On Mon, 8 Dec 2025 at 23:40, Tom Geelen <t.f.g.geelen@gmail.com> wrote:
> > This one I would like your opinion on. I have enabled a testimage in
> > my local AUH configuration but when upgrading something the message is
> > always that 'qmp' cannot be found. I adjusted auh now such that this
> > is detected from the image passed via 'testimage' in the conf file.
> > However it could also be that I'm overlooking something, in that case apologies.
>
> Can you provide a simple way to reproduce and observe this? It looks
> like a complicated workaround for an issue that is not fully
> understood.
>
> This particular feature of AUH is not used in the autobuilder jobs,
> and AUH has no tests of its own, so it's prone to regressions. And if
> I didn't inherit the code from somewhere else, I would just throw it
> all away and do a clean rewrite (possibly as a devtool
> plugin/command). It barely works :)
>
> Alex
On Tue, 9 Dec 2025 at 23:06, Tom Geelen <t.f.g.geelen@gmail.com> wrote: > ERROR: core-image-test-1.0-r0 do_testimage: qemurunner: qmp module > missing, please ensure it's installed in > /home/tom/projects/meta-sanctum/build-homeassistant/build/upgrade-helper/20251209222910/recipe-sysroot-native/usr/lib/qemu-python > (No module named 'qmp') Thanks, this is the key part that I wanted to see. The path is obviously wrong, so I went and checked: 1. meta/lib/oeqa/utils/qemurunner.py has this bit of code: # use logfile to determine the recipe-sysroot-native path and # then add in the site-packages path components and add that # to the python sys.path so the qmp module can be found. python_path = os.path.dirname(os.path.dirname(self.logfile)) python_path += "/recipe-sysroot-native/usr/lib/qemu-python" sys.path.append(python_path) importlib.invalidate_caches() try: qmp = importlib.import_module("qmp") except Exception as e: self.logger.error("qemurunner: qmp module missing, please ensure it's installed in %s (%s)" % (python_path, str(e))) return False 2. logfile is passed into the class from TEST_LOG_DIR variable, which in testimage.bbclass defaults to TEST_LOG_DIR ?= "${WORKDIR}/testimage" 3. AUH on the other hand sets TEST_LOG_DIR to something else, specifically it's own working directory. I'd say in this situation AUH is right, and the code in point 1 is wrong. It needs to figure out the path to qmp module directly from WORKDIR, but I don't know yet which of the many parameters to QemuRunner's constructor and launch() method can be used. Can you look into it? You can tweak the code to print them all, and see if something suitable is in there. > Let me know whenever you are planning on doing an AUH upgrade.. I use > AUH actually quite heavily for my work in meta-homeassistant as it > saves me a tremendous amount of time and I would love to help with > this tool. It has a lot of potential. My interest is ensuring that the twice-a-month autobuilder job that upgrades all of oe-core continues to run. So it's basically just patching things up when they break: https://git.yoctoproject.org/auto-upgrade-helper/log/ I don't have a 'roadmap' for the tool or any clear idea about how to rewrite it (vaguely, I think the upgrade/build/test loop should be a devtool sub-command: this would get rid of the horrible heuristics in AUH that guess what happened from bitbake and devtool human-readable output, and AUH's business then would be mostly figuring out what needs to be upgraded, and then sending the resulting patches). But if you have ideas for improvements that don't require such a drastic refactor, that'd be welcome too! Alex
diff --git a/modules/testimage.py b/modules/testimage.py index 849a9f1..0a0f924 100644 --- a/modules/testimage.py +++ b/modules/testimage.py @@ -73,10 +73,46 @@ class TestImage(): return ' '.join(pkgs_out) def testimage(self, groups, machine, image): + # Minimal: derive native sysroot from the image's environment and expose it via PYTHONPATH. + try: + # Use the image-specific environment; this matches what succeeds outside AUH + bb_env = self.bb.env(image) + rsn = bb_env.get('RECIPE_SYSROOT_NATIVE') + if rsn: + I(f"AUH: RECIPE_SYSROOT_NATIVE (image)={rsn}") + pythonpaths = [] + libdir = os.path.join(rsn, 'usr', 'lib') + if os.path.isdir(libdir): + for entry in os.listdir(libdir): + if entry.startswith('python'): + sp = os.path.join(libdir, entry, 'site-packages') + if os.path.isdir(sp): + pythonpaths.append(sp) + qp = os.path.join(libdir, 'qemu-python') + if os.path.isdir(qp): + pythonpaths.append(qp) + + if pythonpaths: + new_py = os.pathsep.join(pythonpaths) + os.environ['OEQA_PYTHONPATH_NATIVE'] = new_py + os.environ['PYTHONPATH'] = new_py + os.pathsep + os.environ.get('PYTHONPATH', '') + os.environ['BB_ENV_PASSTHROUGH_ADDITIONS'] = os.environ.get('BB_ENV_PASSTHROUGH_ADDITIONS', '') + ' OEQA_PYTHONPATH_NATIVE PYTHONPATH' + # Ensure oeqa picks up changes during task execution + os.environ['TESTIMAGE_UPDATE_VARS'] = (os.environ.get('TESTIMAGE_UPDATE_VARS', '') + ' OEQA_PYTHONPATH_NATIVE PYTHONPATH').strip() + I(f"AUH: OEQA_PYTHONPATH_NATIVE set to: {new_py}") + else: + W('Native sysroot found but no python paths; qmp import may fail') + else: + W('RECIPE_SYSROOT_NATIVE not present; ensure qemu-native/qemu-system-native are available') + except Exception as ex: + W('Failed to configure PYTHONPATH for qmp: {}'.format(ex)) + os.environ['CORE_IMAGE_EXTRA_INSTALL'] = \ self._get_pkgs_to_install(groups) os.environ['TEST_LOG_DIR'] = self.logdir - os.environ['TESTIMAGE_UPDATE_VARS'] = 'TEST_LOG_DIR' + # Keep TEST_LOG_DIR propagating alongside any previously set vars + prev_update = os.environ.get('TESTIMAGE_UPDATE_VARS', '') + os.environ['TESTIMAGE_UPDATE_VARS'] = (prev_update + ' TEST_LOG_DIR').strip() I( " Installing additional packages to the image: {}".format(os.environ['CORE_IMAGE_EXTRA_INSTALL'])) I( " building %s for %s ..." % (image, machine))
When defining a testimage it will try to run an image via Qemu and need qmp. QMP is assumed to be provided. However this throws an error. Use the RECIPE_SYSROOT_NATIVE of the testimage and set up the paths using this. Signed-off-by: Tom Geelen <t.f.g.geelen@gmail.com> --- modules/testimage.py | 38 +++++++++++++++++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-)