mbox series

[v9,0/3] pkg-database and systemd-sysext image

Message ID 20240515223500.3464486-1-johannes.schneider@leica-geosystems.com
Headers show
Series pkg-database and systemd-sysext image | expand

Message

Johannes Schneider May 15, 2024, 10:34 p.m. UTC 
systemd-sysext allows to overlay another image (or multiple) ontop of
a "base-image" = the current rootfs, via the use of overlayfs; to add
tools and features meant for development purposes.

To quote the documentation on systemd-sysext:
" ...addition in order to make debugging/development easier). System
extension images should not be misunderstood as a generic software
packaging framework, ..."

To build a lean image, that only holds packages that are not already
part of the base-image, a snapshot of the package-database is taken
after the installation of the base-rootfs is done, and picked up again
when collecting the rootfs of such a extension image.

with all this in place an example usage could look like this:
some-core-image.bb
  inherit core-image
  IMAGE_GEN_PKGDBFS = "1"

extending-image.bb
  inherit image-sysext
  IMAGE_FSTYPES = "squashfs"
  IMAGE_BASE_PKGDB = "some-core-image"
  # the above pointing at a package-db similar to:
  # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz

then on the device, running some-core-image, with the extension image placed at FN:
$> ln -s "$FN" /run/extensions/$(basename $FN).raw
$> systemd-sysext list
$> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge

As long as the VERSION_ID of the extension image matches the os-release
in the base image, the above commands return sucessfully;
for details on the compativility check see the docs for systemd-sysext.

=========

changes with v2:
        rebase from 'kirkstone' onto 'master'

changes with v3:
	incorporate review suggestions for simplification
	add task dependency handling
	add oe-selftest for the pkgdb handling
	add variable documentation and
	some more comments, and examples in the commit-msg

changes with v4:
	rebase onto 'master' => no functional changes
	fixed patchtest findings

changes with v5:
        rebase onto 'master'
	add '.sysext' to the deployed symlink name
        sidenote on the tests and autobuilder failure: run locally they succeed, e.g.:
          #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs

changes with v6:
	tests: restructure to call 'bitbake' only once in the testcase itself
	  (in hopes of solving the autobuilder problem; local test runs succeed)

changes with v7:
        tests: undo the restructuring of done in v6, in favour of explicitly forcing steps:
	  '-c rootfs -f'.  since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue?

changes with v8:
        tests: fixed missing feature=PACKAGE_CLASSES during 'test_image_gen_pkgdbfs'
	   which is now split into three, one for each of the three available packagemanager: ipk, deb, rpm
	   
changes with v9:
	tests: fix singular failure in 'oe-selftest-armhost'
	   which was due to the built image not containing a 'kernel-image...', which the gen-pkgdb sanity
	   check expected, but 'packagegroup-core-boot' instead

Comments

Alexandre Belloni May 17, 2024, 6:33 p.m. UTC | #1 
Hello,

deb and rpm seem successful but ipk still fails:

https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3335/steps/14/logs/stdio

On 16/05/2024 00:34:57+0200, Johannes Schneider via lists.openembedded.org wrote:
> systemd-sysext allows to overlay another image (or multiple) ontop of
> a "base-image" = the current rootfs, via the use of overlayfs; to add
> tools and features meant for development purposes.
> 
> To quote the documentation on systemd-sysext:
> " ...addition in order to make debugging/development easier). System
> extension images should not be misunderstood as a generic software
> packaging framework, ..."
> 
> To build a lean image, that only holds packages that are not already
> part of the base-image, a snapshot of the package-database is taken
> after the installation of the base-rootfs is done, and picked up again
> when collecting the rootfs of such a extension image.
> 
> with all this in place an example usage could look like this:
> some-core-image.bb
>   inherit core-image
>   IMAGE_GEN_PKGDBFS = "1"
> 
> extending-image.bb
>   inherit image-sysext
>   IMAGE_FSTYPES = "squashfs"
>   IMAGE_BASE_PKGDB = "some-core-image"
>   # the above pointing at a package-db similar to:
>   # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz
> 
> then on the device, running some-core-image, with the extension image placed at FN:
> $> ln -s "$FN" /run/extensions/$(basename $FN).raw
> $> systemd-sysext list
> $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge
> 
> As long as the VERSION_ID of the extension image matches the os-release
> in the base image, the above commands return sucessfully;
> for details on the compativility check see the docs for systemd-sysext.
> 
> =========
> 
> changes with v2:
>         rebase from 'kirkstone' onto 'master'
> 
> changes with v3:
> 	incorporate review suggestions for simplification
> 	add task dependency handling
> 	add oe-selftest for the pkgdb handling
> 	add variable documentation and
> 	some more comments, and examples in the commit-msg
> 
> changes with v4:
> 	rebase onto 'master' => no functional changes
> 	fixed patchtest findings
> 
> changes with v5:
>         rebase onto 'master'
> 	add '.sysext' to the deployed symlink name
>         sidenote on the tests and autobuilder failure: run locally they succeed, e.g.:
>           #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs
> 
> changes with v6:
> 	tests: restructure to call 'bitbake' only once in the testcase itself
> 	  (in hopes of solving the autobuilder problem; local test runs succeed)
> 
> changes with v7:
>         tests: undo the restructuring of done in v6, in favour of explicitly forcing steps:
> 	  '-c rootfs -f'.  since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue?
> 
> changes with v8:
>         tests: fixed missing feature=PACKAGE_CLASSES during 'test_image_gen_pkgdbfs'
> 	   which is now split into three, one for each of the three available packagemanager: ipk, deb, rpm
> 	   
> changes with v9:
> 	tests: fix singular failure in 'oe-selftest-armhost'
> 	   which was due to the built image not containing a 'kernel-image...', which the gen-pkgdb sanity
> 	   check expected, but 'packagegroup-core-boot' instead

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#199430): https://lists.openembedded.org/g/openembedded-core/message/199430
> Mute This Topic: https://lists.openembedded.org/mt/106124365/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Johannes Schneider May 18, 2024, 6:30 a.m. UTC | #2 
Hoi Alexandre,


is there a way to get more output out of the autobuilder?
the logs only tell "that" the test failed, not "why" = which assertion was missed

if i run the test locally with the differing variables set like this:
#> SDKMACHINE=aarch64 MACHINE=qemuarm64 oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk

it succeeds :-S
... ideas/thoughts?


gruß
Johannes
Alexander Kanavin May 21, 2024, 10:08 a.m. UTC | #3 
If you scroll further up, there's:

2024-05-16 22:05:57,101 - oe-selftest - INFO - FAIL:
imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk
(subunit.RemotedTestCase)
2024-05-16 22:05:57,102 - oe-selftest - INFO -
----------------------------------------------------------------------
2024-05-16 22:05:57,102 - oe-selftest - INFO -
testtools.testresult.real._StringException: Traceback (most recent
call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/imagefeatures.py",
line 344, in test_image_gen_pkgdbfs_ipk
    result = runCmd(cmd)
  File "/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/utils/commands.py",
line 212, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status
%d:\n%s" % (command, result.status, exc_output))
AssertionError: Command
'/home/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-4137794/tmp/sysroots/aarch64/usr/bin/opkg
--volatile-cache
--offline-root=/home/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-4137794/tmp/deploy/images/qemuarm64
--add-arch qemuarm64:11 --add-arch qemuarm64:12 list-installed'
returned non-zero exit status 255:


Also note that this happened only on the arm host, and not on x86
hosts. Which does make it tricky to reproduce, but at least gives a
lead:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6921

Alex

On Sat, 18 May 2024 at 08:30, SCHNEIDER Johannes
<johannes.schneider@leica-geosystems.com> wrote:
>
> Hoi Alexandre,
>
>
> is there a way to get more output out of the autobuilder?
> the logs only tell "that" the test failed, not "why" = which assertion was missed
>
> if i run the test locally with the differing variables set like this:
> #> SDKMACHINE=aarch64 MACHINE=qemuarm64 oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk
>
> it succeeds :-S
> ... ideas/thoughts?
>
>
> gruß
> Johannes
>
> ________________________________________
> From: Alexandre Belloni <alexandre.belloni@bootlin.com>
> Sent: Friday, May 17, 2024 20:33
> To: SCHNEIDER Johannes
> Cc: openembedded-core@lists.openembedded.org; richard.purdie@linuxfoundation.org; alex.kanavin@gmail.com
> Subject: Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
>
> This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email.
>
>
> Hello,
>
> deb and rpm seem successful but ipk still fails:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3335/steps/14/logs/stdio
>
> On 16/05/2024 00:34:57+0200, Johannes Schneider via lists.openembedded.org wrote:
> > systemd-sysext allows to overlay another image (or multiple) ontop of
> > a "base-image" = the current rootfs, via the use of overlayfs; to add
> > tools and features meant for development purposes.
> >
> > To quote the documentation on systemd-sysext:
> > " ...addition in order to make debugging/development easier). System
> > extension images should not be misunderstood as a generic software
> > packaging framework, ..."
> >
> > To build a lean image, that only holds packages that are not already
> > part of the base-image, a snapshot of the package-database is taken
> > after the installation of the base-rootfs is done, and picked up again
> > when collecting the rootfs of such a extension image.
> >
> > with all this in place an example usage could look like this:
> > some-core-image.bb
> >   inherit core-image
> >   IMAGE_GEN_PKGDBFS = "1"
> >
> > extending-image.bb
> >   inherit image-sysext
> >   IMAGE_FSTYPES = "squashfs"
> >   IMAGE_BASE_PKGDB = "some-core-image"
> >   # the above pointing at a package-db similar to:
> >   # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz
> >
> > then on the device, running some-core-image, with the extension image placed at FN:
> > $> ln -s "$FN" /run/extensions/$(basename $FN).raw
> > $> systemd-sysext list
> > $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge
> >
> > As long as the VERSION_ID of the extension image matches the os-release
> > in the base image, the above commands return sucessfully;
> > for details on the compativility check see the docs for systemd-sysext.
> >
> > =========
> >
> > changes with v2:
> >         rebase from 'kirkstone' onto 'master'
> >
> > changes with v3:
> >       incorporate review suggestions for simplification
> >       add task dependency handling
> >       add oe-selftest for the pkgdb handling
> >       add variable documentation and
> >       some more comments, and examples in the commit-msg
> >
> > changes with v4:
> >       rebase onto 'master' => no functional changes
> >       fixed patchtest findings
> >
> > changes with v5:
> >         rebase onto 'master'
> >       add '.sysext' to the deployed symlink name
> >         sidenote on the tests and autobuilder failure: run locally they succeed, e.g.:
> >           #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs
> >
> > changes with v6:
> >       tests: restructure to call 'bitbake' only once in the testcase itself
> >         (in hopes of solving the autobuilder problem; local test runs succeed)
> >
> > changes with v7:
> >         tests: undo the restructuring of done in v6, in favour of explicitly forcing steps:
> >         '-c rootfs -f'.  since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue?
> >
> > changes with v8:
> >         tests: fixed missing feature=PACKAGE_CLASSES during 'test_image_gen_pkgdbfs'
> >          which is now split into three, one for each of the three available packagemanager: ipk, deb, rpm
> >
> > changes with v9:
> >       tests: fix singular failure in 'oe-selftest-armhost'
> >          which was due to the built image not containing a 'kernel-image...', which the gen-pkgdb sanity
> >          check expected, but 'packagegroup-core-boot' instead
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#199430): https://lists.openembedded.org/g/openembedded-core/message/199430
> > Mute This Topic: https://lists.openembedded.org/mt/106124365/3617179
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com/
Johannes Schneider May 22, 2024, 10:59 a.m. UTC | #4 
Hoi,


>  If you scroll further up, there's:

ah... stupid me relied on the search function in the webui (which caches only part of the output...) instead of looking at/searching through the raw log; sorry!

> Also note that this happened only on the arm host, and not on x86
hosts. Which does make it tricky to reproduce, but at least gives a
lead:

curiously if i use an "actual arm host" = a debian arm64 schroot (see below),
building and running the tests in there also succeeds
-> so my current lead is that there is sth "wrong" with the qemu disk-image :-S

i'll have to pick apart how the yocto-autobuilder builds itself qemu-arm, a disk image and then runs the tests inside of that... convoluted :-D

since this most likely ends up in a special case/exception inside the test; would excluding that machine+test be an option?


gruß
Johannes


debian-arm schroot:
#> apt install qemu-user-static
#> mkdir -p debian-arm64/usr/bin/; cp /usr/bin/qemu-arm-static debian-arm64/usr/bin/
#> debootstrap --arch=arm64 bookworm debian-arm64 http://deb.debian.org/debian/
#chroot> apt install -y gawk wget git diffstat unzip texinfo gcc build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 python3-subunit zstd liblz4-tool file locales libacl1
#chroot /tmp/openembedded-core/build> SDKMACHINE=aarch64 MACHINE=qemuarm64 PACKAGE_CLASSES=package_rpm oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk
<SNIP>
2024-05-22 05:15:52,357 - oe-selftest - DEBUG - running cmd: /tmp/openembedded-core/build-st/tmp-glibc/sysroots/aarch64/usr/bin/opkg --volatile-cache --offline-root=/tmp/openembedded-core/build-st/tmp-glibc/deploy/images/qemuarm64 --add-arch qemuarm64:11 --add-arch qemuarm64:12 list-installed
<SNIP>
2024-05-22 05:16:06,765 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)