mbox series

[v4,0/3] pkg-database and systemd-sysext image

Message ID 20240304061553.2281009-1-johannes.schneider@leica-geosystems.com
Headers show
Series pkg-database and systemd-sysext image | expand

Message

SCHNEIDER Johannes March 4, 2024, 6:15 a.m. UTC 
systemd-sysext allows to overlay another image (or multiple) ontop of
a "base-image" = the current rootfs, via the use of overlayfs; to add
tools and features meant for development purposes.

To quote the documentation on systemd-sysext:
" ...addition in order to make debugging/development easier). System
extension images should not be misunderstood as a generic software
packaging framework, ..."

To build a lean image, that only holds packages that are not already
part of the base-image, a snapshot of the package-database is taken
after the installation of the base-rootfs is done, and picked up again
when collecting the rootfs of such a extension image.

with all this in place an example usage could look like this:
some-core-image.bb
  inherit core-image
  IMAGE_GEN_PKGDBFS = "1"

extending-image.bb
  inherit image-sysext
  IMAGE_FSTYPES = "squashfs"
  IMAGE_BASE_PKGDB = "some-core-image"
  # the above pointing at a package-db similar to:
  # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz

then on the device, running some-core-image, with the extension image placed at FN:
$> ln -s "$FN" /run/extensions/$(basename $FN).raw
$> systemd-sysext list
$> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge

As long as the VERSION_ID of the extension image matches the os-release
in the base image, the above commands return sucessfully;
for details on the compativility check see the docs for systemd-sysext.

=========

changes with v2:
        rebase from 'kirkstone' onto 'master'

changes with v3;
	incorporate review suggestions for simplification
	add task dependency handling
	add oe-selftest for the pkgdb handling
	add variable documentation and
	some more comments, and examples in the commit-msg

changes with v4;
	rebase onto 'master' => no functional changes
	fixed patchtest findings

Comments

Alexander Kanavin March 4, 2024, 9:25 a.m. UTC | #1 
There is a feature freeze now, so this might have to wait for after that.

Alex

On Mon, 4 Mar 2024 at 07:16, Johannes Schneider via
lists.openembedded.org
<johannes.schneider=leica-geosystems.com@lists.openembedded.org>
wrote:
>
> systemd-sysext allows to overlay another image (or multiple) ontop of
> a "base-image" = the current rootfs, via the use of overlayfs; to add
> tools and features meant for development purposes.
>
> To quote the documentation on systemd-sysext:
> " ...addition in order to make debugging/development easier). System
> extension images should not be misunderstood as a generic software
> packaging framework, ..."
>
> To build a lean image, that only holds packages that are not already
> part of the base-image, a snapshot of the package-database is taken
> after the installation of the base-rootfs is done, and picked up again
> when collecting the rootfs of such a extension image.
>
> with all this in place an example usage could look like this:
> some-core-image.bb
>   inherit core-image
>   IMAGE_GEN_PKGDBFS = "1"
>
> extending-image.bb
>   inherit image-sysext
>   IMAGE_FSTYPES = "squashfs"
>   IMAGE_BASE_PKGDB = "some-core-image"
>   # the above pointing at a package-db similar to:
>   # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz
>
> then on the device, running some-core-image, with the extension image placed at FN:
> $> ln -s "$FN" /run/extensions/$(basename $FN).raw
> $> systemd-sysext list
> $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge
>
> As long as the VERSION_ID of the extension image matches the os-release
> in the base image, the above commands return sucessfully;
> for details on the compativility check see the docs for systemd-sysext.
>
> =========
>
> changes with v2:
>         rebase from 'kirkstone' onto 'master'
>
> changes with v3;
>         incorporate review suggestions for simplification
>         add task dependency handling
>         add oe-selftest for the pkgdb handling
>         add variable documentation and
>         some more comments, and examples in the commit-msg
>
> changes with v4;
>         rebase onto 'master' => no functional changes
>         fixed patchtest findings
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#196588): https://lists.openembedded.org/g/openembedded-core/message/196588
> Mute This Topic: https://lists.openembedded.org/mt/104717642/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
SCHNEIDER Johannes March 4, 2024, 9:51 a.m. UTC | #2 
Hoi Alex,


thanks for the heads up - i'll patiently wait then :-)


gruß
Johannes
Alexandre Belloni March 23, 2024, 1:40 p.m. UTC | #3 
Hello,

The series causes failures on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6548/steps/14/logs/stdio


2024-03-21 21:05:49,848 - oe-selftest - INFO - 6: 20/46 500/587 (137.73s) (0 failed) (imagefeatures.ImageFeatures.test_image_gen_pkgdbfs)
2024-03-21 21:05:49,848 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/imagefeatures.py", line 327, in test_image_gen_pkgdbfs
    self.assertTrue(os.path.exists(os.path.join(img_vars['DEPLOY_DIR_IMAGE'], 'var/lib/opkg/status')), 'opkg\'s status file was not present in: %s' % tar_file)
  File "/usr/lib64/python3.9/unittest/case.py", line 688, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : opkg's status file was not present in: /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-3332915/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs-pkgdb.tar.gz


On 04/03/2024 07:15:50+0100, Johannes Schneider via lists.openembedded.org wrote:
> systemd-sysext allows to overlay another image (or multiple) ontop of
> a "base-image" = the current rootfs, via the use of overlayfs; to add
> tools and features meant for development purposes.
> 
> To quote the documentation on systemd-sysext:
> " ...addition in order to make debugging/development easier). System
> extension images should not be misunderstood as a generic software
> packaging framework, ..."
> 
> To build a lean image, that only holds packages that are not already
> part of the base-image, a snapshot of the package-database is taken
> after the installation of the base-rootfs is done, and picked up again
> when collecting the rootfs of such a extension image.
> 
> with all this in place an example usage could look like this:
> some-core-image.bb
>   inherit core-image
>   IMAGE_GEN_PKGDBFS = "1"
> 
> extending-image.bb
>   inherit image-sysext
>   IMAGE_FSTYPES = "squashfs"
>   IMAGE_BASE_PKGDB = "some-core-image"
>   # the above pointing at a package-db similar to:
>   # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz
> 
> then on the device, running some-core-image, with the extension image placed at FN:
> $> ln -s "$FN" /run/extensions/$(basename $FN).raw
> $> systemd-sysext list
> $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge
> 
> As long as the VERSION_ID of the extension image matches the os-release
> in the base image, the above commands return sucessfully;
> for details on the compativility check see the docs for systemd-sysext.
> 
> =========
> 
> changes with v2:
>         rebase from 'kirkstone' onto 'master'
> 
> changes with v3;
> 	incorporate review suggestions for simplification
> 	add task dependency handling
> 	add oe-selftest for the pkgdb handling
> 	add variable documentation and
> 	some more comments, and examples in the commit-msg
> 
> changes with v4;
> 	rebase onto 'master' => no functional changes
> 	fixed patchtest findings

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#196588): https://lists.openembedded.org/g/openembedded-core/message/196588
> Mute This Topic: https://lists.openembedded.org/mt/104717642/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
SCHNEIDER Johannes March 25, 2024, 12:42 p.m. UTC | #4 
Hoi,


thanks for pointing the build failure out,

Im trying to reproduce it locally but fail so far to see the problem :-S
rebased on the latest master, when the two selftests added with the patches are run selecively, they pass:
#> oe-selftest -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs imagefeatures.ImageFeatures.test_image_use_pkgdbfs

but running the whole test-routine, same way the buildserver does in that log, currently fails me for other reasons (lack of resources? or i might be calling it the wrong way)

any hints/ideas/pointers would be appreciated :-)


gruß
Johannes