Message ID | EERs.1719320900453164786.rlUz@lists.yoctoproject.org (mailing list archive) |
---|---|
State | New |
Headers | show |
Series | execute pkg_postinst_ontarget when read-only-rootfs-delayed-postinsts is set for a read only rootfs | expand |
I’m not exactly sure, but is one supposed to specify one or the other, but not both at the same time? Alex On Tue 25. Jun 2024 at 15.08, Gassner, Tobias.ext via lists.yoctoproject.org <Tobias.Gassner.ext=karlstorz.com@lists.yoctoproject.org> wrote: > Hi, > > we specify read-only-rootfs and use an overlayFS to* "make the roofts > writable"*. We also want to execute a postinstall script for our package > on the target: > > pkg_postinst_ontarget:${PN}() { > > #some code... > } > > in order for bitbake to not complain that a read-only rootfs and > pkg_postinst_ontarget don't work together, we also specify > read-only-rootfs-delayed-postinsts > <https://docs.yoctoproject.org/dev/ref-manual/features.html> > > However, it seems that the run-postinsts package is removed by the _uninstall_unneeded > function (in file meta/lib/oe/rootfs.py), because the image_rorfs flag > only checks for read-only-rootfs and not for read-only-rootfs-delayed-postinsts > as well. Hence, the postinst code is not executed. > > When I use the fix at the end of this post, the behavior is as expected. > However, I am not an expert in this area of the project and want to collect > your feedback first, before proposing to merge. > > PS: a pointer to read-only-rootfs-delayed-postinsts in the docu > <https://docs.yoctoproject.org/dev/dev-manual/read-only-rootfs.html#post-installation-scripts-and-read-only-root-filesystem> would > be very helpful. > > > Best regards > -Tobias > > > > From 608050984c5be67f506a0877fadbbe0b61575e54 Mon Sep 17 00:00:00 2001 > From: "Gassner, Tobias.ext" <tobias.gassner.ext@karlstorz.com> > Date: Tue, 25 Jun 2024 13:30:17 +0200 > Subject: [PATCH] keep delayed-postinsts when > read-only-rootfs-delayed-postinsts is set for a read only rootfs > > Signed-off-by: Gassner, Tobias.ext <tobias.gassner.ext@karlstorz.com> > --- > meta/lib/oe/rootfs.py | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py > index 8cd48f9450..5abce4ad7d 100644 > --- a/meta/lib/oe/rootfs.py > +++ b/meta/lib/oe/rootfs.py > @@ -269,7 +269,11 @@ class Rootfs(object, metaclass=ABCMeta): > self.pm.remove(["run-postinsts"]) > > image_rorfs = bb.utils.contains("IMAGE_FEATURES", > "read-only-rootfs", > + True, False, self.d) and \ > + not bb.utils.contains("IMAGE_FEATURES", > + > "read-only-rootfs-delayed-postinsts", > True, False, self.d) > + > image_rorfs_force = self.d.getVar('FORCE_RO_REMOVE') > > if image_rorfs or image_rorfs_force == "1": > > > -- > 2.45.2 > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#13391): > https://lists.yoctoproject.org/g/poky/message/13391 > Mute This Topic: https://lists.yoctoproject.org/mt/106869018/1686489 > Group Owner: poky+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/poky/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
Hi, (sent again, to post on group as well) the docu ( https://docs.yoctoproject.org/dev/ref-manual/features.html ) on read-only-rootfs-delayed-postinsts says: > > > > read-only-rootfs-delayed-postinsts: when specified in *conjunction* with read-only-rootfs > , [...] > > > so I guess I have to specify both. Greetings -Tobias
Right. I know nothing about it, I would only suggest to grep the poky tree for read-only-rootfs-delayed-postinsts to see how it's implemented, and whether tests exist for it (and if they do, then how do they set things up). Then you have the full picture. Alex On Tue, 25 Jun 2024 at 15:43, Gassner, Tobias.ext via lists.yoctoproject.org <Tobias.Gassner.ext=karlstorz.com@lists.yoctoproject.org> wrote: > > Hi, > > (sent again, to post on group as well) > > the docu on read-only-rootfs-delayed-postinsts says: > > read-only-rootfs-delayed-postinsts: when specified in conjunction with read-only-rootfs, [...] > > so I guess I have to specify both. > > Greetings > -Tobias > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#13393): https://lists.yoctoproject.org/g/poky/message/13393 > Mute This Topic: https://lists.yoctoproject.org/mt/106869018/1686489 > Group Owner: poky+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/poky/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hi Alex, > > I would only suggest to grep the poky tree for > read-only-rootfs-delayed-postinsts to see how it's implemented > already did before my first post. Basically, the only place in all of poky where this feature is used is in rootfs.py to turn off a bb.fatal when you have a delayed postinstall script and a read-only rootfs. This is also what the doc says > > read-only-rootfs-delayed-postinsts: when specified in conjunction with read-only-rootfs > , specifies that post-install scripts are still permitted (this assumes > that the root filesystem will be made writeable for the first boot; *this > feature does not do anything to ensure that - it just disables the check > for post-install scripts* ) The _uninstall_unneeded method in rootfs.py removes packages with postinstall steps if read-only-rootfs is set. However, as I suggested in my patch, the check should also take into account the read-only-rootfs-delayed-postinsts feature, which allows postinstall code even for read-only rootfs.
On Wed, 26 Jun 2024 at 10:03, Gassner, Tobias.ext via lists.yoctoproject.org <Tobias.Gassner.ext=karlstorz.com@lists.yoctoproject.org> wrote: > read-only-rootfs-delayed-postinsts: when specified in conjunction with read-only-rootfs, specifies that post-install scripts are still permitted (this assumes that the root filesystem will be made writeable for the first boot; this feature does not do anything to ensure that - it just disables the check for post-install scripts) > > The _uninstall_unneeded method in rootfs.py removes packages with postinstall steps if read-only-rootfs is set. However, as I suggested in my patch, the check should also take into account the read-only-rootfs-delayed-postinsts feature, which allows postinstall code even for read-only rootfs. It's possible this used to work when it was first added, then regressed via lack of tests. Can you check if this can be added/modified in meta/lib/oeqa/selftest/cases/buildoptions.py or meta/lib/oeqa/selftest/cases/overlayfs.py ? Alex
Hi Alex, I wrote a test in meta/lib/oeqa/selftest/cases/buildoptions.py that fails without my patch and passes with my patch. Please review if the test as specified makes sense to you: From e125ef8504a83cedd014d275387a16f519880eb5 Mon Sep 17 00:00:00 2001 From: "Gassner, Tobias.ext" <tobias.gassner.ext@karlstorz.com> Date: Tue, 2 Jul 2024 14:53:04 +0200 Subject: [PATCH] Test that pkg_postinst_ontarget task is executed on read-only-rootfs, when read-only-rootfs-delayed-postinsts is specified Signed-off-by: Gassner, Tobias.ext <tobias.gassner.ext@karlstorz.com> --- meta/lib/oeqa/selftest/cases/buildoptions.py | 39 +++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/meta/lib/oeqa/selftest/cases/buildoptions.py b/meta/lib/oeqa/selftest/cases/buildoptions.py index 31dafaa9c5..f166de89dd 100644 --- a/meta/lib/oeqa/selftest/cases/buildoptions.py +++ b/meta/lib/oeqa/selftest/cases/buildoptions.py @@ -12,7 +12,7 @@ import tempfile from oeqa.selftest.case import OESelftestTestCase from oeqa.selftest.cases.buildhistory import BuildhistoryBase from oeqa.core.decorator.data import skipIfMachine -from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars +from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars, runqemu import oeqa.utils.ftools as ftools from oeqa.core.decorator import OETestTag @@ -59,6 +59,43 @@ class ImageOptionsTests(OESelftestTestCase): bitbake("core-image-sato core-image-weston") # do_image will fail if there are any pending postinsts + def test_postinst_ontarget_executes_on_read_only_image(self): + # Test checks that pkg_postinst_ontarget task is executed even + # for read-only-rootfs, when read-only-rootfs-delayed-postinsts + # is specified as well + + # Define a recipe that touches a file in its postinst ontarget step + emptytest_inc = """ +SUMMARY = "summary" +DESCRIPTION = "description" +LICENSE:forcevariable = "CLOSED" + +ALLOW_EMPTY:${PN} = "1" + +inherit allarch + +pkg_postinst_ontarget:${PN} () { + touch /tmp/ro_rootfs_postinst_ontarget +} +""" + + self.write_recipeinc('emptytest', emptytest_inc) + + # Make rootfs read-only, but allow postinst ontarget task + self.write_config('IMAGE_FEATURES += "read-only-rootfs"') + self.append_config('IMAGE_FEATURES += "read-only-rootfs-delayed-postinsts"') + + self.append_config('IMAGE_INSTALL:append = " emptytest"') + + self.append_config('INIT_MANAGER = "systemd"') + + bitbake("core-image-minimal") + + with runqemu('core-image-minimal') as qemu: + # Check that the file, touched in postinst ontarget task exists + status, output = qemu.run_serial("[ -f /tmp/ro_rootfs_postinst_ontarget ] && echo 1 || echo 0") + self.assertTrue("1" in output, msg=output) + class DiskMonTest(OESelftestTestCase): def test_stoptask_behavior(self): -- 2.45.2
Hello Tobias, the test is basically ok, but I looked a bit further, and I think there should be improvements: - there is already meta-selftest/recipes-test/postinst/postinst_1.0.bb for the purpose of testing postinsts, the test should be using that instead of writing out its own recipe - that recipe is used in meta/lib/oeqa/selftest/cases/runtime_test.py so I think that's actually the correct place for the test, specifically in Postinst class. Alex On Tue, 2 Jul 2024 at 15:08, Gassner, Tobias.ext via lists.yoctoproject.org <Tobias.Gassner.ext=karlstorz.com@lists.yoctoproject.org> wrote: > > Hi Alex, > > I wrote a test in meta/lib/oeqa/selftest/cases/buildoptions.py that fails without my patch and passes with my patch. Please review if the test as specified makes sense to you: > > From e125ef8504a83cedd014d275387a16f519880eb5 Mon Sep 17 00:00:00 2001 > From: "Gassner, Tobias.ext" <tobias.gassner.ext@karlstorz.com> > Date: Tue, 2 Jul 2024 14:53:04 +0200 > Subject: [PATCH] Test that pkg_postinst_ontarget task is executed on > read-only-rootfs, when read-only-rootfs-delayed-postinsts is specified > > Signed-off-by: Gassner, Tobias.ext <tobias.gassner.ext@karlstorz.com> > --- > meta/lib/oeqa/selftest/cases/buildoptions.py | 39 +++++++++++++++++++- > 1 file changed, 38 insertions(+), 1 deletion(-) > > diff --git a/meta/lib/oeqa/selftest/cases/buildoptions.py b/meta/lib/oeqa/selftest/cases/buildoptions.py > index 31dafaa9c5..f166de89dd 100644 > --- a/meta/lib/oeqa/selftest/cases/buildoptions.py > +++ b/meta/lib/oeqa/selftest/cases/buildoptions.py > @@ -12,7 +12,7 @@ import tempfile > from oeqa.selftest.case import OESelftestTestCase > from oeqa.selftest.cases.buildhistory import BuildhistoryBase > from oeqa.core.decorator.data import skipIfMachine > -from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars > +from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars, runqemu > import oeqa.utils.ftools as ftools > from oeqa.core.decorator import OETestTag > > @@ -59,6 +59,43 @@ class ImageOptionsTests(OESelftestTestCase): > bitbake("core-image-sato core-image-weston") > # do_image will fail if there are any pending postinsts > > + def test_postinst_ontarget_executes_on_read_only_image(self): > + # Test checks that pkg_postinst_ontarget task is executed even > + # for read-only-rootfs, when read-only-rootfs-delayed-postinsts > + # is specified as well > + > + # Define a recipe that touches a file in its postinst ontarget step > + emptytest_inc = """ > +SUMMARY = "summary" > +DESCRIPTION = "description" > +LICENSE:forcevariable = "CLOSED" > + > +ALLOW_EMPTY:${PN} = "1" > + > +inherit allarch > + > +pkg_postinst_ontarget:${PN} () { > + touch /tmp/ro_rootfs_postinst_ontarget > +} > +""" > + > + self.write_recipeinc('emptytest', emptytest_inc) > + > + # Make rootfs read-only, but allow postinst ontarget task > + self.write_config('IMAGE_FEATURES += "read-only-rootfs"') > + self.append_config('IMAGE_FEATURES += "read-only-rootfs-delayed-postinsts"') > + > + self.append_config('IMAGE_INSTALL:append = " emptytest"') > + > + self.append_config('INIT_MANAGER = "systemd"') > + > + bitbake("core-image-minimal") > + > + with runqemu('core-image-minimal') as qemu: > + # Check that the file, touched in postinst ontarget task exists > + status, output = qemu.run_serial("[ -f /tmp/ro_rootfs_postinst_ontarget ] && echo 1 || echo 0") > + self.assertTrue("1" in output, msg=output) > + > class DiskMonTest(OESelftestTestCase): > > def test_stoptask_behavior(self): > -- > 2.45.2 > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#13401): https://lists.yoctoproject.org/g/poky/message/13401 > Mute This Topic: https://lists.yoctoproject.org/mt/106869018/1686489 > Group Owner: poky+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/poky/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hi Alex, I moved the test to meta/lib/oeqa/selftest/cases/runtime_test.py, please review the following patch: From 69685cc1bf4fa97e1105ce336c08e30a09834c4c Mon Sep 17 00:00:00 2001 From: "Gassner, Tobias.ext" <tobias.gassner.ext@karlstorz.com> Date: Thu, 4 Jul 2024 12:48:20 +0200 Subject: [PATCH] added test case to verify that post-installation on target scripts are executed even if using read-only rootfs when read-only-rootfs-delayed-postinsts is set Signed-off-by: Gassner, Tobias.ext <tobias.gassner.ext@karlstorz.com> --- .../recipes-test/postinst/postinst_1.0.bb | 18 ++++++++++- meta/lib/oeqa/selftest/cases/runtime_test.py | 30 +++++++++++++++++++ 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/meta-selftest/recipes-test/postinst/postinst_1.0.bb b/meta-selftest/recipes-test/postinst/postinst_1.0.bb index b936c4f44b..000ea24208 100644 --- a/meta-selftest/recipes-test/postinst/postinst_1.0.bb +++ b/meta-selftest/recipes-test/postinst/postinst_1.0.bb @@ -3,12 +3,13 @@ LICENSE = "MIT" inherit allarch -PACKAGES = "${PN}-rootfs ${PN}-delayed-a ${PN}-delayed-b ${PN}-rootfs-failing" +PACKAGES = "${PN}-rootfs ${PN}-delayed-a ${PN}-delayed-b ${PN}-rootfs-failing ${PN}-tmp" ALLOW_EMPTY:${PN}-rootfs = "1" ALLOW_EMPTY:${PN}-delayed-a = "1" ALLOW_EMPTY:${PN}-delayed-b = "1" ALLOW_EMPTY:${PN}-rootfs-failing = "1" +ALLOW_EMPTY:${PN}-tmp = "1" RDEPENDS:${PN}-delayed-a = "${PN}-rootfs" RDEPENDS:${PN}-delayed-b = "${PN}-delayed-a" @@ -70,3 +71,18 @@ pkg_postinst:${PN}-rootfs-failing () { # (oe-selftest checks for it). touch $D${TESTDIR}/rootfs-after-failure } + +# This scriptlet is used by a test that configures a read-only rootfs, +# however, writing in /tmp is still possible. +pkg_postinst_ontarget:${PN}-tmp () { + set -e + + TESTFILE='/tmp/postinst_tmp' + + if [ -e ${TESTFILE} ]; then + echo "${PN}-tmp: ${TESTFILE} does already exist" + exit 1 + fi + + touch ${TESTFILE} +} \ No newline at end of file diff --git a/meta/lib/oeqa/selftest/cases/runtime_test.py b/meta/lib/oeqa/selftest/cases/runtime_test.py index 12000aac16..5b097e5f92 100644 --- a/meta/lib/oeqa/selftest/cases/runtime_test.py +++ b/meta/lib/oeqa/selftest/cases/runtime_test.py @@ -407,6 +407,36 @@ class Postinst(OESelftestTestCase): self.assertFalse(os.path.isfile(os.path.join(hosttestdir, "rootfs-after-failure")), "rootfs-after-failure file was created") + + def test_postinst_on_target_for_read_only_rootfs(self): + """ + Summary: The purpose of this test case is to verify that post-installation + on target scripts are executed even if using read-only rootfs when + read-only-rootfs-delayed-postinsts is set + Expected: The test file in /tmp is created on first boot + Product: oe-core + """ + + for classes in ("package_rpm", "package_deb", "package_ipk"): + with self.subTest(package_class=classes): + features = 'CORE_IMAGE_EXTRA_INSTALL = "postinst-tmp"\n' + features += 'IMAGE_FEATURES += "package-management empty-root-password"\n' + features += 'PACKAGE_CLASSES = "%s"\n' % classes + features += 'INIT_MANAGER = "systemd"\n' + + features += 'IMAGE_FEATURES += "read-only-rootfs"\n' + features += 'IMAGE_FEATURES += "read-only-rootfs-delayed-postinsts"\n' + + self.write_config(features) + + bitbake('core-image-minimal') + + with runqemu('core-image-minimal') as qemu: + testfile = "/tmp/postinst_tmp" + status, output = qemu.run_serial("test -f %s && echo found" % testfile) + self.assertIn("found", output, "%s was not present on boot" % testfile) + + @OETestTag("runqemu") class SystemTap(OESelftestTestCase): """ -- 2.45.2
On Thu, 4 Jul 2024 at 12:50, Gassner, Tobias.ext via lists.yoctoproject.org <Tobias.Gassner.ext=karlstorz.com@lists.yoctoproject.org> wrote: > + for classes in ("package_rpm", "package_deb", "package_ipk"): Do we need to try every package manager in a loop? I know other tests do it, but if the issue being tested is read-only rootfs with read-write /tmp, then maybe testing just once is enough? > + with self.subTest(package_class=classes): > + features = 'CORE_IMAGE_EXTRA_INSTALL = "postinst-tmp"\n' > + features += 'IMAGE_FEATURES += "package-management empty-root-password"\n' > + features += 'PACKAGE_CLASSES = "%s"\n' % classes > + features += 'INIT_MANAGER = "systemd"\n' Why systemd? Is it needed for something? Note that any deviation from default config risks adding full builds without sstate to the test, preventing sstate reuse. Alex
On Thu, 2024-07-04 at 13:07 +0200, Alexander Kanavin via lists.yoctoproject.org wrote: > On Thu, 4 Jul 2024 at 12:50, Gassner, Tobias.ext via > lists.yoctoproject.org > <Tobias.Gassner.ext=karlstorz.com@lists.yoctoproject.org> wrote: > > + for classes in ("package_rpm", "package_deb", "package_ipk"): > > Do we need to try every package manager in a loop? I know other tests > do it, but if the issue being tested is read-only rootfs with > read-write /tmp, then maybe testing just once is enough? If postinsts are involved, that area of the code is quite package manager specific so that may be why the test code is doing this... Cheers, Richard
> > Do we need to try every package manager in a loop? I know other tests do > it, but if the issue being tested is read-only rootfs with read-write /tmp, > then maybe testing just once is enough? > The combination of not setting systemd as init and rpm as package class leads to a hanging boot of qemu, see below. > > Why systemd? Is it needed for something? Note that any deviation from default > config risks adding full builds without sstate to the test, preventing > sstate reuse. > As far as I understand the postinst_ontarget mechanism, a script is deployed to the target, which is executed on the first boot by init and deleted afterwards. since I configured a read-only rootfs, the script cannot be deleted. if I don't set systemd as init manager, qemu hangs on boot but *only* for the rpm package class. I have the assumption that this is related to the fact that the postinst_ontarget script cannot be deleted. With systemd as init manager qemu boots normally for all package classes and the test passes. I guess that if I would configure an overlayFS, the test would probably pass for other init managers as well, but that would add a lot of boilerplate code so I tried the KISS solution in the first attempt. 2024-07-04 15:25:44,358 - oe-selftest - INFO - Ran 1 test in 1193.515s 2024-07-04 15:25:44,358 - oe-selftest - INFO - FAILED 2024-07-04 15:25:44,358 - oe-selftest - INFO - (errors=1) 2024-07-04 15:25:47,294 - oe-selftest - INFO - RESULTS: 2024-07-04 15:25:47,294 - oe-selftest - INFO - RESULTS - runtime_test.Postinst.test_postinst_on_target_for_read_only_rootfs: UNKNOWN (1193.40s) 2024-07-04 15:25:47,394 - oe-selftest - INFO - SUMMARY: 2024-07-04 15:25:47,395 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 1193.517s 2024-07-04 15:25:47,395 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=0, skipped=0, failures=0, errors=1)
On Thu, 4 Jul 2024 at 15:34, Gassner, Tobias.ext via lists.yoctoproject.org <Tobias.Gassner.ext=karlstorz.com@lists.yoctoproject.org> wrote: > Why systemd? Is it needed for something? Note that any deviation fromdefault config risks adding full builds without sstate to the test,preventing sstate reuse. > > As far as I understand the postinst_ontarget mechanism, a script is deployed to the target, which is executed on the first boot by init and deleted afterwards. since I configured a read-only rootfs, the script cannot be deleted. if I don't set systemd as init manager, qemu hangs on boot but only for the rpm package class. I have the assumption that this is related to the fact that the postinst_ontarget script cannot be deleted. With systemd as init manager qemu boots normally for all package classes and the test passes. > I'm afraid you have to look into the hang. Setting 'systemd' is a workaround that covers up some kind of issue. I also wonder how the feature is supposed to work in the first place, if postinst mechanism relies on being able to delete the postinst, so it's executed only once, Alex
I digged a bit into the hanging issue by starting-up poky outside the test environment, but with the same configuration (rpm, no systemd, read-only-rootfs and read-only-rootfs-delayed-postinsts set, plus my first patch). After the kernel boot log messages I get the following (for testing purposes I use opkg-keyrings , as it provides a pkg_postinst_ontarget task) : Configuring packages on first boot.... (This may take several minutes. Please do not power off the machine.) Running postinst /etc/rpm-postinsts/100-opkg-keyrings... rm: remove '/etc/rpm-postinsts/100-opkg-keyrings'? Since I have to type 'yes' into the console and press enter to continue the test execution hangs on that line. The behavior (and hence the log messages) at that point are determined by the following script: https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts ( https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts ) When I apply the following patch From c144a7dcad403f41dd33b4429fe6481188512bb0 Mon Sep 17 00:00:00 2001 From: "Gassner, Tobias.ext" <tobias.gassner.ext@karlstorz.com> Date: Fri, 5 Jul 2024 09:51:25 +0200 Subject: [PATCH] adding force to remove postinst on target script Signed-off-by: Gassner, Tobias.ext <tobias.gassner.ext@karlstorz.com> --- meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts b/meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts index 1f3e692029..7f197bf89b 100755 --- a/meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts +++ b/meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts @@ -68,7 +68,7 @@ exec_postinst_scriptlets() { [ "$POSTINST_LOGGING" = "1" ] && eval echo "Running postinst $i..." $append_log if [ -x $i ]; then (sh -c $i $append_log) - rm $i + rm -f $i else echo "ERROR: postinst $i failed." [ "$POSTINST_LOGGING" = "1" ] && eval echo "ERROR: postinst $i failed." $append_log -- 2.45.2 I get the following output, and the test executes without hanging for rpm: Configuring packages on first boot.... (This may take several minutes. Please do not power off the machine.) Running postinst /etc/rpm-postinsts/100-opkg-keyrings... rm: can't remove '/etc/rpm-postinsts/100-opkg-keyrings': Read-only file system update-rc.d: /etc/init.d/run-postinsts exists during rc.d purge (continuing) Removing any system startup links for run-postinsts ... /etc/rcS.d/S99run-postinsts rm: can't remove '/etc/rcS.d/S99run-postinsts': Read-only file system Note: Since overlayFS is not configured for this test, the postinst ontarget script file cannot be removed, but this is expected. This specific test does not rely on the fact that the file can be removed so the test passes.
Here's an idea. How about adding (or extending) one of the tests in selftest/cases/overlayfs.py? I see there's plenty of examples there with /etc mounted that way (there's a whole class for it), and it might be a simple tweak to check for postinst as well. Alex On Fri, 5 Jul 2024 at 09:58, Gassner, Tobias.ext via lists.yoctoproject.org <Tobias.Gassner.ext=karlstorz.com@lists.yoctoproject.org> wrote: > > I digged a bit into the hanging issue by starting-up poky outside the test environment, but with the same configuration (rpm, no systemd, read-only-rootfs and read-only-rootfs-delayed-postinsts set, plus my first patch). After the kernel boot log messages I get the following (for testing purposes I use opkg-keyrings, as it provides a pkg_postinst_ontarget task): > > Configuring packages on first boot.... > (This may take several minutes. Please do not power off the machine.) > Running postinst /etc/rpm-postinsts/100-opkg-keyrings... > rm: remove '/etc/rpm-postinsts/100-opkg-keyrings'? > > Since I have to type 'yes' into the console and press enter to continue the test execution hangs on that line. > > The behavior (and hence the log messages) at that point are determined by the following script: https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts > > When I apply the following patch > > From c144a7dcad403f41dd33b4429fe6481188512bb0 Mon Sep 17 00:00:00 2001 > From: "Gassner, Tobias.ext" <tobias.gassner.ext@karlstorz.com> > Date: Fri, 5 Jul 2024 09:51:25 +0200 > Subject: [PATCH] adding force to remove postinst on target script > > Signed-off-by: Gassner, Tobias.ext <tobias.gassner.ext@karlstorz.com> > --- > meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts b/meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts > index 1f3e692029..7f197bf89b 100755 > --- a/meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts > +++ b/meta/recipes-devtools/run-postinsts/run-postinsts/run-postinsts > @@ -68,7 +68,7 @@ exec_postinst_scriptlets() { > [ "$POSTINST_LOGGING" = "1" ] && eval echo "Running postinst $i..." $append_log > if [ -x $i ]; then > (sh -c $i $append_log) > - rm $i > + rm -f $i > else > echo "ERROR: postinst $i failed." > [ "$POSTINST_LOGGING" = "1" ] && eval echo "ERROR: postinst $i failed." $append_log > -- > 2.45.2 > > I get the following output, and the test executes without hanging for rpm: > > Configuring packages on first boot.... > (This may take several minutes. Please do not power off the machine.) > Running postinst /etc/rpm-postinsts/100-opkg-keyrings... > rm: can't remove '/etc/rpm-postinsts/100-opkg-keyrings': Read-only file system > update-rc.d: /etc/init.d/run-postinsts exists during rc.d purge (continuing) > Removing any system startup links for run-postinsts ... > /etc/rcS.d/S99run-postinsts > rm: can't remove '/etc/rcS.d/S99run-postinsts': Read-only file system > > Note: Since overlayFS is not configured for this test, the postinst ontarget script file cannot be removed, but this is expected. This specific test does not rely on the fact that the file can be removed so the test passes. > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#13408): https://lists.yoctoproject.org/g/poky/message/13408 > Mute This Topic: https://lists.yoctoproject.org/mt/106869018/1686489 > Group Owner: poky+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/poky/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hi Alex, I looked into overlayfs.py but I'm not convinced that this is the right path. First of all, basically all relevant tests that could be used as a starting point use systemd as well which doesn't make them more generic compared to my test in runtime_test.py. Secondly, when I set IMAGE_FEATURES += "read-only-rootfs" in the test_correct_image_fstab test case, qemu also hangs during boot. Again, I build a poky with the same configuration as the test case and got the following log messages during boot: [FAILED] Failed to mount /mnt/overlay. See 'systemctl status mnt-overlay.mount' for details. [DEPEND] Dependency failed for Local File Systems. [DEPEND] Dependency failed for Overlayfs directories setup. [DEPEND] Dependency failed for Overlayfs mount unit. [DEPEND] Dependency failed for Overlayfs directories setup. [DEPEND] Dependency failed for Overlayfs mount unit. and it start hanging at the following line: You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, or "exit" to continue bootup. Press Enter for maintenance (or press Control-D to continue): when I enter systemctl status mnt-overlay.mount , I get: Jul 05 13:49:39 qemux86-64 mount[138]: mount: /mnt/overlay: mount point does not exist. /mnt/overlay is the mount point that is configured by the test. When I make the rootfs read only, the mount point can not be created and I get the error message. For a test, I changed the mount point to /tmp (which is writable even for a read-only rootfs), with that change, the system boots and the test passes. Overall, I would say that such a kind of test is even worse and in no way more generic than my previous test implementation in runtime_test.py. What do you think?
On Fri, 5 Jul 2024 at 15:58, Gassner, Tobias.ext via lists.yoctoproject.org <Tobias.Gassner.ext=karlstorz.com@lists.yoctoproject.org> wrote: > I looked into overlayfs.py but I'm not convinced that this is the right path. First of all, basically all relevant tests that could be used as a starting point use systemd as well which doesn't make them more generic compared to my test in runtime_test.py. Secondly, when I set IMAGE_FEATURES += "read-only-rootfs" in the test_correct_image_fstab test case, qemu also hangs during boot. ... > /mnt/overlay is the mount point that is configured by the test. When I make the rootfs read only, the mount point can not be created and I get the error message. For a test, I changed the mount point to /tmp (which is writable even for a read-only rootfs), with that change, the system boots and the test passes. I think these are the tests in the wrong class (OverlayFSTests). I was referring to class OverlayFSEtcRunTimeTests and the tests in it (those are specifically setting up /etc as the overlay and the rest of the image as read only). Wherever the test ends up, it needs to work in a way that allows deleting the postinst. Alex
Hi Alex, please have a look at the following patch in overlayfs.py: From da51462765e24b186c55431b53856eebd5f6e8da Mon Sep 17 00:00:00 2001 From: "Gassner, Tobias.ext" <tobias.gassner.ext@karlstorz.com> Date: Mon, 8 Jul 2024 15:50:58 +0200 Subject: [PATCH] added test case to verify that post-installation on target scripts are executed even if using read-only rootfs when read-only-rootfs-delayed-postinsts is set Signed-off-by: Gassner, Tobias.ext <tobias.gassner.ext@karlstorz.com> --- meta/lib/oeqa/selftest/cases/overlayfs.py | 41 ++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/meta/lib/oeqa/selftest/cases/overlayfs.py b/meta/lib/oeqa/selftest/cases/overlayfs.py index e31063567b..580fbdcb9c 100644 --- a/meta/lib/oeqa/selftest/cases/overlayfs.py +++ b/meta/lib/oeqa/selftest/cases/overlayfs.py @@ -5,7 +5,7 @@ # from oeqa.selftest.case import OESelftestTestCase -from oeqa.utils.commands import bitbake, runqemu +from oeqa.utils.commands import bitbake, runqemu, get_bb_vars from oeqa.core.decorator import OETestTag from oeqa.core.decorator.data import skipIfNotMachine @@ -466,6 +466,45 @@ IMAGE_INSTALL:append = " overlayfs-user" line = getline_qemu(output, "Read-only file system") self.assertTrue(line, msg=output) + @skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently") + def test_postinst_on_target_for_read_only_rootfs(self): + """ + Summary: The purpose of this test case is to verify that post-installation + on target scripts are executed even if using read-only rootfs when + read-only-rootfs-delayed-postinsts is set + Expected: The test files are created on first boot + """ + + import oe.path + + vars = get_bb_vars(("IMAGE_ROOTFS", "sysconfdir"), "core-image-minimal") + sysconfdir = vars["sysconfdir"] + self.assertIsNotNone(sysconfdir) + # Need to use oe.path here as sysconfdir starts with / + targettestdir = os.path.join(sysconfdir, "postinst-test") + + config = self.get_working_config() + + args = { + 'OVERLAYFS_INIT_OPTION': "", + 'OVERLAYFS_ETC_USE_ORIG_INIT_NAME': 1, + 'OVERLAYFS_ROOTFS_TYPE': "ext4", + 'OVERLAYFS_ETC_CREATE_MOUNT_DIRS': 1 + } + + # read-only-rootfs is already set in get_working_config() + config += 'EXTRA_IMAGE_FEATURES += "read-only-rootfs-delayed-postinsts"\n' + config += 'CORE_IMAGE_EXTRA_INSTALL = "postinst-delayed-b"\n' + + self.write_config(config.format(**args)) + + res = bitbake('core-image-minimal') + + with runqemu('core-image-minimal', image_fstype='wic') as qemu: + for filename in ("rootfs", "delayed-a", "delayed-b"): + status, output = qemu.run_serial("test -f %s && echo found" % os.path.join(targettestdir, filename)) + self.assertIn("found", output, "%s was not present on boot" % filename) + def get_working_config(self): return """ # Use systemd as init manager -- 2.45.2 Greetings -Tobias
Looks alright to me. Does it fail without the patch and pass with it? Alex On Mon, 8 Jul 2024 at 15:53, Gassner, Tobias.ext via lists.yoctoproject.org <Tobias.Gassner.ext=karlstorz.com@lists.yoctoproject.org> wrote: > > Hi Alex, > > please have a look at the following patch in overlayfs.py: > > From da51462765e24b186c55431b53856eebd5f6e8da Mon Sep 17 00:00:00 2001 > From: "Gassner, Tobias.ext" <tobias.gassner.ext@karlstorz.com> > Date: Mon, 8 Jul 2024 15:50:58 +0200 > Subject: [PATCH] added test case to verify that post-installation on target > scripts are executed even if using read-only rootfs when > read-only-rootfs-delayed-postinsts is set > > Signed-off-by: Gassner, Tobias.ext <tobias.gassner.ext@karlstorz.com> > --- > meta/lib/oeqa/selftest/cases/overlayfs.py | 41 ++++++++++++++++++++++- > 1 file changed, 40 insertions(+), 1 deletion(-) > > diff --git a/meta/lib/oeqa/selftest/cases/overlayfs.py b/meta/lib/oeqa/selftest/cases/overlayfs.py > index e31063567b..580fbdcb9c 100644 > --- a/meta/lib/oeqa/selftest/cases/overlayfs.py > +++ b/meta/lib/oeqa/selftest/cases/overlayfs.py > @@ -5,7 +5,7 @@ > # > > from oeqa.selftest.case import OESelftestTestCase > -from oeqa.utils.commands import bitbake, runqemu > +from oeqa.utils.commands import bitbake, runqemu, get_bb_vars > from oeqa.core.decorator import OETestTag > from oeqa.core.decorator.data import skipIfNotMachine > > @@ -466,6 +466,45 @@ IMAGE_INSTALL:append = " overlayfs-user" > line = getline_qemu(output, "Read-only file system") > self.assertTrue(line, msg=output) > > + @skipIfNotMachine("qemux86-64", "tests are qemux86-64 specific currently") > + def test_postinst_on_target_for_read_only_rootfs(self): > + """ > + Summary: The purpose of this test case is to verify that post-installation > + on target scripts are executed even if using read-only rootfs when > + read-only-rootfs-delayed-postinsts is set > + Expected: The test files are created on first boot > + """ > + > + import oe.path > + > + vars = get_bb_vars(("IMAGE_ROOTFS", "sysconfdir"), "core-image-minimal") > + sysconfdir = vars["sysconfdir"] > + self.assertIsNotNone(sysconfdir) > + # Need to use oe.path here as sysconfdir starts with / > + targettestdir = os.path.join(sysconfdir, "postinst-test") > + > + config = self.get_working_config() > + > + args = { > + 'OVERLAYFS_INIT_OPTION': "", > + 'OVERLAYFS_ETC_USE_ORIG_INIT_NAME': 1, > + 'OVERLAYFS_ROOTFS_TYPE': "ext4", > + 'OVERLAYFS_ETC_CREATE_MOUNT_DIRS': 1 > + } > + > + # read-only-rootfs is already set in get_working_config() > + config += 'EXTRA_IMAGE_FEATURES += "read-only-rootfs-delayed-postinsts"\n' > + config += 'CORE_IMAGE_EXTRA_INSTALL = "postinst-delayed-b"\n' > + > + self.write_config(config.format(**args)) > + > + res = bitbake('core-image-minimal') > + > + with runqemu('core-image-minimal', image_fstype='wic') as qemu: > + for filename in ("rootfs", "delayed-a", "delayed-b"): > + status, output = qemu.run_serial("test -f %s && echo found" % os.path.join(targettestdir, filename)) > + self.assertIn("found", output, "%s was not present on boot" % filename) > + > def get_working_config(self): > return """ > # Use systemd as init manager > -- > 2.45.2 > > > Greetings > -Tobias > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#13416): https://lists.yoctoproject.org/g/poky/message/13416 > Mute This Topic: https://lists.yoctoproject.org/mt/106869018/1686489 > Group Owner: poky+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/poky/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
> > Does it fail without the patch and pass with it? > Yes.
Then I suppose everything is ready to produce a patchset and send it to oe-core mailing list for official review/integration? Alex On Mon, 8 Jul 2024 at 16:15, Gassner, Tobias.ext via lists.yoctoproject.org <Tobias.Gassner.ext=karlstorz.com@lists.yoctoproject.org> wrote: > > Does it fail without the patch and pass with it? > > Yes. > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#13418): https://lists.yoctoproject.org/g/poky/message/13418 > Mute This Topic: https://lists.yoctoproject.org/mt/106869018/1686489 > Group Owner: poky+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/poky/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
The patch has been submitted to oe-core, see https://lists.openembedded.org/g/openembedded-core/topic/patch_execute/107118664 ( https://lists.openembedded.org/g/openembedded-core/topic/patch_execute/107118664 )
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py index 8cd48f9450..5abce4ad7d 100644 --- a/meta/lib/oe/rootfs.py +++ b/meta/lib/oe/rootfs.py @@ -269,7 +269,11 @@ class Rootfs(object, metaclass=ABCMeta): self.pm.remove(["run-postinsts"]) image_rorfs = bb.utils.contains("IMAGE_FEATURES", "read-only-rootfs", + True, False, self.d) and \ + not bb.utils.contains("IMAGE_FEATURES", + "read-only-rootfs-delayed-postinsts", True, False, self.d) + image_rorfs_force = self.d.getVar('FORCE_RO_REMOVE') if image_rorfs or image_rorfs_force == "1":