| Message ID | 20240819082215.1322013-1-niko.mauno@vaisala.com |
|---|---|
| State | Accepted, archived |
| Commit | dc086d9a8613143607af3583c72ed892e20b4d66 |
| Headers | show |
| Series | [v2] util-linux: Add PACKAGECONFIG option to mitigate rootfs remount error | expand |
I don’t think this should be disabled by default actually. If the default upstream behavior is to enable, and all current linux-yocto kernels have the needed support, then we should follow that. Alex On Mon 19. Aug 2024 at 10.23, Niko Mauno via lists.openembedded.org <niko.mauno=vaisala.com@lists.openembedded.org> wrote: > The 2.39 version of util-linux took new file descriptors based mount > kernel API into use. In relation to this change, the upstream release > notes in > > https://github.com/util-linux/util-linux/blob/v2.39/Documentation/releases/v2.39-ReleaseNotes#L14-L21 > mention that > > This change is very aggressive to libmount code, but hopefully, it does > not introduce regressions in traditional mount(8) behavior. > > After observing following failure when booting a board using a bit > older 6.1 series kernel together with initramfs rootfs based boot flow > > [FAILED] Failed to start Remount Root and Kernel File Systems. > See 'systemctl status systemd-remount-fs.service' for details. > > closer inspection revealed: > > demoboard ~ # systemctl status -l systemd-remount-fs.service > x systemd-remount-fs.service - Remount Root and Kernel File Systems > Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; > enabled-runtime; preset: disabled) > Active: failed (Result: exit-code) since Wed 2024-08-14 14:53:48 > UTC; 1min 22s ago > Docs: man:systemd-remount-fs.service(8) > > https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems > Process: 76 ExecStart=/usr/lib/systemd/systemd-remount-fs > (code=exited, status=1/FAILURE) > Main PID: 76 (code=exited, status=1/FAILURE) > > Aug 14 14:53:48 demoboard systemd-remount-fs[76]: /usr/bin/mount for / > exited with exit status 32. > Aug 14 14:53:48 demoboard systemd-remount-fs[81]: mount: /: mount point > not mounted or bad option. > Aug 14 14:53:48 demoboard systemd-remount-fs[81]: dmesg(1) may > have more information after failed mount system call. > Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: Main > process exited, code=exited, status=1/FAILURE > Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: Failed > with result 'exit-code'. > Aug 14 14:53:48 demoboard systemd[1]: Failed to start Remount Root and > Kernel File Systems. > > also consequentially, 'systemctl status' reported: > > State: degraded > > When issuing 'strace -ff mount -o remount /' the failure occurred at > > mount_setattr(3, "", AT_EMPTY_PATH, > {attr_set=MOUNT_ATTR_RDONLY|MOUNT_ATTR_NOATIME|MOUNT_ATTR_NODIRATIME, > attr_clr=MOUNT_ATTR_NOSUID|MOUNT_ATTR_NODEV|MOUNT_ATTR_NOEXEC|MOUNT_ATTR_NOATIME|MOUNT_ATTR_STRICTATIME|MOUNT_ATTR_NOSYMFOLLOW|0x40, > propagation=0 /* MS_??? */, userns_fd=0}, 32) = -1 EINVAL (Invalid argument) > > After further investigation, The issue was pinpointed to lack of Linux > kernel commit > > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=95de4ad173ca0e61034f3145d66917970961c210 > ("fs: relax mount_setattr() permission checks") in the kernel version > that was being used. Above mitigation was discussed in email related to > then-rejected CVE-2024-26821: > > https://lore.kernel.org/linux-cve-announce/2024051606-imaging-entrench-b327@gregkh/T/ > > After testing with qemuarm64 machine different linux-yocto versions, > it was observed that the issue impacts following versions of currently > supported LTS kernels: > - 6.6.17 (fixed since 6.6.18 i.e. mount_setattr() returns 0) > - 6.1.78 (fixed since 6.1.79 i.e. mount_setattr() returns 0) > - 5.15.164 which is currently the newest of 5.15.y series (i.e. no > known working version) > > Taking the above findings into consideration, add a new PACKAGECONFIG > option which by default opts users out from using the feature which > can cause issues with some older kernels. > > Versions 5.10.223, 5.4.279 and 4.10.317 were also tested but the issue > was not reproduced with those versions - using strace showed that the > mount_setattr call associated with the new mount API problem was not > issued with these LTS kernel versions, which seemed to be confirmed > also by following libmount debug message in these cases: > > 415: libmount: HOOK: [0x7fa115e818]: failed to init new API > > Note: In addition to the aforementioned, this change was > tested also briefly using the current latest kernel versions 6.1.104, > 6.6.45 and 6.10.3 that using the old mount API with newest kernels > did not introduce any observable regression to the boot flow. > > Signed-off-by: Niko Mauno <niko.mauno@vaisala.com> > --- > meta/recipes-core/util-linux/util-linux_2.40.1.bb | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/meta/recipes-core/util-linux/util-linux_2.40.1.bb > b/meta/recipes-core/util-linux/util-linux_2.40.1.bb > index a1aab94055..2d89eb6745 100644 > --- a/meta/recipes-core/util-linux/util-linux_2.40.1.bb > +++ b/meta/recipes-core/util-linux/util-linux_2.40.1.bb > @@ -107,6 +107,13 @@ PACKAGECONFIG[cryptsetup] = > "--with-cryptsetup,--without-cryptsetup,cryptsetup" > PACKAGECONFIG[chfn-chsh] = "--enable-chfn-chsh,--disable-chfn-chsh," > PACKAGECONFIG[selinux] = "--with-selinux,--without-selinux,libselinux" > PACKAGECONFIG[lastlog2] = > "--enable-liblastlog2,--disable-liblastlog2,sqlite3" > +# Using the new file descriptors based mount kernel API can cause rootfs > remount failure with some older kernels. > +# Of currently supported LTS kernels, it seems safe to enable this option > with: > +# - version 6.6.18 and newer in the 6.6.y series. > +# - version 6.1.79 and newer in the 6.1.y series. > +# - with the 5.15.y series, versions till at least 5.15.164 can not use > it. > +# - with 5.10.y, 5.4.y and 4.19.y series kernels libmount seemed to use > the old API regardless of this option. > +PACKAGECONFIG[new-mount-api] = > "--enable-libmount-mountfd-support,--disable-libmount-mountfd-support" > > EXTRA_OEMAKE = "ARCH=${TARGET_ARCH} CPU= CPUOPT= 'OPT=${CFLAGS}'" > > -- > 2.39.2 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#203486): > https://lists.openembedded.org/g/openembedded-core/message/203486 > Mute This Topic: https://lists.openembedded.org/mt/107977031/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
Please see my comments in your v1 patch. I don't think we should add this PACKAGECONFIG, instead the util-linux's check codes need to be improved. Regards, Qi On 8/19/24 16:22, Niko Mauno via lists.openembedded.org wrote: > The 2.39 version of util-linux took new file descriptors based mount > kernel API into use. In relation to this change, the upstream release > notes in > https://github.com/util-linux/util-linux/blob/v2.39/Documentation/releases/v2.39-ReleaseNotes#L14-L21 > mention that > > This change is very aggressive to libmount code, but hopefully, it does not introduce regressions in traditional mount(8) behavior. > > After observing following failure when booting a board using a bit > older 6.1 series kernel together with initramfs rootfs based boot flow > > [FAILED] Failed to start Remount Root and Kernel File Systems. > See 'systemctl status systemd-remount-fs.service' for details. > > closer inspection revealed: > > demoboard ~ # systemctl status -l systemd-remount-fs.service > x systemd-remount-fs.service - Remount Root and Kernel File Systems > Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; enabled-runtime; preset: disabled) > Active: failed (Result: exit-code) since Wed 2024-08-14 14:53:48 UTC; 1min 22s ago > Docs: man:systemd-remount-fs.service(8) > https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems > Process: 76 ExecStart=/usr/lib/systemd/systemd-remount-fs (code=exited, status=1/FAILURE) > Main PID: 76 (code=exited, status=1/FAILURE) > > Aug 14 14:53:48 demoboard systemd-remount-fs[76]: /usr/bin/mount for / exited with exit status 32. > Aug 14 14:53:48 demoboard systemd-remount-fs[81]: mount: /: mount point not mounted or bad option. > Aug 14 14:53:48 demoboard systemd-remount-fs[81]: dmesg(1) may have more information after failed mount system call. > Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE > Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'. > Aug 14 14:53:48 demoboard systemd[1]: Failed to start Remount Root and Kernel File Systems. > > also consequentially, 'systemctl status' reported: > > State: degraded > > When issuing 'strace -ff mount -o remount /' the failure occurred at > > mount_setattr(3, "", AT_EMPTY_PATH, {attr_set=MOUNT_ATTR_RDONLY|MOUNT_ATTR_NOATIME|MOUNT_ATTR_NODIRATIME, attr_clr=MOUNT_ATTR_NOSUID|MOUNT_ATTR_NODEV|MOUNT_ATTR_NOEXEC|MOUNT_ATTR_NOATIME|MOUNT_ATTR_STRICTATIME|MOUNT_ATTR_NOSYMFOLLOW|0x40, propagation=0 /* MS_??? */, userns_fd=0}, 32) = -1 EINVAL (Invalid argument) > > After further investigation, The issue was pinpointed to lack of Linux > kernel commit > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=95de4ad173ca0e61034f3145d66917970961c210 > ("fs: relax mount_setattr() permission checks") in the kernel version > that was being used. Above mitigation was discussed in email related to > then-rejected CVE-2024-26821: > https://lore.kernel.org/linux-cve-announce/2024051606-imaging-entrench-b327@gregkh/T/ > > After testing with qemuarm64 machine different linux-yocto versions, > it was observed that the issue impacts following versions of currently > supported LTS kernels: > - 6.6.17 (fixed since 6.6.18 i.e. mount_setattr() returns 0) > - 6.1.78 (fixed since 6.1.79 i.e. mount_setattr() returns 0) > - 5.15.164 which is currently the newest of 5.15.y series (i.e. no > known working version) > > Taking the above findings into consideration, add a new PACKAGECONFIG > option which by default opts users out from using the feature which > can cause issues with some older kernels. > > Versions 5.10.223, 5.4.279 and 4.10.317 were also tested but the issue > was not reproduced with those versions - using strace showed that the > mount_setattr call associated with the new mount API problem was not > issued with these LTS kernel versions, which seemed to be confirmed > also by following libmount debug message in these cases: > > 415: libmount: HOOK: [0x7fa115e818]: failed to init new API > > Note: In addition to the aforementioned, this change was > tested also briefly using the current latest kernel versions 6.1.104, > 6.6.45 and 6.10.3 that using the old mount API with newest kernels > did not introduce any observable regression to the boot flow. > > Signed-off-by: Niko Mauno <niko.mauno@vaisala.com> > --- > meta/recipes-core/util-linux/util-linux_2.40.1.bb | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/meta/recipes-core/util-linux/util-linux_2.40.1.bb b/meta/recipes-core/util-linux/util-linux_2.40.1.bb > index a1aab94055..2d89eb6745 100644 > --- a/meta/recipes-core/util-linux/util-linux_2.40.1.bb > +++ b/meta/recipes-core/util-linux/util-linux_2.40.1.bb > @@ -107,6 +107,13 @@ PACKAGECONFIG[cryptsetup] = "--with-cryptsetup,--without-cryptsetup,cryptsetup" > PACKAGECONFIG[chfn-chsh] = "--enable-chfn-chsh,--disable-chfn-chsh," > PACKAGECONFIG[selinux] = "--with-selinux,--without-selinux,libselinux" > PACKAGECONFIG[lastlog2] = "--enable-liblastlog2,--disable-liblastlog2,sqlite3" > +# Using the new file descriptors based mount kernel API can cause rootfs remount failure with some older kernels. > +# Of currently supported LTS kernels, it seems safe to enable this option with: > +# - version 6.6.18 and newer in the 6.6.y series. > +# - version 6.1.79 and newer in the 6.1.y series. > +# - with the 5.15.y series, versions till at least 5.15.164 can not use it. > +# - with 5.10.y, 5.4.y and 4.19.y series kernels libmount seemed to use the old API regardless of this option. > +PACKAGECONFIG[new-mount-api] = "--enable-libmount-mountfd-support,--disable-libmount-mountfd-support" > > EXTRA_OEMAKE = "ARCH=${TARGET_ARCH} CPU= CPUOPT= 'OPT=${CFLAGS}'" > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#203486): https://lists.openembedded.org/g/openembedded-core/message/203486 > Mute This Topic: https://lists.openembedded.org/mt/107977031/7304865 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [Qi.Chen@eng.windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- >
I agree with Alex, if we really need to add this PACKAGECONFIG, the default should be enabling mountfd support. Regards, Qi On 8/19/24 16:42, Chen Qi via lists.openembedded.org wrote: > Please see my comments in your v1 patch. > I don't think we should add this PACKAGECONFIG, instead the > util-linux's check codes need to be improved. > > Regards, > Qi > > On 8/19/24 16:22, Niko Mauno via lists.openembedded.org wrote: >> The 2.39 version of util-linux took new file descriptors based mount >> kernel API into use. In relation to this change, the upstream release >> notes in >> https://github.com/util-linux/util-linux/blob/v2.39/Documentation/releases/v2.39-ReleaseNotes#L14-L21 >> >> mention that >> >> This change is very aggressive to libmount code, but hopefully, it >> does not introduce regressions in traditional mount(8) behavior. >> >> After observing following failure when booting a board using a bit >> older 6.1 series kernel together with initramfs rootfs based boot flow >> >> [FAILED] Failed to start Remount Root and Kernel File Systems. >> See 'systemctl status systemd-remount-fs.service' for details. >> >> closer inspection revealed: >> >> demoboard ~ # systemctl status -l systemd-remount-fs.service >> x systemd-remount-fs.service - Remount Root and Kernel File Systems >> Loaded: loaded >> (/usr/lib/systemd/system/systemd-remount-fs.service; enabled-runtime; >> preset: disabled) >> Active: failed (Result: exit-code) since Wed 2024-08-14 >> 14:53:48 UTC; 1min 22s ago >> Docs: man:systemd-remount-fs.service(8) >> https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems >> Process: 76 ExecStart=/usr/lib/systemd/systemd-remount-fs >> (code=exited, status=1/FAILURE) >> Main PID: 76 (code=exited, status=1/FAILURE) >> >> Aug 14 14:53:48 demoboard systemd-remount-fs[76]: /usr/bin/mount >> for / exited with exit status 32. >> Aug 14 14:53:48 demoboard systemd-remount-fs[81]: mount: /: mount >> point not mounted or bad option. >> Aug 14 14:53:48 demoboard systemd-remount-fs[81]: dmesg(1) may >> have more information after failed mount system call. >> Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: >> Main process exited, code=exited, status=1/FAILURE >> Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: >> Failed with result 'exit-code'. >> Aug 14 14:53:48 demoboard systemd[1]: Failed to start Remount Root >> and Kernel File Systems. >> >> also consequentially, 'systemctl status' reported: >> >> State: degraded >> >> When issuing 'strace -ff mount -o remount /' the failure occurred at >> >> mount_setattr(3, "", AT_EMPTY_PATH, >> {attr_set=MOUNT_ATTR_RDONLY|MOUNT_ATTR_NOATIME|MOUNT_ATTR_NODIRATIME, >> attr_clr=MOUNT_ATTR_NOSUID|MOUNT_ATTR_NODEV|MOUNT_ATTR_NOEXEC|MOUNT_ATTR_NOATIME|MOUNT_ATTR_STRICTATIME|MOUNT_ATTR_NOSYMFOLLOW|0x40, >> propagation=0 /* MS_??? */, userns_fd=0}, 32) = -1 EINVAL (Invalid >> argument) >> >> After further investigation, The issue was pinpointed to lack of Linux >> kernel commit >> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=95de4ad173ca0e61034f3145d66917970961c210 >> >> ("fs: relax mount_setattr() permission checks") in the kernel version >> that was being used. Above mitigation was discussed in email related to >> then-rejected CVE-2024-26821: >> https://lore.kernel.org/linux-cve-announce/2024051606-imaging-entrench-b327@gregkh/T/ >> >> >> After testing with qemuarm64 machine different linux-yocto versions, >> it was observed that the issue impacts following versions of currently >> supported LTS kernels: >> - 6.6.17 (fixed since 6.6.18 i.e. mount_setattr() returns 0) >> - 6.1.78 (fixed since 6.1.79 i.e. mount_setattr() returns 0) >> - 5.15.164 which is currently the newest of 5.15.y series (i.e. no >> known working version) >> >> Taking the above findings into consideration, add a new PACKAGECONFIG >> option which by default opts users out from using the feature which >> can cause issues with some older kernels. >> >> Versions 5.10.223, 5.4.279 and 4.10.317 were also tested but the issue >> was not reproduced with those versions - using strace showed that the >> mount_setattr call associated with the new mount API problem was not >> issued with these LTS kernel versions, which seemed to be confirmed >> also by following libmount debug message in these cases: >> >> 415: libmount: HOOK: [0x7fa115e818]: failed to init new API >> >> Note: In addition to the aforementioned, this change was >> tested also briefly using the current latest kernel versions 6.1.104, >> 6.6.45 and 6.10.3 that using the old mount API with newest kernels >> did not introduce any observable regression to the boot flow. >> >> Signed-off-by: Niko Mauno <niko.mauno@vaisala.com> >> --- >> meta/recipes-core/util-linux/util-linux_2.40.1.bb | 7 +++++++ >> 1 file changed, 7 insertions(+) >> >> diff --git a/meta/recipes-core/util-linux/util-linux_2.40.1.bb >> b/meta/recipes-core/util-linux/util-linux_2.40.1.bb >> index a1aab94055..2d89eb6745 100644 >> --- a/meta/recipes-core/util-linux/util-linux_2.40.1.bb >> +++ b/meta/recipes-core/util-linux/util-linux_2.40.1.bb >> @@ -107,6 +107,13 @@ PACKAGECONFIG[cryptsetup] = >> "--with-cryptsetup,--without-cryptsetup,cryptsetup" >> PACKAGECONFIG[chfn-chsh] = "--enable-chfn-chsh,--disable-chfn-chsh," >> PACKAGECONFIG[selinux] = "--with-selinux,--without-selinux,libselinux" >> PACKAGECONFIG[lastlog2] = >> "--enable-liblastlog2,--disable-liblastlog2,sqlite3" >> +# Using the new file descriptors based mount kernel API can cause >> rootfs remount failure with some older kernels. >> +# Of currently supported LTS kernels, it seems safe to enable this >> option with: >> +# - version 6.6.18 and newer in the 6.6.y series. >> +# - version 6.1.79 and newer in the 6.1.y series. >> +# - with the 5.15.y series, versions till at least 5.15.164 can not >> use it. >> +# - with 5.10.y, 5.4.y and 4.19.y series kernels libmount seemed to >> use the old API regardless of this option. >> +PACKAGECONFIG[new-mount-api] = >> "--enable-libmount-mountfd-support,--disable-libmount-mountfd-support" >> EXTRA_OEMAKE = "ARCH=${TARGET_ARCH} CPU= CPUOPT= 'OPT=${CFLAGS}'" >> >> >> > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#203490): https://lists.openembedded.org/g/openembedded-core/message/203490 > Mute This Topic: https://lists.openembedded.org/mt/107977031/3618072 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [Qi.Chen@windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Mon, 19 Aug 2024 at 10:42, Chen Qi via lists.openembedded.org <Qi.Chen=windriver.com@lists.openembedded.org> wrote: > Please see my comments in your v1 patch. > I don't think we should add this PACKAGECONFIG, instead the util-linux's > check codes need to be improved. If upstream provides an option to configure things in a release that we currently provide, then we should absolutely have a PACKAGECONFIG for it. And I think it should preserve existing default behaviour, e.g. enable the new API. Whether an upstream check can be improved is a separate question. First, it would be good to establish what it's actually doing, and why is it arriving at an incorrect decision for older kernels. Then we can see if there's something upstream could reasonably improve (I suspect not, as it's likely checking kernel headers from linux-libc-headers). Alex
On 19.8.2024 11.27, Alexander Kanavin wrote: > I don’t think this should be disabled by default actually. If the > default upstream behavior is to enable, and all current linux-yocto > kernels have the needed support, then we should follow that. Thanks, addressed in v3 which I just submitted. -Niko
diff --git a/meta/recipes-core/util-linux/util-linux_2.40.1.bb b/meta/recipes-core/util-linux/util-linux_2.40.1.bb index a1aab94055..2d89eb6745 100644 --- a/meta/recipes-core/util-linux/util-linux_2.40.1.bb +++ b/meta/recipes-core/util-linux/util-linux_2.40.1.bb @@ -107,6 +107,13 @@ PACKAGECONFIG[cryptsetup] = "--with-cryptsetup,--without-cryptsetup,cryptsetup" PACKAGECONFIG[chfn-chsh] = "--enable-chfn-chsh,--disable-chfn-chsh," PACKAGECONFIG[selinux] = "--with-selinux,--without-selinux,libselinux" PACKAGECONFIG[lastlog2] = "--enable-liblastlog2,--disable-liblastlog2,sqlite3" +# Using the new file descriptors based mount kernel API can cause rootfs remount failure with some older kernels. +# Of currently supported LTS kernels, it seems safe to enable this option with: +# - version 6.6.18 and newer in the 6.6.y series. +# - version 6.1.79 and newer in the 6.1.y series. +# - with the 5.15.y series, versions till at least 5.15.164 can not use it. +# - with 5.10.y, 5.4.y and 4.19.y series kernels libmount seemed to use the old API regardless of this option. +PACKAGECONFIG[new-mount-api] = "--enable-libmount-mountfd-support,--disable-libmount-mountfd-support" EXTRA_OEMAKE = "ARCH=${TARGET_ARCH} CPU= CPUOPT= 'OPT=${CFLAGS}'"
The 2.39 version of util-linux took new file descriptors based mount kernel API into use. In relation to this change, the upstream release notes in https://github.com/util-linux/util-linux/blob/v2.39/Documentation/releases/v2.39-ReleaseNotes#L14-L21 mention that This change is very aggressive to libmount code, but hopefully, it does not introduce regressions in traditional mount(8) behavior. After observing following failure when booting a board using a bit older 6.1 series kernel together with initramfs rootfs based boot flow [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. closer inspection revealed: demoboard ~ # systemctl status -l systemd-remount-fs.service x systemd-remount-fs.service - Remount Root and Kernel File Systems Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; enabled-runtime; preset: disabled) Active: failed (Result: exit-code) since Wed 2024-08-14 14:53:48 UTC; 1min 22s ago Docs: man:systemd-remount-fs.service(8) https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems Process: 76 ExecStart=/usr/lib/systemd/systemd-remount-fs (code=exited, status=1/FAILURE) Main PID: 76 (code=exited, status=1/FAILURE) Aug 14 14:53:48 demoboard systemd-remount-fs[76]: /usr/bin/mount for / exited with exit status 32. Aug 14 14:53:48 demoboard systemd-remount-fs[81]: mount: /: mount point not mounted or bad option. Aug 14 14:53:48 demoboard systemd-remount-fs[81]: dmesg(1) may have more information after failed mount system call. Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'. Aug 14 14:53:48 demoboard systemd[1]: Failed to start Remount Root and Kernel File Systems. also consequentially, 'systemctl status' reported: State: degraded When issuing 'strace -ff mount -o remount /' the failure occurred at mount_setattr(3, "", AT_EMPTY_PATH, {attr_set=MOUNT_ATTR_RDONLY|MOUNT_ATTR_NOATIME|MOUNT_ATTR_NODIRATIME, attr_clr=MOUNT_ATTR_NOSUID|MOUNT_ATTR_NODEV|MOUNT_ATTR_NOEXEC|MOUNT_ATTR_NOATIME|MOUNT_ATTR_STRICTATIME|MOUNT_ATTR_NOSYMFOLLOW|0x40, propagation=0 /* MS_??? */, userns_fd=0}, 32) = -1 EINVAL (Invalid argument) After further investigation, The issue was pinpointed to lack of Linux kernel commit https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=95de4ad173ca0e61034f3145d66917970961c210 ("fs: relax mount_setattr() permission checks") in the kernel version that was being used. Above mitigation was discussed in email related to then-rejected CVE-2024-26821: https://lore.kernel.org/linux-cve-announce/2024051606-imaging-entrench-b327@gregkh/T/ After testing with qemuarm64 machine different linux-yocto versions, it was observed that the issue impacts following versions of currently supported LTS kernels: - 6.6.17 (fixed since 6.6.18 i.e. mount_setattr() returns 0) - 6.1.78 (fixed since 6.1.79 i.e. mount_setattr() returns 0) - 5.15.164 which is currently the newest of 5.15.y series (i.e. no known working version) Taking the above findings into consideration, add a new PACKAGECONFIG option which by default opts users out from using the feature which can cause issues with some older kernels. Versions 5.10.223, 5.4.279 and 4.10.317 were also tested but the issue was not reproduced with those versions - using strace showed that the mount_setattr call associated with the new mount API problem was not issued with these LTS kernel versions, which seemed to be confirmed also by following libmount debug message in these cases: 415: libmount: HOOK: [0x7fa115e818]: failed to init new API Note: In addition to the aforementioned, this change was tested also briefly using the current latest kernel versions 6.1.104, 6.6.45 and 6.10.3 that using the old mount API with newest kernels did not introduce any observable regression to the boot flow. Signed-off-by: Niko Mauno <niko.mauno@vaisala.com> --- meta/recipes-core/util-linux/util-linux_2.40.1.bb | 7 +++++++ 1 file changed, 7 insertions(+)