| Message ID | 20240819092739.2063887-1-niko.mauno@vaisala.com |
|---|---|
| State | Accepted, archived |
| Commit | dc086d9a8613143607af3583c72ed892e20b4d66 |
| Headers | show |
| Series | [v3] util-linux: Add PACKAGECONFIG option to mitigate rootfs remount error | expand |
Thanks, this version lgtm. Alex On Mon, 19 Aug 2024 at 11:27, Niko Mauno via lists.openembedded.org <niko.mauno=vaisala.com@lists.openembedded.org> wrote: > > The 2.39 version of util-linux took new file descriptors based mount > kernel API into use. In relation to this change, the upstream release > notes in > https://github.com/util-linux/util-linux/blob/v2.39/Documentation/releases/v2.39-ReleaseNotes#L14-L21 > mention that > > This change is very aggressive to libmount code, but hopefully, it does not introduce regressions in traditional mount(8) behavior. > > After observing following failure when booting a board using a bit > older 6.1 series kernel together with initramfs rootfs based boot flow > > [FAILED] Failed to start Remount Root and Kernel File Systems. > See 'systemctl status systemd-remount-fs.service' for details. > > closer inspection revealed: > > demoboard ~ # systemctl status -l systemd-remount-fs.service > x systemd-remount-fs.service - Remount Root and Kernel File Systems > Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; enabled-runtime; preset: disabled) > Active: failed (Result: exit-code) since Wed 2024-08-14 14:53:48 UTC; 1min 22s ago > Docs: man:systemd-remount-fs.service(8) > https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems > Process: 76 ExecStart=/usr/lib/systemd/systemd-remount-fs (code=exited, status=1/FAILURE) > Main PID: 76 (code=exited, status=1/FAILURE) > > Aug 14 14:53:48 demoboard systemd-remount-fs[76]: /usr/bin/mount for / exited with exit status 32. > Aug 14 14:53:48 demoboard systemd-remount-fs[81]: mount: /: mount point not mounted or bad option. > Aug 14 14:53:48 demoboard systemd-remount-fs[81]: dmesg(1) may have more information after failed mount system call. > Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE > Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'. > Aug 14 14:53:48 demoboard systemd[1]: Failed to start Remount Root and Kernel File Systems. > > also consequentially, 'systemctl status' reported: > > State: degraded > > When issuing 'strace -ff mount -o remount /' the failure occurred at > > mount_setattr(3, "", AT_EMPTY_PATH, {attr_set=MOUNT_ATTR_RDONLY|MOUNT_ATTR_NOATIME|MOUNT_ATTR_NODIRATIME, attr_clr=MOUNT_ATTR_NOSUID|MOUNT_ATTR_NODEV|MOUNT_ATTR_NOEXEC|MOUNT_ATTR_NOATIME|MOUNT_ATTR_STRICTATIME|MOUNT_ATTR_NOSYMFOLLOW|0x40, propagation=0 /* MS_??? */, userns_fd=0}, 32) = -1 EINVAL (Invalid argument) > > After further investigation, The issue was pinpointed to lack of Linux > kernel commit > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=95de4ad173ca0e61034f3145d66917970961c210 > ("fs: relax mount_setattr() permission checks") in the kernel version > that was being used. Above mitigation was discussed in email related to > then-rejected CVE-2024-26821: > https://lore.kernel.org/linux-cve-announce/2024051606-imaging-entrench-b327@gregkh/T/ > > After testing with qemuarm64 machine different linux-yocto versions, > it was observed that the issue impacts following versions of currently > supported LTS kernels: > - 6.6.17 (fixed since 6.6.18 i.e. mount_setattr() returns 0) > - 6.1.78 (fixed since 6.1.79 i.e. mount_setattr() returns 0) > - 5.15.164 which is currently the newest of 5.15.y series (i.e. no > known working version) > > Taking the above findings into consideration, add a new PACKAGECONFIG > option which allows to conveniently opt-out from prematurely using a > feature which can cause issues with a bit older kernels. > > Versions 5.10.223, 5.4.279 and 4.10.317 were also tested but the issue > was not reproduced with those versions - using strace showed that the > mount_setattr call associated with the new mount API problem was not > issued with these LTS kernel versions, which seemed to be confirmed > also by following libmount debug message in these cases: > > 415: libmount: HOOK: [0x7fa115e818]: failed to init new API > > Note: In addition to the aforementioned, this change was > tested also briefly using the current latest kernel versions 6.1.104, > 6.6.45 and 6.10.3 that using the old mount API with newest kernels > did not introduce any observable regression to the boot flow. > > Signed-off-by: Niko Mauno <niko.mauno@vaisala.com> > --- > meta/recipes-core/util-linux/util-linux_2.40.1.bb | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/meta/recipes-core/util-linux/util-linux_2.40.1.bb b/meta/recipes-core/util-linux/util-linux_2.40.1.bb > index a1aab94055..e87657cc70 100644 > --- a/meta/recipes-core/util-linux/util-linux_2.40.1.bb > +++ b/meta/recipes-core/util-linux/util-linux_2.40.1.bb > @@ -107,6 +107,13 @@ PACKAGECONFIG[cryptsetup] = "--with-cryptsetup,--without-cryptsetup,cryptsetup" > PACKAGECONFIG[chfn-chsh] = "--enable-chfn-chsh,--disable-chfn-chsh," > PACKAGECONFIG[selinux] = "--with-selinux,--without-selinux,libselinux" > PACKAGECONFIG[lastlog2] = "--enable-liblastlog2,--disable-liblastlog2,sqlite3" > +# Using the new file descriptors based mount kernel API can cause rootfs remount failure with some older kernels. > +# Of currently supported LTS kernels, it seems safe to enable this option with: > +# - version 6.6.18 and newer in the 6.6.y series. > +# - version 6.1.79 and newer in the 6.1.y series. > +# - with the 5.15.y series, versions till at least 5.15.164 can not use it. > +# - with 5.10.y, 5.4.y and 4.19.y series kernels libmount seemed to use the old API regardless of this option. > +PACKAGECONFIG[no-new-mount-api] = "--disable-libmount-mountfd-support,--enable-libmount-mountfd-support" > > EXTRA_OEMAKE = "ARCH=${TARGET_ARCH} CPU= CPUOPT= 'OPT=${CFLAGS}'" > > -- > 2.39.2 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#203494): https://lists.openembedded.org/g/openembedded-core/message/203494 > Mute This Topic: https://lists.openembedded.org/mt/107977417/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Sorry, the comments added in this v3 still adhere to v2 content. I will still submit v4 soon which amends to added comment lines. -Niko On 19.8.2024 12.29, Alexander Kanavin wrote: > Thanks, this version lgtm. > > Alex > > On Mon, 19 Aug 2024 at 11:27, Niko Mauno via lists.openembedded.org > <niko.mauno=vaisala.com@lists.openembedded.org> wrote: >> >> The 2.39 version of util-linux took new file descriptors based mount >> kernel API into use. In relation to this change, the upstream release >> notes in >> https://github.com/util-linux/util-linux/blob/v2.39/Documentation/releases/v2.39-ReleaseNotes#L14-L21 >> mention that >> >> This change is very aggressive to libmount code, but hopefully, it does not introduce regressions in traditional mount(8) behavior. >> >> After observing following failure when booting a board using a bit >> older 6.1 series kernel together with initramfs rootfs based boot flow >> >> [FAILED] Failed to start Remount Root and Kernel File Systems. >> See 'systemctl status systemd-remount-fs.service' for details. >> >> closer inspection revealed: >> >> demoboard ~ # systemctl status -l systemd-remount-fs.service >> x systemd-remount-fs.service - Remount Root and Kernel File Systems >> Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; enabled-runtime; preset: disabled) >> Active: failed (Result: exit-code) since Wed 2024-08-14 14:53:48 UTC; 1min 22s ago >> Docs: man:systemd-remount-fs.service(8) >> https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems >> Process: 76 ExecStart=/usr/lib/systemd/systemd-remount-fs (code=exited, status=1/FAILURE) >> Main PID: 76 (code=exited, status=1/FAILURE) >> >> Aug 14 14:53:48 demoboard systemd-remount-fs[76]: /usr/bin/mount for / exited with exit status 32. >> Aug 14 14:53:48 demoboard systemd-remount-fs[81]: mount: /: mount point not mounted or bad option. >> Aug 14 14:53:48 demoboard systemd-remount-fs[81]: dmesg(1) may have more information after failed mount system call. >> Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE >> Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'. >> Aug 14 14:53:48 demoboard systemd[1]: Failed to start Remount Root and Kernel File Systems. >> >> also consequentially, 'systemctl status' reported: >> >> State: degraded >> >> When issuing 'strace -ff mount -o remount /' the failure occurred at >> >> mount_setattr(3, "", AT_EMPTY_PATH, {attr_set=MOUNT_ATTR_RDONLY|MOUNT_ATTR_NOATIME|MOUNT_ATTR_NODIRATIME, attr_clr=MOUNT_ATTR_NOSUID|MOUNT_ATTR_NODEV|MOUNT_ATTR_NOEXEC|MOUNT_ATTR_NOATIME|MOUNT_ATTR_STRICTATIME|MOUNT_ATTR_NOSYMFOLLOW|0x40, propagation=0 /* MS_??? */, userns_fd=0}, 32) = -1 EINVAL (Invalid argument) >> >> After further investigation, The issue was pinpointed to lack of Linux >> kernel commit >> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=95de4ad173ca0e61034f3145d66917970961c210 >> ("fs: relax mount_setattr() permission checks") in the kernel version >> that was being used. Above mitigation was discussed in email related to >> then-rejected CVE-2024-26821: >> https://lore.kernel.org/linux-cve-announce/2024051606-imaging-entrench-b327@gregkh/T/ >> >> After testing with qemuarm64 machine different linux-yocto versions, >> it was observed that the issue impacts following versions of currently >> supported LTS kernels: >> - 6.6.17 (fixed since 6.6.18 i.e. mount_setattr() returns 0) >> - 6.1.78 (fixed since 6.1.79 i.e. mount_setattr() returns 0) >> - 5.15.164 which is currently the newest of 5.15.y series (i.e. no >> known working version) >> >> Taking the above findings into consideration, add a new PACKAGECONFIG >> option which allows to conveniently opt-out from prematurely using a >> feature which can cause issues with a bit older kernels. >> >> Versions 5.10.223, 5.4.279 and 4.10.317 were also tested but the issue >> was not reproduced with those versions - using strace showed that the >> mount_setattr call associated with the new mount API problem was not >> issued with these LTS kernel versions, which seemed to be confirmed >> also by following libmount debug message in these cases: >> >> 415: libmount: HOOK: [0x7fa115e818]: failed to init new API >> >> Note: In addition to the aforementioned, this change was >> tested also briefly using the current latest kernel versions 6.1.104, >> 6.6.45 and 6.10.3 that using the old mount API with newest kernels >> did not introduce any observable regression to the boot flow. >> >> Signed-off-by: Niko Mauno <niko.mauno@vaisala.com> >> --- >> meta/recipes-core/util-linux/util-linux_2.40.1.bb | 7 +++++++ >> 1 file changed, 7 insertions(+) >> >> diff --git a/meta/recipes-core/util-linux/util-linux_2.40.1.bb b/meta/recipes-core/util-linux/util-linux_2.40.1.bb >> index a1aab94055..e87657cc70 100644 >> --- a/meta/recipes-core/util-linux/util-linux_2.40.1.bb >> +++ b/meta/recipes-core/util-linux/util-linux_2.40.1.bb >> @@ -107,6 +107,13 @@ PACKAGECONFIG[cryptsetup] = "--with-cryptsetup,--without-cryptsetup,cryptsetup" >> PACKAGECONFIG[chfn-chsh] = "--enable-chfn-chsh,--disable-chfn-chsh," >> PACKAGECONFIG[selinux] = "--with-selinux,--without-selinux,libselinux" >> PACKAGECONFIG[lastlog2] = "--enable-liblastlog2,--disable-liblastlog2,sqlite3" >> +# Using the new file descriptors based mount kernel API can cause rootfs remount failure with some older kernels. >> +# Of currently supported LTS kernels, it seems safe to enable this option with: >> +# - version 6.6.18 and newer in the 6.6.y series. >> +# - version 6.1.79 and newer in the 6.1.y series. >> +# - with the 5.15.y series, versions till at least 5.15.164 can not use it. >> +# - with 5.10.y, 5.4.y and 4.19.y series kernels libmount seemed to use the old API regardless of this option. >> +PACKAGECONFIG[no-new-mount-api] = "--disable-libmount-mountfd-support,--enable-libmount-mountfd-support" >> >> EXTRA_OEMAKE = "ARCH=${TARGET_ARCH} CPU= CPUOPT= 'OPT=${CFLAGS}'" >> >> -- >> 2.39.2 >> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#203494): https://lists.openembedded.org/g/openembedded-core/message/203494 >> Mute This Topic: https://lists.openembedded.org/mt/107977417/1686489 >> Group Owner: openembedded-core+owner@lists.openembedded.org >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub >> -=-=-=-=-=-=-=-=-=-=-=- >>
diff --git a/meta/recipes-core/util-linux/util-linux_2.40.1.bb b/meta/recipes-core/util-linux/util-linux_2.40.1.bb index a1aab94055..e87657cc70 100644 --- a/meta/recipes-core/util-linux/util-linux_2.40.1.bb +++ b/meta/recipes-core/util-linux/util-linux_2.40.1.bb @@ -107,6 +107,13 @@ PACKAGECONFIG[cryptsetup] = "--with-cryptsetup,--without-cryptsetup,cryptsetup" PACKAGECONFIG[chfn-chsh] = "--enable-chfn-chsh,--disable-chfn-chsh," PACKAGECONFIG[selinux] = "--with-selinux,--without-selinux,libselinux" PACKAGECONFIG[lastlog2] = "--enable-liblastlog2,--disable-liblastlog2,sqlite3" +# Using the new file descriptors based mount kernel API can cause rootfs remount failure with some older kernels. +# Of currently supported LTS kernels, it seems safe to enable this option with: +# - version 6.6.18 and newer in the 6.6.y series. +# - version 6.1.79 and newer in the 6.1.y series. +# - with the 5.15.y series, versions till at least 5.15.164 can not use it. +# - with 5.10.y, 5.4.y and 4.19.y series kernels libmount seemed to use the old API regardless of this option. +PACKAGECONFIG[no-new-mount-api] = "--disable-libmount-mountfd-support,--enable-libmount-mountfd-support" EXTRA_OEMAKE = "ARCH=${TARGET_ARCH} CPU= CPUOPT= 'OPT=${CFLAGS}'"
The 2.39 version of util-linux took new file descriptors based mount kernel API into use. In relation to this change, the upstream release notes in https://github.com/util-linux/util-linux/blob/v2.39/Documentation/releases/v2.39-ReleaseNotes#L14-L21 mention that This change is very aggressive to libmount code, but hopefully, it does not introduce regressions in traditional mount(8) behavior. After observing following failure when booting a board using a bit older 6.1 series kernel together with initramfs rootfs based boot flow [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. closer inspection revealed: demoboard ~ # systemctl status -l systemd-remount-fs.service x systemd-remount-fs.service - Remount Root and Kernel File Systems Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; enabled-runtime; preset: disabled) Active: failed (Result: exit-code) since Wed 2024-08-14 14:53:48 UTC; 1min 22s ago Docs: man:systemd-remount-fs.service(8) https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems Process: 76 ExecStart=/usr/lib/systemd/systemd-remount-fs (code=exited, status=1/FAILURE) Main PID: 76 (code=exited, status=1/FAILURE) Aug 14 14:53:48 demoboard systemd-remount-fs[76]: /usr/bin/mount for / exited with exit status 32. Aug 14 14:53:48 demoboard systemd-remount-fs[81]: mount: /: mount point not mounted or bad option. Aug 14 14:53:48 demoboard systemd-remount-fs[81]: dmesg(1) may have more information after failed mount system call. Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE Aug 14 14:53:48 demoboard systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'. Aug 14 14:53:48 demoboard systemd[1]: Failed to start Remount Root and Kernel File Systems. also consequentially, 'systemctl status' reported: State: degraded When issuing 'strace -ff mount -o remount /' the failure occurred at mount_setattr(3, "", AT_EMPTY_PATH, {attr_set=MOUNT_ATTR_RDONLY|MOUNT_ATTR_NOATIME|MOUNT_ATTR_NODIRATIME, attr_clr=MOUNT_ATTR_NOSUID|MOUNT_ATTR_NODEV|MOUNT_ATTR_NOEXEC|MOUNT_ATTR_NOATIME|MOUNT_ATTR_STRICTATIME|MOUNT_ATTR_NOSYMFOLLOW|0x40, propagation=0 /* MS_??? */, userns_fd=0}, 32) = -1 EINVAL (Invalid argument) After further investigation, The issue was pinpointed to lack of Linux kernel commit https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=95de4ad173ca0e61034f3145d66917970961c210 ("fs: relax mount_setattr() permission checks") in the kernel version that was being used. Above mitigation was discussed in email related to then-rejected CVE-2024-26821: https://lore.kernel.org/linux-cve-announce/2024051606-imaging-entrench-b327@gregkh/T/ After testing with qemuarm64 machine different linux-yocto versions, it was observed that the issue impacts following versions of currently supported LTS kernels: - 6.6.17 (fixed since 6.6.18 i.e. mount_setattr() returns 0) - 6.1.78 (fixed since 6.1.79 i.e. mount_setattr() returns 0) - 5.15.164 which is currently the newest of 5.15.y series (i.e. no known working version) Taking the above findings into consideration, add a new PACKAGECONFIG option which allows to conveniently opt-out from prematurely using a feature which can cause issues with a bit older kernels. Versions 5.10.223, 5.4.279 and 4.10.317 were also tested but the issue was not reproduced with those versions - using strace showed that the mount_setattr call associated with the new mount API problem was not issued with these LTS kernel versions, which seemed to be confirmed also by following libmount debug message in these cases: 415: libmount: HOOK: [0x7fa115e818]: failed to init new API Note: In addition to the aforementioned, this change was tested also briefly using the current latest kernel versions 6.1.104, 6.6.45 and 6.10.3 that using the old mount API with newest kernels did not introduce any observable regression to the boot flow. Signed-off-by: Niko Mauno <niko.mauno@vaisala.com> --- meta/recipes-core/util-linux/util-linux_2.40.1.bb | 7 +++++++ 1 file changed, 7 insertions(+)