| Message ID | 20250905164136.2896809-1-vince@underview.tech |
|---|---|
| State | New |
| Headers | show |
| Series | [v3,1/8] bootimg_pcbios: support grub hybrid boot | expand |
On Fri Sep 5, 2025 at 6:41 PM CEST, Vincent Davis Jr wrote: > If caller wants to primarily leverage > grub as the primary boot loader current > oe-core doesn't support booting grub when > the boot firmware is legacy BIOS based and > the partition table format is GPT based. > > Issue GPT header reside where core.img > should be located (at byte 512). > > To navigate around issue core.img was > moved to a seperate partition. > > If disk is a GPT disk caller must specify > the file system type as none and set the > partition type as BIOS boot. No filesystem > will be created on partition. This also > allows wic plugin to know where to dd > core.img. > > Unfortunately No deep dive into grub-install > was perform to know the exact bytes changed > in hybrid boot case. To see the change in > bytes generated boot.img + core.img was compared > to boot.img + core.img after grub-install was > executed using the xxd command. > > Signed-off-by: Vincent Davis Jr <vince@underview.tech> > --- Hi Vincent, We still have the same failure as on v2. Still happening with selftests running on debian host, so there might be something related to the host configuration. ERROR: Qemu log output from /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2287526/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0/testimage/qemu_boot_log.20250907083922: SeaBIOS (version rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org) ... iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+0EFC6560+0EF26560 CA00 Press Ctrl-B to configure iPXE (PCI 00:02.0)... ... Booting from Hard Disk.. 2025-09-07 08:56:24,256 - oe-selftest - INFO - 9: 79/94 577/639 (1085.73s) (0 failed) (wic.Wic2.test_grub_install_biosplusefi_qemu) 2025-09-07 08:56:24,265 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 390, in runqemu qemu.start(params=qemuparams, ssh=ssh, runqemuparams=runqemuparams, launch_cmd=launch_cmd, discard_writes=discard_writes) File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/targetcontrol.py", line 179, in start raise RuntimeError("%s - FAILED to start qemu - check the task log and the boot log" % self.pn) RuntimeError: core-image-minimal - FAILED to start qemu - check the task log and the boot log https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2280 Thanks, Mathieu
Okay Is there anyway I can see qemu_boot_log.20250907083922? Also assuming we are running in a docker container. May I see the Docker file. Ran v3 against latest poky using an ubuntu docker container after cherry picking v3 commits into it. DISTRO = "poky" MACHINE = "qemux86-64" I believe not near computer. My docker file and docker compose file https://github.com/under-view/docker-builds/blob/master/containers/yocto-project/ubuntu-24.04/Dockerfile https://github.com/under-view/docker-builds/blob/master/composes/default-compose.yaml On Sun, Sep 7, 2025, 10:02 AM Mathieu Dubois-Briand < mathieu.dubois-briand@bootlin.com> wrote: > On Fri Sep 5, 2025 at 6:41 PM CEST, Vincent Davis Jr wrote: > > If caller wants to primarily leverage > > grub as the primary boot loader current > > oe-core doesn't support booting grub when > > the boot firmware is legacy BIOS based and > > the partition table format is GPT based. > > > > Issue GPT header reside where core.img > > should be located (at byte 512). > > > > To navigate around issue core.img was > > moved to a seperate partition. > > > > If disk is a GPT disk caller must specify > > the file system type as none and set the > > partition type as BIOS boot. No filesystem > > will be created on partition. This also > > allows wic plugin to know where to dd > > core.img. > > > > Unfortunately No deep dive into grub-install > > was perform to know the exact bytes changed > > in hybrid boot case. To see the change in > > bytes generated boot.img + core.img was compared > > to boot.img + core.img after grub-install was > > executed using the xxd command. > > > > Signed-off-by: Vincent Davis Jr <vince@underview.tech> > > --- > > Hi Vincent, > > We still have the same failure as on v2. Still happening with selftests > running on debian host, so there might be something related to the host > configuration. > > ERROR: Qemu log output from > /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2287526/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0/testimage/qemu_boot_log.20250907083922: > SeaBIOS (version rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org) > ... > iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+0EFC6560+0EF26560 CA00 > Press Ctrl-B to configure iPXE (PCI 00:02.0)... > ... > Booting from Hard Disk.. > 2025-09-07 08:56:24,256 - oe-selftest - INFO - 9: 79/94 577/639 (1085.73s) > (0 failed) (wic.Wic2.test_grub_install_biosplusefi_qemu) > 2025-09-07 08:56:24,265 - oe-selftest - INFO - > testtools.testresult.real._StringException: Traceback (most recent call > last): > File > "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", > line 390, in runqemu > qemu.start(params=qemuparams, ssh=ssh, runqemuparams=runqemuparams, > launch_cmd=launch_cmd, discard_writes=discard_writes) > File > "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/targetcontrol.py", > line 179, in start > raise RuntimeError("%s - FAILED to start qemu - check the task log and > the boot log" % self.pn) > RuntimeError: core-image-minimal - FAILED to start qemu - check the task > log and the boot log > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2280 > > Thanks, > Mathieu > > -- > Mathieu Dubois-Briand, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > >
Hi Vincent, Here is the log file, but there isn't much. About build host, this is not a docker, so I believe I have no easy way to provide you with a reproducible setup. Configuration can be seen here: https://autobuilder.yoctoproject.org/valkyrie/api/v2/logs/3445173/raw_inline Best regards, Mathieu On Sun Sep 7, 2025 at 5:47 PM CEST, Vincent Davis Jr wrote: > Okay > > Is there anyway I can see > > qemu_boot_log.20250907083922? > > Also assuming we are running in a docker container. > > May I see the Docker file. > > Ran v3 against latest poky using an ubuntu docker container after cherry > picking v3 commits into it. > > DISTRO = "poky" > MACHINE = "qemux86-64" I believe not near computer. > > My docker file and docker compose file > > https://github.com/under-view/docker-builds/blob/master/containers/yocto-project/ubuntu-24.04/Dockerfile > > https://github.com/under-view/docker-builds/blob/master/composes/default-compose.yaml > > > > > > > On Sun, Sep 7, 2025, 10:02 AM Mathieu Dubois-Briand < > mathieu.dubois-briand@bootlin.com> wrote: > >> On Fri Sep 5, 2025 at 6:41 PM CEST, Vincent Davis Jr wrote: >> > If caller wants to primarily leverage >> > grub as the primary boot loader current >> > oe-core doesn't support booting grub when >> > the boot firmware is legacy BIOS based and >> > the partition table format is GPT based. >> > >> > Issue GPT header reside where core.img >> > should be located (at byte 512). >> > >> > To navigate around issue core.img was >> > moved to a seperate partition. >> > >> > If disk is a GPT disk caller must specify >> > the file system type as none and set the >> > partition type as BIOS boot. No filesystem >> > will be created on partition. This also >> > allows wic plugin to know where to dd >> > core.img. >> > >> > Unfortunately No deep dive into grub-install >> > was perform to know the exact bytes changed >> > in hybrid boot case. To see the change in >> > bytes generated boot.img + core.img was compared >> > to boot.img + core.img after grub-install was >> > executed using the xxd command. >> > >> > Signed-off-by: Vincent Davis Jr <vince@underview.tech> >> > --- >> >> Hi Vincent, >> >> We still have the same failure as on v2. Still happening with selftests >> running on debian host, so there might be something related to the host >> configuration. >> >> ERROR: Qemu log output from >> /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2287526/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0/testimage/qemu_boot_log.20250907083922: >> SeaBIOS (version rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org) >> ... >> iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+0EFC6560+0EF26560 CA00 >> Press Ctrl-B to configure iPXE (PCI 00:02.0)... >> ... >> Booting from Hard Disk.. >> 2025-09-07 08:56:24,256 - oe-selftest - INFO - 9: 79/94 577/639 (1085.73s) >> (0 failed) (wic.Wic2.test_grub_install_biosplusefi_qemu) >> 2025-09-07 08:56:24,265 - oe-selftest - INFO - >> testtools.testresult.real._StringException: Traceback (most recent call >> last): >> File >> "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", >> line 390, in runqemu >> qemu.start(params=qemuparams, ssh=ssh, runqemuparams=runqemuparams, >> launch_cmd=launch_cmd, discard_writes=discard_writes) >> File >> "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/targetcontrol.py", >> line 179, in start >> raise RuntimeError("%s - FAILED to start qemu - check the task log and >> the boot log" % self.pn) >> RuntimeError: core-image-minimal - FAILED to start qemu - check the task >> log and the boot log >> >> https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2280 >> >> Thanks, >> Mathieu >> >> -- >> Mathieu Dubois-Briand, Bootlin >> Embedded Linux and Kernel engineering >> https://bootlin.com >> >>
This issue has been hard to reproduce. Even when I manually build the same SEABIOS version. Everything boots for me when manually trying with bellow command qemu-system-x86_64 \ -bios seabios/out/bios.bin \ -nographic \ -chardev stdio,id=char0,mux=on,signal=off \ -mon chardev=char0,mode=readline \ -serial chardev:char0 \ -drive file=vm-storage/emmc-wic-udoo-bolt-emmc.rootfs.wic,format=raw,if=virtio Is there anyway with v3 to soley run oe-selftest --verbose -r wic.Wic2.test_grub_install_biosplusefi_qemu. And paste output?
On Sat Sep 13, 2025 at 6:29 AM CEST, Vincent Davis Jr wrote: > This issue has been hard to reproduce. > > Even when I manually build the same SEABIOS version. > > Everything boots for me when manually trying with bellow command > > qemu-system-x86_64 \ > -bios seabios/out/bios.bin \ > -nographic \ > -chardev stdio,id=char0,mux=on,signal=off \ > -mon chardev=char0,mode=readline \ > -serial chardev:char0 \ > -drive file=vm-storage/emmc-wic-udoo-bolt-emmc.rootfs.wic,format=raw,if=virtio > > Is there anyway with v3 to soley run > > oe-selftest --verbose -r wic.Wic2.test_grub_install_biosplusefi_qemu. > > And paste output? Hi Vincent, I ran the selftest manually and captured the output in attached log. I'm also attaching the qemu_boot_log.20250915113946 file, but it's pretty short. I tried to capture most of the build-renamed-st directory, please tell me if there is any file you need. Thanks, Mathieu
diff --git a/scripts/lib/wic/plugins/source/bootimg_pcbios.py b/scripts/lib/wic/plugins/source/bootimg_pcbios.py index caabda6318..a7fb2eca4e 100644 --- a/scripts/lib/wic/plugins/source/bootimg_pcbios.py +++ b/scripts/lib/wic/plugins/source/bootimg_pcbios.py @@ -101,7 +101,7 @@ class BootimgPcbiosPlugin(SourcePlugin): disk_name, full_path, disk.min_size) if cls.loader == 'grub': - cls._do_install_grub(creator, kernel_dir, + cls._do_install_grub(disk, creator, kernel_dir, native_sysroot, full_path) elif cls.loader == 'syslinux': cls._do_install_syslinux(creator, bootimg_dir, @@ -340,6 +340,12 @@ class BootimgPcbiosPlugin(SourcePlugin): @classmethod def _do_configure_grub(cls, part, creator, cr_workdir): + # If partition type is either EFI System or + # BIOS boot no need to generate/copy grub config. + if part.part_type == '21686148-6449-6E6F-744E-656564454649' or \ + part.part_type == 'C12A7328-F81F-11D2-BA4B-00A0C93EC93B': + return 0 + hdddir = "%s/hdd" % cr_workdir bootloader = creator.ks.bootloader @@ -438,6 +444,12 @@ class BootimgPcbiosPlugin(SourcePlugin): grub_mods_path, core_img, builtin_modules) exec_native_cmd(grub_mkimage, native_sysroot) + # If partition type is either EFI System or + # BIOS boot no need to copy in grub modules. + if part.part_type == '21686148-6449-6E6F-744E-656564454649' or \ + part.part_type == 'C12A7328-F81F-11D2-BA4B-00A0C93EC93B': + return 0 + # Copy grub modules install_dir = '%s/%s/%s' % (hdddir, grub_prefix_path, grub_format) os.makedirs(install_dir, exist_ok=True) @@ -454,7 +466,7 @@ class BootimgPcbiosPlugin(SourcePlugin): native_sysroot, False) @classmethod - def _do_install_grub(cls, creator, kernel_dir, + def _do_install_grub(cls, disk, creator, kernel_dir, native_sysroot, full_path): core_img = '%s/grub-bios-core.img' % (kernel_dir) @@ -478,6 +490,45 @@ class BootimgPcbiosPlugin(SourcePlugin): # Install core.img or grub stage 1.5 dd_cmd = "dd if=%s of=%s conv=notrunc bs=1 seek=512" % (core_img, full_path) exec_cmd(dd_cmd, native_sysroot) + elif creator.ptable_format == 'gpt': + # Issue GPT headers reside where core.img should be (at byte 512). + # To navigate around issue core.img was moved to a seperate partition. + # + # If disk is a GPT disk caller must specify the file system + # type as none. As no filesystem may be created. Caller must + # also set the partition type to BIOS boot. So, the plugin + # may embed core.img there. + + # Replicates what grub-install does to boot.img + # Found by comparing xxd output of generated boot.img + # to boot.img after grub-install. + dd_cmd = "echo -ne '\\x00\\x08' | dd of=%s conv=notrunc bs=1 count=2 seek=92" % (full_path) + exec_native_cmd(dd_cmd, native_sysroot) + + dd_cmd = "echo -ne '\\x90\\x90' | dd of=%s conv=notrunc bs=1 count=2 seek=102" % (full_path) + exec_native_cmd(dd_cmd, native_sysroot) + + for part in creator.parts: + if part.part_type == '21686148-6449-6E6F-744E-656564454649': + part_start_byte = part.start * disk.sector_size + + # Install core.img or grub stage 1.5 + dd_cmd = "dd if=%s of=%s conv=notrunc bs=1 seek=%s" % \ + (core_img, full_path, part_start_byte) + exec_cmd(dd_cmd, native_sysroot) + + # Replicates what grub-install does to core.img + # Found by comparing xxd output of generated + # core.img to core.img ater grub install. + dd_cmd = "echo -ne '\\x01\\x08' | dd of=%s conv=notrunc bs=1 count=2 seek=%d" % \ + (full_path, part_start_byte + 500) + exec_native_cmd(dd_cmd, native_sysroot) + + dd_cmd = "echo -ne '\\x2f\\x02' | dd of=%s conv=notrunc bs=1 count=2 seek=%d" % \ + (full_path, part_start_byte + 508) + exec_native_cmd(dd_cmd, native_sysroot) + + break else: raise WicError("Unsupported partition table: %s" % creator.ptable_format)
If caller wants to primarily leverage grub as the primary boot loader current oe-core doesn't support booting grub when the boot firmware is legacy BIOS based and the partition table format is GPT based. Issue GPT header reside where core.img should be located (at byte 512). To navigate around issue core.img was moved to a seperate partition. If disk is a GPT disk caller must specify the file system type as none and set the partition type as BIOS boot. No filesystem will be created on partition. This also allows wic plugin to know where to dd core.img. Unfortunately No deep dive into grub-install was perform to know the exact bytes changed in hybrid boot case. To see the change in bytes generated boot.img + core.img was compared to boot.img + core.img after grub-install was executed using the xxd command. Signed-off-by: Vincent Davis Jr <vince@underview.tech> --- .../lib/wic/plugins/source/bootimg_pcbios.py | 55 ++++++++++++++++++- 1 file changed, 53 insertions(+), 2 deletions(-)