| Message ID | 2cef35c707ab0d985578ca4d042cfb33902dbcb5.1737894876.git.joerg.sommer@navimatix.de |
|---|---|
| State | Rejected |
| Headers | show |
| Series | doc/wic,README.hardware: add dd option conv=sparse | expand |
On Sun, 2025-01-26 at 13:37 +0100, Jörg Sommer via lists.yoctoproject.org wrote: > From: Jörg Sommer <joerg.sommer@navimatix.de> > > The GNU implementation of dd supports the skipping of sparse blocks which > can speed-up the write, esp. of wic images with big holes: > > $ sudo dd if=image.wic of=/dev/sdb bs=2M > 2683+1 Datensätze ein > 2683+1 Datensätze aus > 5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 84,2773 s, 66,8 MB/s > > $ sudo dd if=image.wic of=/dev/sdb bs=2M conv=sparse > 2683+1 Datensätze ein > 2683+1 Datensätze aus > 5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 11,3314 s, 497 MB/s > > Signed-off-by: Jörg Sommer <joerg.sommer@navimatix.de> > --- > documentation/dev-manual/wic.rst | 8 +++++++- > meta-yocto-bsp/README.hardware.md | 20 ++++++++++++++------ > 2 files changed, 21 insertions(+), 7 deletions(-) I'm torn on this as it is potentially dangerous. It depends whether the blocks really need to be zero or their contents doesn't matter. You could leave existing data behind if you use the sparse option. Cheers, Richard
Richard Purdie schrieb am So 26. Jan, 12:53 (+0000): > On Sun, 2025-01-26 at 13:37 +0100, Jörg Sommer via lists.yoctoproject.org wrote: > > From: Jörg Sommer <joerg.sommer@navimatix.de> > > > > The GNU implementation of dd supports the skipping of sparse blocks which > > can speed-up the write, esp. of wic images with big holes: > > > > $ sudo dd if=image.wic of=/dev/sdb bs=2M > > 2683+1 Datensätze ein > > 2683+1 Datensätze aus > > 5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 84,2773 s, 66,8 MB/s > > > > $ sudo dd if=image.wic of=/dev/sdb bs=2M conv=sparse > > 2683+1 Datensätze ein > > 2683+1 Datensätze aus > > 5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 11,3314 s, 497 MB/s > > > > Signed-off-by: Jörg Sommer <joerg.sommer@navimatix.de> > > --- > > documentation/dev-manual/wic.rst | 8 +++++++- > > meta-yocto-bsp/README.hardware.md | 20 ++++++++++++++------ > > 2 files changed, 21 insertions(+), 7 deletions(-) > > I'm torn on this as it is potentially dangerous. It depends whether the > blocks really need to be zero or their contents doesn't matter. You > could leave existing data behind if you use the sparse option. Isn't this the same what bmaptool does? https://github.com/yoctoproject/bmaptool?tab=readme-ov-file#concept Bye, Jörg
Le dim. 26 janv. 2025 à 16:25, Jörg Sommer via lists.yoctoproject.org <joerg.sommer=navimatix.de@lists.yoctoproject.org> a écrit : > Richard Purdie schrieb am So 26. Jan, 12:53 (+0000): > > On Sun, 2025-01-26 at 13:37 +0100, Jörg Sommer via > lists.yoctoproject.org wrote: > > > From: Jörg Sommer <joerg.sommer@navimatix.de> > > > > > > The GNU implementation of dd supports the skipping of sparse blocks > which > > > can speed-up the write, esp. of wic images with big holes: > > > > > > $ sudo dd if=image.wic of=/dev/sdb bs=2M > > > 2683+1 Datensätze ein > > > 2683+1 Datensätze aus > > > 5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 84,2773 s, 66,8 MB/s > > > > > > $ sudo dd if=image.wic of=/dev/sdb bs=2M conv=sparse > > > 2683+1 Datensätze ein > > > 2683+1 Datensätze aus > > > 5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 11,3314 s, 497 MB/s > > > > > > Signed-off-by: Jörg Sommer <joerg.sommer@navimatix.de> > > > --- > > > documentation/dev-manual/wic.rst | 8 +++++++- > > > meta-yocto-bsp/README.hardware.md | 20 ++++++++++++++------ > > > 2 files changed, 21 insertions(+), 7 deletions(-) > > > > I'm torn on this as it is potentially dangerous. It depends whether the > > blocks really need to be zero or their contents doesn't matter. You > > could leave existing data behind if you use the sparse option. > > Isn't this the same what bmaptool does? > https://github.com/yoctoproject/bmaptool?tab=readme-ov-file#concept > It look it does not. bmaptool readme is very particular on what a "unmapped block" is. And is not a zero block. TL;DR: (a gross reinterpretation) that is a block not referenced anywhere in your filesystem. Imagine a file filled with zeroes on your filesystem. Those zeros are referenced so must be flashed. bmaptool will flash them. "dd conv=sparse" will not and this is only right in a corner case : if you already know that your disk/memory is zero filled. But, in the general case, you'll get a "zero file" filled with garbage. Sorry, but I'm against this change since it will be a source of weird/hard bugs. Maybe instead of changing the dd command to speedup we can push users to "bmaptool copy" (we will get speedup without compromising on the validity of the filesystem)? Regards,
Hi all, On Sun Jan 26, 2025 at 5:56 PM CET, Yoann Congal via lists.yoctoproject.org wrote: > Le dim. 26 janv. 2025 à 16:25, Jörg Sommer via lists.yoctoproject.org > <joerg.sommer=navimatix.de@lists.yoctoproject.org> a écrit : > >> Richard Purdie schrieb am So 26. Jan, 12:53 (+0000): >> > On Sun, 2025-01-26 at 13:37 +0100, Jörg Sommer via >> lists.yoctoproject.org wrote: >> > > From: Jörg Sommer <joerg.sommer@navimatix.de> >> > > >> > > The GNU implementation of dd supports the skipping of sparse blocks >> which >> > > can speed-up the write, esp. of wic images with big holes: >> > > >> > > $ sudo dd if=image.wic of=/dev/sdb bs=2M >> > > 2683+1 Datensätze ein >> > > 2683+1 Datensätze aus >> > > 5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 84,2773 s, 66,8 MB/s >> > > >> > > $ sudo dd if=image.wic of=/dev/sdb bs=2M conv=sparse >> > > 2683+1 Datensätze ein >> > > 2683+1 Datensätze aus >> > > 5628735488 Bytes (5,6 GB, 5,2 GiB) kopiert, 11,3314 s, 497 MB/s >> > > >> > > Signed-off-by: Jörg Sommer <joerg.sommer@navimatix.de> >> > > --- >> > > documentation/dev-manual/wic.rst | 8 +++++++- >> > > meta-yocto-bsp/README.hardware.md | 20 ++++++++++++++------ >> > > 2 files changed, 21 insertions(+), 7 deletions(-) >> > >> > I'm torn on this as it is potentially dangerous. It depends whether the >> > blocks really need to be zero or their contents doesn't matter. You >> > could leave existing data behind if you use the sparse option. >> >> Isn't this the same what bmaptool does? >> https://github.com/yoctoproject/bmaptool?tab=readme-ov-file#concept >> > > It look it does not. bmaptool readme is very particular on what a "unmapped > block" is. And is not a zero block. > TL;DR: (a gross reinterpretation) that is a block not referenced anywhere > in your filesystem. > > Imagine a file filled with zeroes on your filesystem. Those zeros are > referenced so must be flashed. bmaptool will flash them. > "dd conv=sparse" will not and this is only right in a corner case : if you > already know that your disk/memory is zero filled. > But, in the general case, you'll get a "zero file" filled with garbage. > > Sorry, but I'm against this change since it will be a source of weird/hard > bugs. > Maybe instead of changing the dd command to speedup we can push users to > "bmaptool copy" (we will get speedup without compromising on the validity > of the filesystem)? I agree with Yoann, I'm not for providing conv=sparse as a default option for dd. I also vote for redirecting to bmaptool. This can be done with: :ref:`bmaptool <dev-manual/bmaptool:flashing images using \`bmaptool\`>` I think in a `..note ::` like you did Jörg, would be good. Antonin
diff --git a/documentation/dev-manual/wic.rst b/documentation/dev-manual/wic.rst index fced0e170c..792fa35fa7 100644 --- a/documentation/dev-manual/wic.rst +++ b/documentation/dev-manual/wic.rst @@ -507,7 +507,13 @@ built your image, and boot from the media. You can write the image by using or :: - $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX + $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX bs=2M conv=sparse + +.. note:: + + The option ``conv=sparse`` is a GNU extension to skip writing of blocks + with purely zeros. If your ``dd`` (e.g. BusyBox) does not support it, you + can safely remove this option. .. note:: diff --git a/meta-yocto-bsp/README.hardware.md b/meta-yocto-bsp/README.hardware.md index d2cc05f909..a824963e4e 100644 --- a/meta-yocto-bsp/README.hardware.md +++ b/meta-yocto-bsp/README.hardware.md @@ -91,7 +91,7 @@ Depending on the device, it can boot from a traditional hard-disk, a USB device, or over the network. Writing generated images to physical media is straightforward with a caveat for USB devices. The following examples assume the target boot device is /dev/sdb, be sure to verify this and use the correct -device as the following commands are run as root and are not reversable. +device as the following commands are run as root and are not reversible. USB Device: @@ -105,7 +105,11 @@ USB Device: 2. Use the `dd` utility to write the image to the raw block device. For example: - # dd if=core-image-minimal-genericx86.hddimg of=/dev/sdb + # dd if=core-image-minimal-genericx86.hddimg of=/dev/sdb bs=2M conv=sparse + + The option `conv=sparse` is a GNU extension to skip writing of blocks with + purely zeros. If your `dd` doesn't support it, you can safely drop this + option. If the device fails to boot with "Boot error" displayed, or apparently stops just after the SYSLINUX version banner, it is likely the BIOS cannot @@ -119,9 +123,11 @@ USB Device: 2. Use a ".wic" image with an EFI partition - 1. With a default grub-efi bootloader: + 1. With a default grub-efi bootloader (`conv=sparse` is a GNU extension to + skip empty blocks for speed up; remove it, if your `dd` doesn't support + it): - # dd if=core-image-minimal-genericx86-64.wic of=/dev/sdb + # dd if=core-image-minimal-genericx86-64.wic of=/dev/sdb bs=2M conv=sparse 2. Use systemd-boot instead. Build an image with `EFI_PROVIDER="systemd-boot"` then use the above `dd` command to write the image to a USB stick. @@ -179,8 +185,10 @@ From a Linux system with access to the image files perform the following steps: $ bitbake core-image-minimal - 2. Use the "dd" utility to write the image to the SD card. For example: + 2. Use the `dd` utility to write the image to the SD card. For example + (`conv=sparse` is a GNU extension to skip empty blocks for speed up; remove + it, if your `dd` doesn't support it): - # dd if=core-image-minimal-beaglebone-yocto.wic of=/dev/sdb + # dd if=core-image-minimal-beaglebone-yocto.wic of=/dev/sdb bs=2M conv=sparse 3. Insert the SD card into the Beaglebone and boot the board.