| Message ID | 20260730091017.171702-1-ecordonnier@snap.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | [v2,1/2] package_manager/ipk: skip checksums for unsigned local feeds | expand |
On Thu, 2026-07-30 at 11:10 +0200, Etienne Cordonnier via lists.openembedded.org wrote: > From: Etienne Cordonnier <ecordonnier@snap.com> > > Computing checksums in the opkg Packages index requires reading every > .ipk file in full. For a large image with 7000+ packages (including > multi-gigabyte debug packages), this adds 150-300s to every do_rootfs run. > > Checksums in the Packages index are only meaningful for signed feeds > (PACKAGE_FEED_SIGN=1): the GPG signature covers the Packages index > which contains the SHA256Sum of each .ipk, forming a chain of trust > that prevents tampered packages being swapped on a remote feed. > > For unsigned local file:// feeds the packages are installed directly > from the build host filesystem where there is no tampering risk. Skip > all checksum generation in that case by passing no --checksum flags to > opkg-make-index (the tool's default behaviour when given no flags). > Pass --force-checksum to opkg so it does not error on the absent > checksum fields. > > On a test image with 7000+ packages (including a 2.3 GB debug > package): write_index time reduced from ~180s to ~22s (8x speedup) > when opkg-make-index is configured to produce no checksums by default. > > See https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796 > ("Enable sha256 checksums in opkg indexer", 2019) for the original rationale. > > AI-Generated: Claude Sonnet 4.6 > Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> Hi, The package feed may be made available over a network and/or externally signed after it has been generated, so I don't think we should disable writing checksums into the feed by default. How about adding a variable to control this? It can be enabled by default in OE-core and you can disable it in a local or distro config if you know the signatures won't be needed in your use case. Best regards,
Hi Paul, my understanding is that my change does not impact the package feed. Checksums are still always calculated for the normal package feed after my change. There are two different code paths: 1. do_package_index (package-index.bb -> generate_index_files() -> OpkgIndexer(d, DEPLOY_DIR_IPK).write_index()): operates on the global DEPLOY_DIR_IPK. This is the feed that can be published, externally signed, or served over a network. This path is not touched by this commit: it still always writes checksums. This is the write_index() call at meta/lib/oe/package_manager/__init__.py:579 2. do_rootfs (OpkgPM.write_index()): operates on WORKDIR/oe-rootfs-repo, a throwaway local copy of packages assembled exclusively for the current rootfs build. This directory is never published or signed. This is the only path the commit changes. This is the write_index() call at meta/lib/oe/package_manager/ipk/rootfs.py:281 Or are you saying that there is a use-case where e.g. core-image-minimal/1.0/oe-rootfs-repo gets published over the network, rather than build/tmp/deploy/ipk? I was referring to yocto_docs where the example given is publishing packages from tmp/deploy: $ cd bitbake-builds/build/tmp/deploy/rpm $ python3 -m http.server Étienne On Thu, Jul 30, 2026 at 1:51 PM Paul Barker <paul@pbarker.dev> wrote: > On Thu, 2026-07-30 at 11:10 +0200, Etienne Cordonnier via > lists.openembedded.org wrote: > > From: Etienne Cordonnier <ecordonnier@snap.com> > > > > Computing checksums in the opkg Packages index requires reading every > > .ipk file in full. For a large image with 7000+ packages (including > > multi-gigabyte debug packages), this adds 150-300s to every do_rootfs > run. > > > > Checksums in the Packages index are only meaningful for signed feeds > > (PACKAGE_FEED_SIGN=1): the GPG signature covers the Packages index > > which contains the SHA256Sum of each .ipk, forming a chain of trust > > that prevents tampered packages being swapped on a remote feed. > > > > For unsigned local file:// feeds the packages are installed directly > > from the build host filesystem where there is no tampering risk. Skip > > all checksum generation in that case by passing no --checksum flags to > > opkg-make-index (the tool's default behaviour when given no flags). > > Pass --force-checksum to opkg so it does not error on the absent > > checksum fields. > > > > On a test image with 7000+ packages (including a 2.3 GB debug > > package): write_index time reduced from ~180s to ~22s (8x speedup) > > when opkg-make-index is configured to produce no checksums by default. > > > > See > https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796 > > ("Enable sha256 checksums in opkg indexer", 2019) for the original > rationale. > > > > AI-Generated: Claude Sonnet 4.6 > > Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> > > Hi, > > The package feed may be made available over a network and/or externally > signed after it has been generated, so I don't think we should disable > writing checksums into the feed by default. > > How about adding a variable to control this? It can be enabled by > default in OE-core and you can disable it in a local or distro config if > you know the signatures won't be needed in your use case. > > Best regards, > > -- > Paul Barker > >
diff --git a/meta/lib/oe/package_manager/ipk/__init__.py b/meta/lib/oe/package_manager/ipk/__init__.py index 344e0852177..93a12fad6a3 100644 --- a/meta/lib/oe/package_manager/ipk/__init__.py +++ b/meta/lib/oe/package_manager/ipk/__init__.py @@ -12,7 +12,7 @@ from oe.package_manager import * from oe.package_manager.common_deb_ipk import OpkgDpkgPM class OpkgIndexer(Indexer): - def write_index(self): + def write_index(self, build_checksums=True): arch_vars = ["ALL_MULTILIB_PACKAGE_ARCHS", "SDK_PACKAGE_ARCHS", ] @@ -44,8 +44,9 @@ class OpkgIndexer(Indexer): if not os.path.exists(pkgs_file): open(pkgs_file, "w").close() - index_cmds.add('%s --checksum md5 --checksum sha256 -r %s -p %s -m %s %s' % - (opkg_index_cmd, pkgs_file, pkgs_file, pkgs_dir, opkg_index_cmd_extra_params)) + checksum_args = '--checksum md5 --checksum sha256 ' if build_checksums else '' + index_cmds.add('%s %s-r %s -p %s -m %s %s' % + (opkg_index_cmd, checksum_args, pkgs_file, pkgs_file, pkgs_dir, opkg_index_cmd_extra_params)) index_sign_files.add(pkgs_file) @@ -103,8 +104,12 @@ class OpkgPM(OpkgDpkgPM): self.deploy_dir = oe.path.join(self.d.getVar('WORKDIR'), ipk_repo_workdir) self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock") self.opkg_cmd = bb.utils.which(os.getenv('PATH'), "opkg") + self.from_feeds = (self.d.getVar('BUILD_IMAGES_FROM_FEEDS') or "") == "1" + self._build_checksums = self.from_feeds or self.d.getVar('PACKAGE_FEED_SIGN') == '1' self.opkg_args = ['--volatile-cache', '-f', config_file, '-t', self.d.expand('${T}/ipktemp/'), '-o', target_rootfs] self.opkg_args.extend(shlex.split(self.d.getVar("OPKG_ARGS"))) + if not self._build_checksums: + self.opkg_args.append('--force-checksum') if prepare_index: create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_IPK"), "package_write_ipk", filterbydependencies) @@ -116,7 +121,6 @@ class OpkgPM(OpkgDpkgPM): if not os.path.exists(self.d.expand('${T}/saved')): bb.utils.mkdirhier(self.d.expand('${T}/saved')) - self.from_feeds = (self.d.getVar('BUILD_IMAGES_FROM_FEEDS') or "") == "1" if self.from_feeds: self._create_custom_config() else: @@ -346,7 +350,7 @@ class OpkgPM(OpkgDpkgPM): def write_index(self): self.deploy_dir_lock() - result = self.indexer.write_index() + result = self.indexer.write_index(build_checksums=self._build_checksums) self.deploy_dir_unlock()