| Message ID | 20220726211328.2122681-1-rustyhowell@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | debian packaging: set md5sums for conffiles | expand |
Hello Rusty, On Tue, 26 Jul 2022 15:13:28 -0600 "Rusty Howell" <rustyhowell@gmail.com> wrote: > Currently in an image created with debian package management, dpkg is missing the md5sums in > /var/lib/dpkg/status for files designated as conffiles. They are just listed as "newconffile", > which is an intermediate state when dpkg installs a package with an conf file. Also, when the > device first boots the script /usr/sbin/run-postinsts does not cause /var/lib/dpkg/status to update > those hashes either. > > The problem here is that when using a package feed, all the conf files are listed as "newconffile", > which will indicate to dpkg to replace them all on the first update. dpkg will not follow the rules for > confold, confnew, confdef. This patch does not apply on the current master branch of oe-core. Maybe you wrote it on an older commit or on a branch (e.g. kirkstone)? Also we require you so sign-off the patches you send on the mailing list ('git commit -s' does that automatically). Best regards.
Also, I'm not sure I understand why this needs to be fixed with custom code. Isn't this something that dpkg/apt take care of themselves? Alex On Wed, 27 Jul 2022 at 12:53, Luca Ceresoli via lists.openembedded.org <luca.ceresoli=bootlin.com@lists.openembedded.org> wrote: > > Hello Rusty, > > On Tue, 26 Jul 2022 15:13:28 -0600 > "Rusty Howell" <rustyhowell@gmail.com> wrote: > > > Currently in an image created with debian package management, dpkg is missing the md5sums in > > /var/lib/dpkg/status for files designated as conffiles. They are just listed as "newconffile", > > which is an intermediate state when dpkg installs a package with an conf file. Also, when the > > device first boots the script /usr/sbin/run-postinsts does not cause /var/lib/dpkg/status to update > > those hashes either. > > > > The problem here is that when using a package feed, all the conf files are listed as "newconffile", > > which will indicate to dpkg to replace them all on the first update. dpkg will not follow the rules for > > confold, confnew, confdef. > > This patch does not apply on the current master branch of oe-core. > Maybe you wrote it on an older commit or on a branch (e.g. kirkstone)? > > Also we require you so sign-off the patches you send on the mailing > list ('git commit -s' does that automatically). > > Best regards. > -- > Luca Ceresoli, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#168552): https://lists.openembedded.org/g/openembedded-core/message/168552 > Mute This Topic: https://lists.openembedded.org/mt/92637328/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
> On 27 Jul 2022, at 12:48, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > > Also, I'm not sure I understand why this needs to be fixed with custom > code. Isn't this something that dpkg/apt take care of themselves? Yes, I suspect this is a problem with how we’re generating the DEBIAN/. Ross
So we’re definitely not generating the DEBIAN/md5sums files that dpkg-deb likes to have. However, this file in proper Debian packages doesn’t contain hashes for the conffiles, so I can only imagine that dpkg generates them on install, which leads to the question of where that is meant to happen. Ross
> So we’re definitely not generating the DEBIAN/md5sums files that dpkg-deb likes to have. However, this file in proper Debian packages doesn’t contain hashes for the conffiles, so I can only imagine that dpkg generates them on install, which leads to the question of where that is meant to happen.
dpkg —configure does this, but we’re neutering it by renaming all foo.dpkg-new files to foo in the deb install() function.
If we’re going to rename foo.dpkg-new to foo, then that is where we go and update the status file.
Alternatively, we don’t do that, and every package needs to run its configure step on first boot.
Ross
Thanks for the quick responses. It took us a while to figure out why we were seeing weird behaviour with our products, and we ultimately figured out the conffiles are being replaced during the update. I am working off of Hardknott currently, so that's my mistake not switching to master. And I guess I misunderstood who is allowed to "sign off" on patches. I assumed I couldn't since I'm not a regular. But in any case, it looks like Ross Burton may be on to a better solution than what I have to fix the issue. Rusty On Wed, Jul 27, 2022 at 8:09 AM Ross Burton <ross.burton@arm.com> wrote: > > So we’re definitely not generating the DEBIAN/md5sums files that > dpkg-deb likes to have. However, this file in proper Debian packages > doesn’t contain hashes for the conffiles, so I can only imagine that dpkg > generates them on install, which leads to the question of where that is > meant to happen. > > dpkg —configure does this, but we’re neutering it by renaming all > foo.dpkg-new files to foo in the deb install() function. > > If we’re going to rename foo.dpkg-new to foo, then that is where we go and > update the status file. > > Alternatively, we don’t do that, and every package needs to run its > configure step on first boot. > > Ross > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#168563): > https://lists.openembedded.org/g/openembedded-core/message/168563 > Mute This Topic: https://lists.openembedded.org/mt/92637328/4800265 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > rustyhowell@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
I think Ross is merely helping you to reach the correct solution, but everyone would appreciate if you write and submit the actual fix :-) Alex On Wed, 27 Jul 2022 at 16:32, Rusty Howell <rustyhowell@gmail.com> wrote: > > Thanks for the quick responses. It took us a while to figure out why we were seeing weird behaviour with our products, and we ultimately figured out the conffiles are being replaced during the update. I am working off of Hardknott currently, so that's my mistake not switching to master. And I guess I misunderstood who is allowed to "sign off" on patches. I assumed I couldn't since I'm not a regular. But in any case, it looks like Ross Burton may be on to a better solution than what I have to fix the issue. > > Rusty > > On Wed, Jul 27, 2022 at 8:09 AM Ross Burton <ross.burton@arm.com> wrote: >> >> > So we’re definitely not generating the DEBIAN/md5sums files that dpkg-deb likes to have. However, this file in proper Debian packages doesn’t contain hashes for the conffiles, so I can only imagine that dpkg generates them on install, which leads to the question of where that is meant to happen. >> >> dpkg —configure does this, but we’re neutering it by renaming all foo.dpkg-new files to foo in the deb install() function. >> >> If we’re going to rename foo.dpkg-new to foo, then that is where we go and update the status file. >> >> Alternatively, we don’t do that, and every package needs to run its configure step on first boot. >> >> Ross >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#168563): https://lists.openembedded.org/g/openembedded-core/message/168563 >> Mute This Topic: https://lists.openembedded.org/mt/92637328/4800265 >> Group Owner: openembedded-core+owner@lists.openembedded.org >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [rustyhowell@gmail.com] >> -=-=-=-=-=-=-=-=-=-=-=- >>
Will do. On Wed, Jul 27, 2022 at 8:35 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > I think Ross is merely helping you to reach the correct solution, but > everyone would appreciate if you write and submit the actual fix :-) > > Alex > > On Wed, 27 Jul 2022 at 16:32, Rusty Howell <rustyhowell@gmail.com> wrote: > > > > Thanks for the quick responses. It took us a while to figure out why > we were seeing weird behaviour with our products, and we ultimately figured > out the conffiles are being replaced during the update. I am working off > of Hardknott currently, so that's my mistake not switching to master. And > I guess I misunderstood who is allowed to "sign off" on patches. I assumed > I couldn't since I'm not a regular. But in any case, it looks like Ross > Burton may be on to a better solution than what I have to fix the issue. > > > > Rusty > > > > On Wed, Jul 27, 2022 at 8:09 AM Ross Burton <ross.burton@arm.com> wrote: > >> > >> > So we’re definitely not generating the DEBIAN/md5sums files that > dpkg-deb likes to have. However, this file in proper Debian packages > doesn’t contain hashes for the conffiles, so I can only imagine that dpkg > generates them on install, which leads to the question of where that is > meant to happen. > >> > >> dpkg —configure does this, but we’re neutering it by renaming all > foo.dpkg-new files to foo in the deb install() function. > >> > >> If we’re going to rename foo.dpkg-new to foo, then that is where we go > and update the status file. > >> > >> Alternatively, we don’t do that, and every package needs to run its > configure step on first boot. > >> > >> Ross > >> -=-=-=-=-=-=-=-=-=-=-=- > >> Links: You receive all messages sent to this group. > >> View/Reply Online (#168563): > https://lists.openembedded.org/g/openembedded-core/message/168563 > >> Mute This Topic: https://lists.openembedded.org/mt/92637328/4800265 > >> Group Owner: openembedded-core+owner@lists.openembedded.org > >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > rustyhowell@gmail.com] > >> -=-=-=-=-=-=-=-=-=-=-=- > >> >
diff --git a/meta/lib/oe/package_manager/deb/__init__.py b/meta/lib/oe/package_manager/deb/__init__.py index 2ee68fefb1..fabb955558 100644 --- a/meta/lib/oe/package_manager/deb/__init__.py +++ b/meta/lib/oe/package_manager/deb/__init__.py @@ -3,6 +3,7 @@ # import re +import glob import subprocess from oe.package_manager import * @@ -216,6 +217,47 @@ class DpkgPM(OpkgDpkgPM): os.rename(status_file + ".tmp", status_file) + def set_conffile_hashes(self): + """ + This function will populate the md5sums for all the conffiles listed in /var/lib/dpkg/status + """ + status_file = self.target_rootfs + "/var/lib/dpkg/status" + + with open(status_file, "r") as sf: + with open(status_file + ".tmp", "w+") as tmp_sf: + status = sf.read() + + conffiles = glob.glob(self.target_rootfs + "/var/lib/dpkg/info/*.conffiles") + for cf in conffiles: + fname = os.path.basename(cf) + pkg = fname.split('.')[0] + + bb.note("Populating md5sums for pkg %s" % pkg) + conffiles = "%s/var/lib/dpkg/info/%s.conffiles" % (self.target_rootfs, pkg) + md5sums = "%s/var/lib/dpkg/info/%s.md5sums" % (self.target_rootfs, pkg) + if os.path.exists(conffiles) and os.path.exists(md5sums): + conf_hashes = {} + with open(md5sums, "r") as md5s: + for line in md5s.read().split('\n'): + m = re.match(r"^([a-f0-9]+)\s+(\S+)", line) + if m: + hash_ = m.group(1) + fpath = m.group(2) + if not fpath.startswith('/'): + fpath = '/' + fpath + conf_hashes[fpath] = hash_ + + with open(conffiles, "r") as cf: + for fpath in cf.read().split('\n'): + hash_ = conf_hashes.get(fpath, None) + if hash_: + status = re.sub("%s newconffile" % fpath, "%s %s" % (fpath, hash_), status) + + + tmp_sf.write(status) + + os.rename(status_file + ".tmp", status_file) + def run_pre_post_installs(self, package_name=None): """ Run the pre/post installs for package "package_name". If package_name is diff --git a/meta/lib/oe/package_manager/deb/rootfs.py b/meta/lib/oe/package_manager/deb/rootfs.py index 8fbaca11d6..06e99158de 100644 --- a/meta/lib/oe/package_manager/deb/rootfs.py +++ b/meta/lib/oe/package_manager/deb/rootfs.py @@ -186,6 +186,8 @@ class PkgRootfs(DpkgOpkgRootfs): execute_pre_post_process(self.d, deb_post_process_cmds) + self.pm.set_conffile_hashes() + if self.progress_reporter: self.progress_reporter.next_stage()