| Message ID | 20220728085407.293067-1-jose.quaresma@foundries.io |
|---|---|
| State | New |
| Headers | show |
| Series | create-spdx: Runs the do_create_spdx task after the do_package_write_* | expand |
On Thu, 2022-07-28 at 09:54 +0100, Jose Quaresma wrote: > Otherwise spdx can have references for data that is not packed. > in the package delivered. > > During do_package_write_ipk task in do_package_ipk some files > is cleaned up from packages-split directory in the funcion > cleanupcontrol. > > This also fixes the following race condictions when the do_create_spdx > task runs the add_package_files function and these files is been deleted > at same time in the task do_package_write_ipk: > > ERROR: alsa-topology-conf-1.2.5.1-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated: > > The stack trace of python calls that resulted in this exception/failure was: > File: 'exec_func_python() autogenerated', lineno: 2, function: <module> > 0001: > *** 0002:do_create_spdx(d) > 0003: > File: '/srv/oe/build/conf/../../layers/openembedded-core/meta/classes/create-spdx.bbclass', lineno: 567, function: do_create_spdx > 0563: package_doc.add_relationship(package_doc, "DESCRIBES", spdx_package) > 0564: > 0565: package_archive = deploy_dir_spdx / "packages" / (package_doc.name + ".tar.zst") > 0566: with optional_tarfile(package_archive, archive_packaged) as archive: > *** 0567: package_files = add_package_files( > 0568: d, > 0569: package_doc, > 0570: spdx_package, > 0571: pkgdest / package, > File: '/srv/oe/build/conf/../../layers/openembedded-core/meta/classes/create-spdx.bbclass', lineno: 234, function: add_package_files > 0230: info.mtime = source_date_epoch > 0231: > 0232: archive.addfile(info, f) > 0233: > *** 0234: sha1 = bb.utils.sha1_file(filepath) > 0235: sha1s.append(sha1) > 0236: spdx_file.checksums.append(oe.spdx.SPDXChecksum( > 0237: algorithm="SHA1", > 0238: checksumValue=sha1, > File: '/srv/oe/bitbake/lib/bb/utils.py', lineno: 559, function: sha1_file > 0555: """ > 0556: Return the hex string representation of the SHA1 checksum of the filename > 0557: """ > 0558: import hashlib > *** 0559: return _hasher(hashlib.sha1(), filename) > 0560: > 0561:def sha384_file(filename): > 0562: """ > 0563: Return the hex string representation of the SHA384 checksum of the filename > File: '/srv/oe/bitbake/lib/bb/utils.py', lineno: 528, function: _hasher > 0524: > 0525:def _hasher(method, filename): > 0526: import mmap > 0527: > *** 0528: with open(filename, "rb") as f: > 0529: try: > 0530: with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm: > 0531: for chunk in iter(lambda: mm.read(8192), b''): > 0532: method.update(chunk) > Exception: FileNotFoundError: [Errno 2] No such file or directory: '/srv/oe/build/tmp-lmp/work/all-lmp-linux/alsa-topology-conf/1.2.5.1-r0/packages-split/alsa-topology-conf/CONTROL/control' We have other places where we've had to teach the code to ignore the control files and we'll probably have to do that here (grep for CONTROL). Ideally we'd fix opkg (and maybe dpkg) to allow creation of packages with the files placed externally. I have also wondered if we should create a specific hardlinked tree to handle this but that does come at an IO cost. I don't really want to add a dependency constraint like this for something we already handle differently elsewhere though. Cheers, Richard
Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia quinta, 28/07/2022 à(s) 12:01: > On Thu, 2022-07-28 at 09:54 +0100, Jose Quaresma wrote: > > Otherwise spdx can have references for data that is not packed. > > in the package delivered. > > > > During do_package_write_ipk task in do_package_ipk some files > > is cleaned up from packages-split directory in the funcion > > cleanupcontrol. > > > > This also fixes the following race condictions when the do_create_spdx > > task runs the add_package_files function and these files is been deleted > > at same time in the task do_package_write_ipk: > > > > ERROR: alsa-topology-conf-1.2.5.1-r0 do_create_spdx: Error executing a > python function in exec_func_python() autogenerated: > > > > The stack trace of python calls that resulted in this exception/failure > was: > > File: 'exec_func_python() autogenerated', lineno: 2, function: <module> > > 0001: > > *** 0002:do_create_spdx(d) > > 0003: > > File: > '/srv/oe/build/conf/../../layers/openembedded-core/meta/classes/create-spdx.bbclass', > lineno: 567, function: do_create_spdx > > 0563: package_doc.add_relationship(package_doc, > "DESCRIBES", spdx_package) > > 0564: > > 0565: package_archive = deploy_dir_spdx / "packages" / ( > package_doc.name + ".tar.zst") > > 0566: with optional_tarfile(package_archive, > archive_packaged) as archive: > > *** 0567: package_files = add_package_files( > > 0568: d, > > 0569: package_doc, > > 0570: spdx_package, > > 0571: pkgdest / package, > > File: > '/srv/oe/build/conf/../../layers/openembedded-core/meta/classes/create-spdx.bbclass', > lineno: 234, function: add_package_files > > 0230: info.mtime = source_date_epoch > > 0231: > > 0232: archive.addfile(info, f) > > 0233: > > *** 0234: sha1 = bb.utils.sha1_file(filepath) > > 0235: sha1s.append(sha1) > > 0236: > spdx_file.checksums.append(oe.spdx.SPDXChecksum( > > 0237: algorithm="SHA1", > > 0238: checksumValue=sha1, > > File: '/srv/oe/bitbake/lib/bb/utils.py', lineno: 559, function: sha1_file > > 0555: """ > > 0556: Return the hex string representation of the SHA1 checksum > of the filename > > 0557: """ > > 0558: import hashlib > > *** 0559: return _hasher(hashlib.sha1(), filename) > > 0560: > > 0561:def sha384_file(filename): > > 0562: """ > > 0563: Return the hex string representation of the SHA384 > checksum of the filename > > File: '/srv/oe/bitbake/lib/bb/utils.py', lineno: 528, function: _hasher > > 0524: > > 0525:def _hasher(method, filename): > > 0526: import mmap > > 0527: > > *** 0528: with open(filename, "rb") as f: > > 0529: try: > > 0530: with mmap.mmap(f.fileno(), 0, > access=mmap.ACCESS_READ) as mm: > > 0531: for chunk in iter(lambda: mm.read(8192), b''): > > 0532: method.update(chunk) > > Exception: FileNotFoundError: [Errno 2] No such file or directory: > '/srv/oe/build/tmp-lmp/work/all-lmp-linux/alsa-topology-conf/1.2.5.1-r0/packages-split/alsa-topology-conf/CONTROL/control' > > We have other places where we've had to teach the code to ignore the > control files and we'll probably have to do that here (grep for > CONTROL). > The snip that delete this files is in cleanupcontrol https://git.yoctoproject.org/poky/tree/meta/classes/package_ipk.bbclass#n51 So these files are not shipped in the ipk package produced. It breaks the do_create_spdx because it runs after the do_package and the same happens for do_package_write_ipk so the two tasks can run at the same time. > Ideally we'd fix opkg (and maybe dpkg) to allow creation of packages > with the files placed externally. > I think I didn't quite understand this part, can you please explain it a bit more? > > I have also wondered if we should create a specific hardlinked tree to > handle this but that does come at an IO cost. > > I don't really want to add a dependency constraint like this for > something we already handle differently elsewhere though. > yeah, It's a bit ugly because it adds some tasks that don't even run (all types of packages in this case). Since I don't know what package types are chosen by the user, I added all of them. Jose > > Cheers, > > Richard >
Jose Quaresma via lists.openembedded.org <quaresma.jose= gmail.com@lists.openembedded.org> escreveu no dia quinta, 28/07/2022 à(s) 12:46: > > > Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia > quinta, 28/07/2022 à(s) 12:01: > >> On Thu, 2022-07-28 at 09:54 +0100, Jose Quaresma wrote: >> > Otherwise spdx can have references for data that is not packed. >> > in the package delivered. >> > >> > During do_package_write_ipk task in do_package_ipk some files >> > is cleaned up from packages-split directory in the funcion >> > cleanupcontrol. >> > >> > This also fixes the following race condictions when the do_create_spdx >> > task runs the add_package_files function and these files is been deleted >> > at same time in the task do_package_write_ipk: >> > >> > ERROR: alsa-topology-conf-1.2.5.1-r0 do_create_spdx: Error executing a >> python function in exec_func_python() autogenerated: >> > >> > The stack trace of python calls that resulted in this exception/failure >> was: >> > File: 'exec_func_python() autogenerated', lineno: 2, function: <module> >> > 0001: >> > *** 0002:do_create_spdx(d) >> > 0003: >> > File: >> '/srv/oe/build/conf/../../layers/openembedded-core/meta/classes/create-spdx.bbclass', >> lineno: 567, function: do_create_spdx >> > 0563: package_doc.add_relationship(package_doc, >> "DESCRIBES", spdx_package) >> > 0564: >> > 0565: package_archive = deploy_dir_spdx / "packages" / ( >> package_doc.name + ".tar.zst") >> > 0566: with optional_tarfile(package_archive, >> archive_packaged) as archive: >> > *** 0567: package_files = add_package_files( >> > 0568: d, >> > 0569: package_doc, >> > 0570: spdx_package, >> > 0571: pkgdest / package, >> > File: >> '/srv/oe/build/conf/../../layers/openembedded-core/meta/classes/create-spdx.bbclass', >> lineno: 234, function: add_package_files >> > 0230: info.mtime = source_date_epoch >> > 0231: >> > 0232: archive.addfile(info, f) >> > 0233: >> > *** 0234: sha1 = bb.utils.sha1_file(filepath) >> > 0235: sha1s.append(sha1) >> > 0236: >> spdx_file.checksums.append(oe.spdx.SPDXChecksum( >> > 0237: algorithm="SHA1", >> > 0238: checksumValue=sha1, >> > File: '/srv/oe/bitbake/lib/bb/utils.py', lineno: 559, function: >> sha1_file >> > 0555: """ >> > 0556: Return the hex string representation of the SHA1 checksum >> of the filename >> > 0557: """ >> > 0558: import hashlib >> > *** 0559: return _hasher(hashlib.sha1(), filename) >> > 0560: >> > 0561:def sha384_file(filename): >> > 0562: """ >> > 0563: Return the hex string representation of the SHA384 >> checksum of the filename >> > File: '/srv/oe/bitbake/lib/bb/utils.py', lineno: 528, function: _hasher >> > 0524: >> > 0525:def _hasher(method, filename): >> > 0526: import mmap >> > 0527: >> > *** 0528: with open(filename, "rb") as f: >> > 0529: try: >> > 0530: with mmap.mmap(f.fileno(), 0, >> access=mmap.ACCESS_READ) as mm: >> > 0531: for chunk in iter(lambda: mm.read(8192), b''): >> > 0532: method.update(chunk) >> > Exception: FileNotFoundError: [Errno 2] No such file or directory: >> '/srv/oe/build/tmp-lmp/work/all-lmp-linux/alsa-topology-conf/1.2.5.1-r0/packages-split/alsa-topology-conf/CONTROL/control' >> >> We have other places where we've had to teach the code to ignore the >> control files and we'll probably have to do that here (grep for >> CONTROL). >> > > The snip that delete this files is in cleanupcontrol > https://git.yoctoproject.org/poky/tree/meta/classes/package_ipk.bbclass#n51 > > So these files are not shipped in the ipk package produced. > What I mean is those files are not part of the content deployed and it is a requirement of the packing standard used so tracking it the sbom doesn't make much sense imo. ar -xv deploy/ipk/all/alsa-topology-conf_1.2.5.1-r0_all.ipk x - debian-binary x - control.tar.gz x - data.tar.xz For and deb and ipk we have in deb_write_pkg and ipk_write_pkg: finally: cleanupcontrol(root) bb.utils.unlockfile(lf) For rpm we have in do_package_rpm: # rpm 4 creates various empty directories in _topdir, let's clean them up cleanupcmd = "rm -rf %s/BUILDROOT %s/SOURCES %s/SPECS %s/SRPMS" % (workdir, workdir, workdir, workdir) So all of them delete this package control content at the end but as do_create_spdx runs at the same time it can see and track this content as well. Thanks, Jose It breaks the do_create_spdx because it runs after the do_package > and the same happens for do_package_write_ipk so the two tasks > can run at the same time. > > >> Ideally we'd fix opkg (and maybe dpkg) to allow creation of packages >> with the files placed externally. >> > > I think I didn't quite understand this part, can you please explain it a > bit more? > > >> >> I have also wondered if we should create a specific hardlinked tree to >> handle this but that does come at an IO cost. >> >> I don't really want to add a dependency constraint like this for >> something we already handle differently elsewhere though. >> > > yeah, It's a bit ugly because it adds some tasks that don't even run (all > types of packages > in this case). Since I don't know what package types are chosen by the > user, I added all of them. > > Jose > > >> >> Cheers, >> >> Richard >> > > > -- > Best regards, > > José Quaresma > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#168603): > https://lists.openembedded.org/g/openembedded-core/message/168603 > Mute This Topic: https://lists.openembedded.org/mt/92666627/5052612 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > quaresma.jose@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On Thu, 2022-07-28 at 15:32 +0100, Jose Quaresma wrote: > > > Jose Quaresma via lists.openembedded.org > <quaresma.jose=gmail.com@lists.openembedded.org> escreveu no dia > quinta, 28/07/2022 à(s) 12:46: > > > > > > Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia > > quinta, 28/07/2022 à(s) 12:01: > > > On Thu, 2022-07-28 at 09:54 +0100, Jose Quaresma wrote: > > > > Otherwise spdx can have references for data that is not packed. > > > > in the package delivered. > > > > > > > > During do_package_write_ipk task in do_package_ipk some files > > > > is cleaned up from packages-split directory in the funcion > > > > cleanupcontrol. > > > > > > > > This also fixes the following race condictions when the > > > > do_create_spdx > > > > task runs the add_package_files function and these files is > > > > been deleted > > > > at same time in the task do_package_write_ipk: > > > > > > > > ERROR: alsa-topology-conf-1.2.5.1-r0 do_create_spdx: Error > > > > executing a python function in exec_func_python() > > > > autogenerated: > > > > > > > > The stack trace of python calls that resulted in this > > > > exception/failure was: > > > > File: 'exec_func_python() autogenerated', lineno: 2, function: > > > > <module> > > > > 0001: > > > > *** 0002:do_create_spdx(d) > > > > 0003: > > > > File: '/srv/oe/build/conf/../../layers/openembedded- > > > > core/meta/classes/create-spdx.bbclass', lineno: 567, function: > > > > do_create_spdx > > > > 0563: > > > > package_doc.add_relationship(package_doc, "DESCRIBES", > > > > spdx_package) > > > > 0564: > > > > 0565: package_archive = deploy_dir_spdx / > > > > "packages" / (package_doc.name + ".tar.zst") > > > > 0566: with optional_tarfile(package_archive, > > > > archive_packaged) as archive: > > > > *** 0567: package_files = add_package_files( > > > > 0568: d, > > > > 0569: package_doc, > > > > 0570: spdx_package, > > > > 0571: pkgdest / package, > > > > File: '/srv/oe/build/conf/../../layers/openembedded- > > > > core/meta/classes/create-spdx.bbclass', lineno: 234, function: > > > > add_package_files > > > > 0230: info.mtime = > > > > source_date_epoch > > > > 0231: > > > > 0232: archive.addfile(info, f) > > > > 0233: > > > > *** 0234: sha1 = bb.utils.sha1_file(filepath) > > > > 0235: sha1s.append(sha1) > > > > 0236: > > > > spdx_file.checksums.append(oe.spdx.SPDXChecksum( > > > > 0237: algorithm="SHA1", > > > > 0238: checksumValue=sha1, > > > > File: '/srv/oe/bitbake/lib/bb/utils.py', lineno: 559, function: > > > > sha1_file > > > > 0555: """ > > > > 0556: Return the hex string representation of the SHA1 > > > > checksum of the filename > > > > 0557: """ > > > > 0558: import hashlib > > > > *** 0559: return _hasher(hashlib.sha1(), filename) > > > > 0560: > > > > 0561:def sha384_file(filename): > > > > 0562: """ > > > > 0563: Return the hex string representation of the > > > > SHA384 checksum of the filename > > > > File: '/srv/oe/bitbake/lib/bb/utils.py', lineno: 528, function: > > > > _hasher > > > > 0524: > > > > 0525:def _hasher(method, filename): > > > > 0526: import mmap > > > > 0527: > > > > *** 0528: with open(filename, "rb") as f: > > > > 0529: try: > > > > 0530: with mmap.mmap(f.fileno(), 0, > > > > access=mmap.ACCESS_READ) as mm: > > > > 0531: for chunk in iter(lambda: > > > > mm.read(8192), b''): > > > > 0532: method.update(chunk) > > > > Exception: FileNotFoundError: [Errno 2] No such file or > > > > directory: '/srv/oe/build/tmp-lmp/work/all-lmp-linux/alsa- > > > > topology-conf/1.2.5.1-r0/packages-split/alsa-topology- > > > > conf/CONTROL/control' > > > > > > We have other places where we've had to teach the code to ignore > > > the > > > control files and we'll probably have to do that here (grep for > > > CONTROL). > > > > > > > > > The snip that delete this files is in cleanupcontrol > > https://git.yoctoproject.org/poky/tree/meta/classes/package_ipk.bbclass#n51 > > > > So these files are not shipped in the ipk package produced. > > > > > What I mean is those files are not part of the content deployed and > it is a requirement of the packing > standard used so tracking it the sbom doesn't make much sense imo. Right, it would be fine to make the spdx class skip them if they were present. > ar -xv deploy/ipk/all/alsa-topology-conf_1.2.5.1-r0_all.ipk > x - debian-binary > x - control.tar.gz > x - data.tar.xz > > For and deb and ipk we have in deb_write_pkg and ipk_write_pkg: > > finally: > cleanupcontrol(root) > bb.utils.unlockfile(lf) > > For rpm we have in do_package_rpm: > > # rpm 4 creates various empty directories in _topdir, let's > clean them up > cleanupcmd = "rm -rf %s/BUILDROOT %s/SOURCES %s/SPECS > %s/SRPMS" % (workdir, workdir, workdir, workdir) > > So all of them delete this package control content at the end but as > do_create_spdx > runs at the same time it can see and track this content as well. Yes, it is a race. We've generally tweaked the code using these directories to ignore these files rather than change the dependencies though. Ideally we'd change dpkg/opkg to place the files somewhere else during package build but we've not worked out how to do that as yet. Cheers, Richard
Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia quinta, 28/07/2022 à(s) 22:15: > On Thu, 2022-07-28 at 15:32 +0100, Jose Quaresma wrote: > > > > > > Jose Quaresma via lists.openembedded.org > > <quaresma.jose=gmail.com@lists.openembedded.org> escreveu no dia > > quinta, 28/07/2022 à(s) 12:46: > > > > > > > > > Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia > > > quinta, 28/07/2022 à(s) 12:01: > > > > On Thu, 2022-07-28 at 09:54 +0100, Jose Quaresma wrote: > > > > > Otherwise spdx can have references for data that is not packed. > > > > > in the package delivered. > > > > > > > > > > During do_package_write_ipk task in do_package_ipk some files > > > > > is cleaned up from packages-split directory in the funcion > > > > > cleanupcontrol. > > > > > > > > > > This also fixes the following race condictions when the > > > > > do_create_spdx > > > > > task runs the add_package_files function and these files is > > > > > been deleted > > > > > at same time in the task do_package_write_ipk: > > > > > > > > > > ERROR: alsa-topology-conf-1.2.5.1-r0 do_create_spdx: Error > > > > > executing a python function in exec_func_python() > > > > > autogenerated: > > > > > > > > > > The stack trace of python calls that resulted in this > > > > > exception/failure was: > > > > > File: 'exec_func_python() autogenerated', lineno: 2, function: > > > > > <module> > > > > > 0001: > > > > > *** 0002:do_create_spdx(d) > > > > > 0003: > > > > > File: '/srv/oe/build/conf/../../layers/openembedded- > > > > > core/meta/classes/create-spdx.bbclass', lineno: 567, function: > > > > > do_create_spdx > > > > > 0563: > > > > > package_doc.add_relationship(package_doc, "DESCRIBES", > > > > > spdx_package) > > > > > 0564: > > > > > 0565: package_archive = deploy_dir_spdx / > > > > > "packages" / (package_doc.name + ".tar.zst") > > > > > 0566: with optional_tarfile(package_archive, > > > > > archive_packaged) as archive: > > > > > *** 0567: package_files = add_package_files( > > > > > 0568: d, > > > > > 0569: package_doc, > > > > > 0570: spdx_package, > > > > > 0571: pkgdest / package, > > > > > File: '/srv/oe/build/conf/../../layers/openembedded- > > > > > core/meta/classes/create-spdx.bbclass', lineno: 234, function: > > > > > add_package_files > > > > > 0230: info.mtime = > > > > > source_date_epoch > > > > > 0231: > > > > > 0232: archive.addfile(info, f) > > > > > 0233: > > > > > *** 0234: sha1 = bb.utils.sha1_file(filepath) > > > > > 0235: sha1s.append(sha1) > > > > > 0236: > > > > > spdx_file.checksums.append(oe.spdx.SPDXChecksum( > > > > > 0237: algorithm="SHA1", > > > > > 0238: checksumValue=sha1, > > > > > File: '/srv/oe/bitbake/lib/bb/utils.py', lineno: 559, function: > > > > > sha1_file > > > > > 0555: """ > > > > > 0556: Return the hex string representation of the SHA1 > > > > > checksum of the filename > > > > > 0557: """ > > > > > 0558: import hashlib > > > > > *** 0559: return _hasher(hashlib.sha1(), filename) > > > > > 0560: > > > > > 0561:def sha384_file(filename): > > > > > 0562: """ > > > > > 0563: Return the hex string representation of the > > > > > SHA384 checksum of the filename > > > > > File: '/srv/oe/bitbake/lib/bb/utils.py', lineno: 528, function: > > > > > _hasher > > > > > 0524: > > > > > 0525:def _hasher(method, filename): > > > > > 0526: import mmap > > > > > 0527: > > > > > *** 0528: with open(filename, "rb") as f: > > > > > 0529: try: > > > > > 0530: with mmap.mmap(f.fileno(), 0, > > > > > access=mmap.ACCESS_READ) as mm: > > > > > 0531: for chunk in iter(lambda: > > > > > mm.read(8192), b''): > > > > > 0532: method.update(chunk) > > > > > Exception: FileNotFoundError: [Errno 2] No such file or > > > > > directory: '/srv/oe/build/tmp-lmp/work/all-lmp-linux/alsa- > > > > > topology-conf/1.2.5.1-r0/packages-split/alsa-topology- > > > > > conf/CONTROL/control' > > > > > > > > We have other places where we've had to teach the code to ignore > > > > the > > > > control files and we'll probably have to do that here (grep for > > > > CONTROL). > > > > > > > > > > > > > The snip that delete this files is in cleanupcontrol > > > > https://git.yoctoproject.org/poky/tree/meta/classes/package_ipk.bbclass#n51 > > > > > > So these files are not shipped in the ipk package produced. > > > > > > > > > What I mean is those files are not part of the content deployed and > > it is a requirement of the packing > > standard used so tracking it the sbom doesn't make much sense imo. > > Right, it would be fine to make the spdx class skip them if they were > present. > > > ar -xv deploy/ipk/all/alsa-topology-conf_1.2.5.1-r0_all.ipk > > x - debian-binary > > x - control.tar.gz > > x - data.tar.xz > > > > For and deb and ipk we have in deb_write_pkg and ipk_write_pkg: > > > > finally: > > cleanupcontrol(root) > > bb.utils.unlockfile(lf) > > > > For rpm we have in do_package_rpm: > > > > # rpm 4 creates various empty directories in _topdir, let's > > clean them up > > cleanupcmd = "rm -rf %s/BUILDROOT %s/SOURCES %s/SPECS > > %s/SRPMS" % (workdir, workdir, workdir, workdir) > > > > So all of them delete this package control content at the end but as > > do_create_spdx > > runs at the same time it can see and track this content as well. > > Yes, it is a race. We've generally tweaked the code using these > directories to ignore these files rather than change the dependencies > though. > > Ideally we'd change dpkg/opkg to place the files somewhere else during > package build but we've not worked out how to do that as yet. > Many thanks for the review and your time. I will do that in this way, change dpkg/opkg to place the files somewhere else. Jose > Cheers, > > Richard > > >
diff --git a/meta/classes/create-spdx.bbclass b/meta/classes/create-spdx.bbclass index 15cccac84b..c4930b0de7 100644 --- a/meta/classes/create-spdx.bbclass +++ b/meta/classes/create-spdx.bbclass @@ -588,7 +588,7 @@ python do_create_spdx() { oe.sbom.write_doc(d, package_doc, "packages", indent=get_json_indent(d)) } # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_package_write_rpm do_package_write_ipk do_package_write_deb do_unpack before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
Otherwise spdx can have references for data that is not packed. in the package delivered. During do_package_write_ipk task in do_package_ipk some files is cleaned up from packages-split directory in the funcion cleanupcontrol. This also fixes the following race condictions when the do_create_spdx task runs the add_package_files function and these files is been deleted at same time in the task do_package_write_ipk: ERROR: alsa-topology-conf-1.2.5.1-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:do_create_spdx(d) 0003: File: '/srv/oe/build/conf/../../layers/openembedded-core/meta/classes/create-spdx.bbclass', lineno: 567, function: do_create_spdx 0563: package_doc.add_relationship(package_doc, "DESCRIBES", spdx_package) 0564: 0565: package_archive = deploy_dir_spdx / "packages" / (package_doc.name + ".tar.zst") 0566: with optional_tarfile(package_archive, archive_packaged) as archive: *** 0567: package_files = add_package_files( 0568: d, 0569: package_doc, 0570: spdx_package, 0571: pkgdest / package, File: '/srv/oe/build/conf/../../layers/openembedded-core/meta/classes/create-spdx.bbclass', lineno: 234, function: add_package_files 0230: info.mtime = source_date_epoch 0231: 0232: archive.addfile(info, f) 0233: *** 0234: sha1 = bb.utils.sha1_file(filepath) 0235: sha1s.append(sha1) 0236: spdx_file.checksums.append(oe.spdx.SPDXChecksum( 0237: algorithm="SHA1", 0238: checksumValue=sha1, File: '/srv/oe/bitbake/lib/bb/utils.py', lineno: 559, function: sha1_file 0555: """ 0556: Return the hex string representation of the SHA1 checksum of the filename 0557: """ 0558: import hashlib *** 0559: return _hasher(hashlib.sha1(), filename) 0560: 0561:def sha384_file(filename): 0562: """ 0563: Return the hex string representation of the SHA384 checksum of the filename File: '/srv/oe/bitbake/lib/bb/utils.py', lineno: 528, function: _hasher 0524: 0525:def _hasher(method, filename): 0526: import mmap 0527: *** 0528: with open(filename, "rb") as f: 0529: try: 0530: with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm: 0531: for chunk in iter(lambda: mm.read(8192), b''): 0532: method.update(chunk) Exception: FileNotFoundError: [Errno 2] No such file or directory: '/srv/oe/build/tmp-lmp/work/all-lmp-linux/alsa-topology-conf/1.2.5.1-r0/packages-split/alsa-topology-conf/CONTROL/control' ERROR: Logfile of failure stored in: /srv/oe/build/tmp-lmp/work/all-lmp-linux/alsa-topology-conf/1.2.5.1-r0/temp/log.do_create_spdx.998864 INFO: recipe alsa-topology-conf-1.2.5.1-r0: task do_create_spdx: Failed Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> --- meta/classes/create-spdx.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)