| Message ID | fd53e270-fe25-c1ad-522a-8444ca8ba642@windriver.com |
|---|---|
| State | New |
| Headers | show |
| Series | About the sstate cache directory hierarchy | expand |
I do like the idea; not everyone has a pcie gen4/5 ssd for builds, or rigorously trims sstate on a schedule. But there may be consequences or regressions, maybe RP will immediately shoot it down :) I would however still place a single level of hash[:2] *under* the pn/task/, to avoid too many files piling up in a single directory. Can you send this as an RFC patch? Alex On Fri, 28 Jan 2022 at 08:06, Chen Qi <Qi.Chen@windriver.com> wrote: > Hi All, > > I'm sending out this email because I'm wondering if we can change the > sstate cache directory to use ${PN} and taskname as subditories. Hope to > hear your opinions. > > Below is the long story. > > Recently I noticed that running `bitbake xxx -c cleansstate' usually > takes more than 10 minutes. > > After some investigation, I can see that most of the time is spent on > file searching. This is because we have: > > SSTATE_PATHSPEC = > > "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*" > > And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as > sub-directories. > > This essentially means that all sub-directories are searched. This would > take a long time, especially when run for the first time. I made some > changes to output the time and the logs are as below. > > $ bitbake glibc -c cleansstate > WARNING: glibc-2.34-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds > > I figured that unlike git, we do have knowledge on our sstate objects. > It does not seem necessary to use hash value as sub directory. So I > changed the sstate directory hierarchy to use ${PN}/taskname/ as sub > directories, and here's the result. > > $ bitbake libgcc -c cleansstate > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.020630598068237305 seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0011608600616455078 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007557868957519531 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0013995170593261719 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007488727569580078 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0005896091461181641 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125 seconds > > It's much faster. > > In addition, the sub dirs now give more info, which should potentially > make sstate cache easier to manage. > > Attached is the patch to quickly try things out. Hope to hear your > opinions. > > Best Regards, > > Chen Qi > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#161067): > https://lists.openembedded.org/g/openembedded-core/message/161067 > Mute This Topic: https://lists.openembedded.org/mt/88740516/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
On 1/28/22 6:29 PM, Alexander Kanavin wrote: > I do like the idea; not everyone has a pcie gen4/5 ssd for builds, or > rigorously trims sstate on a schedule. But there may be consequences > or regressions, maybe RP will immediately shoot it down :) > > I would however still place a single level of hash[:2] *under* the > pn/task/, to avoid too many files piling up in a single directory. > > Can you send this as an RFC patch? > > Alex Yes. I will ensure that all oe selftests pass before sending out patch. Regards, Qi > > On Fri, 28 Jan 2022 at 08:06, Chen Qi <Qi.Chen@windriver.com > <mailto:Qi.Chen@windriver.com>> wrote: > > Hi All, > > I'm sending out this email because I'm wondering if we can change the > sstate cache directory to use ${PN} and taskname as subditories. > Hope to > hear your opinions. > > Below is the long story. > > Recently I noticed that running `bitbake xxx -c cleansstate' usually > takes more than 10 minutes. > > After some investigation, I can see that most of the time is spent on > file searching. This is because we have: > > SSTATE_PATHSPEC = > "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*" > > And our sstate cache directory's hierarchy uses > hash[:2]/hash[2:4]/ as > sub-directories. > > This essentially means that all sub-directories are searched. This > would > take a long time, especially when run for the first time. I made some > changes to output the time and the logs are as below. > > $ bitbake glibc -c cleansstate > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst* > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds > > I figured that unlike git, we do have knowledge on our sstate > objects. > It does not seem necessary to use hash value as sub directory. So I > changed the sstate directory hierarchy to use ${PN}/taskname/ as sub > directories, and here's the result. > > $ bitbake libgcc -c cleansstate > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took > 0.020630598068237305 seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took > 0.0011608600616455078 seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took > 0.0007557868957519531 seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took > 0.0013995170593261719 seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took > 0.0007488727569580078 seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took > 0.0005896091461181641 seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst* > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125 > seconds > > It's much faster. > > In addition, the sub dirs now give more info, which should > potentially > make sstate cache easier to manage. > > Attached is the patch to quickly try things out. Hope to hear your > opinions. > > Best Regards, > > Chen Qi > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#161067): > https://lists.openembedded.org/g/openembedded-core/message/161067 > <https://urldefense.com/v3/__https://lists.openembedded.org/g/openembedded-core/message/161067__;!!AjveYdw8EvQ!N5Dyvu4eenTGK1TY1BrreaZ7uMlcahVqvjTbg01fXC5Sptlu1wS-nKbk5PGtw6ODQA$> > Mute This Topic: > https://lists.openembedded.org/mt/88740516/1686489 > <https://urldefense.com/v3/__https://lists.openembedded.org/mt/88740516/1686489__;!!AjveYdw8EvQ!N5Dyvu4eenTGK1TY1BrreaZ7uMlcahVqvjTbg01fXC5Sptlu1wS-nKbk5PGWaXA87A$> > Group Owner: openembedded-core+owner@lists.openembedded.org > <mailto:openembedded-core%2Bowner@lists.openembedded.org> > Unsubscribe: > https://lists.openembedded.org/g/openembedded-core/unsub > <https://urldefense.com/v3/__https://lists.openembedded.org/g/openembedded-core/unsub__;!!AjveYdw8EvQ!N5Dyvu4eenTGK1TY1BrreaZ7uMlcahVqvjTbg01fXC5Sptlu1wS-nKbk5PEkTurZng$> > [alex.kanavin@gmail.com <mailto:alex.kanavin@gmail.com>] > -=-=-=-=-=-=-=-=-=-=-=- >
On 1/28/22 08:06, Chen Qi wrote: > Hi All, > > I'm sending out this email because I'm wondering if we can change the > sstate cache directory to use ${PN} and taskname as subditories. Hope to > hear your opinions. > > Below is the long story. > > Recently I noticed that running `bitbake xxx -c cleansstate' usually > takes more than 10 minutes. > > After some investigation, I can see that most of the time is spent on > file searching. This is because we have: > > SSTATE_PATHSPEC = > "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*" > > > And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as > sub-directories. > > This essentially means that all sub-directories are searched. This would > take a long time, especially when run for the first time. I made some > changes to output the time and the logs are as below. > > $ bitbake glibc -c cleansstate > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds > > I figured that unlike git, we do have knowledge on our sstate objects. > It does not seem necessary to use hash value as sub directory. So I > changed the sstate directory hierarchy to use ${PN}/taskname/ as sub > directories, and here's the result. > > $ bitbake libgcc -c cleansstate > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.020630598068237305 seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0011608600616455078 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007557868957519531 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0013995170593261719 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007488727569580078 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0005896091461181641 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125 seconds > > It's much faster. > > In addition, the sub dirs now give more info, which should potentially > make sstate cache easier to manage. > > Attached is the patch to quickly try things out. Hope to hear your > opinions. > > Best Regards, > > Chen Qi > I thought this sounded interesting. Has anything come out of this that I missed ? Jacob
Hi Jacob, No. I haven't done anything since then, because I've been involved in something else. Regards, Qi -----Original Message----- From: Jacob Kroon <jacob.kroon@gmail.com> Sent: Monday, May 16, 2022 1:30 PM To: Chen, Qi <Qi.Chen@windriver.com>; openembedded-core@lists.openembedded.org; Richard Purdie <richard.purdie@linuxfoundation.org> Subject: Re: [OE-core] About the sstate cache directory hierarchy On 1/28/22 08:06, Chen Qi wrote: > Hi All, > > I'm sending out this email because I'm wondering if we can change the > sstate cache directory to use ${PN} and taskname as subditories. Hope > to hear your opinions. > > Below is the long story. > > Recently I noticed that running `bitbake xxx -c cleansstate' usually > takes more than 10 minutes. > > After some investigation, I can see that most of the time is spent on > file searching. This is because we have: > > SSTATE_PATHSPEC = > "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*" > > > And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as > sub-directories. > > This essentially means that all sub-directories are searched. This > would take a long time, especially when run for the first time. I made > some changes to output the time and the logs are as below. > > $ bitbake glibc -c cleansstate > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor > e2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst > * > > WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor > e2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor > e2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor > e2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor > e2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2. > 34:r0::7:*_populate_lic.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor > e2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds > WARNING: glibc-2.34-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor > e2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst* > > WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds > > I figured that unlike git, we do have knowledge on our sstate objects. > It does not seem necessary to use hash value as sub directory. So I > changed the sstate directory hierarchy to use ${PN}/taskname/ as sub > directories, and here's the result. > > $ bitbake libgcc -c cleansstate > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source > _date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_d > eploy_source_date_epoch.tar.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.020630598068237305 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstat > e:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0011608600616455078 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/ss > tate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar. > zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007557868957519531 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write > _rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_ > write_rpm.tar.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0013995170593261719 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/s > state:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.ta > r.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007488727569580078 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/ > sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0005896091461181641 > seconds > WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing > /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysr > oot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_ > sysroot.tar.zst* > > WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125 > seconds > > It's much faster. > > In addition, the sub dirs now give more info, which should potentially > make sstate cache easier to manage. > > Attached is the patch to quickly try things out. Hope to hear your > opinions. > > Best Regards, > > Chen Qi > I thought this sounded interesting. Has anything come out of this that I missed ? Jacob
From 6f148d399c2ac575a9079eae682a56802dfac3ca Mon Sep 17 00:00:00 2001 From: Chen Qi <Qi.Chen@windriver.com> Date: Thu, 27 Jan 2022 22:57:57 -0800 Subject: [PATCH] sstate.bbclasss: use /taskname as sub dirs Signed-off-by: Chen Qi <Qi.Chen@windriver.com> --- meta/classes/sstate.bbclass | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index 17dcf4cc17..c657d40151 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -17,6 +17,7 @@ def generate_sstatefn(spec, hash, taskname, siginfo, d): if not hash: hash = "INVALID" fn = spec + hash + "_" + taskname + extension + pn = spec.split(':')[1] # If the filename is too long, attempt to reduce it if len(fn) > limit: components = spec.split(":") @@ -30,7 +31,7 @@ def generate_sstatefn(spec, hash, taskname, siginfo, d): fn = spec + hash + "_" + taskname + extension if len(fn) > limit: bb.fatal("Unable to reduce sstate name to less than 255 chararacters") - return hash[:2] + "/" + hash[2:4] + "/" + fn + return pn + "/" + taskname + "/" + fn SSTATE_PKGARCH = "${PACKAGE_ARCH}" SSTATE_PKGSPEC = "sstate:${PN}:${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}:${PV}:${PR}:${SSTATE_PKGARCH}:${SSTATE_VERSION}:" @@ -39,7 +40,7 @@ SSTATE_PKGNAME = "${SSTATE_EXTRAPATH}${@generate_sstatefn(d.getVar('SSTATE_PK SSTATE_PKG = "${SSTATE_DIR}/${SSTATE_PKGNAME}" SSTATE_EXTRAPATH = "" SSTATE_EXTRAPATHWILDCARD = "" -SSTATE_PATHSPEC = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}*/*/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*" +SSTATE_PATHSPEC = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*" # explicitly make PV to depend on evaluated value of PV variable PV[vardepvalue] = "${PV}" @@ -482,12 +483,16 @@ python sstate_hardcode_path_unpack () { def sstate_clean_cachefile(ss, d): import oe.path + import time if d.getVarFlag('do_%s' % ss['task'], 'task'): d.setVar("SSTATE_PATH_CURRTASK", ss['task']) sstatepkgfile = d.getVar('SSTATE_PATHSPEC') - bb.note("Removing %s" % sstatepkgfile) + bb.warn("Removing %s" % sstatepkgfile) + start = time.time() oe.path.remove(sstatepkgfile) + end = time.time() + bb.warn("Took %s seconds" % (end-start)) def sstate_clean_cachefiles(d): for task in (d.getVar('SSTATETASKS') or "").split(): -- 2.33.0
Hi All, I'm sending out this email because I'm wondering if we can change the sstate cache directory to use ${PN} and taskname as subditories. Hope to hear your opinions. Below is the long story. Recently I noticed that running `bitbake xxx -c cleansstate' usually takes more than 10 minutes. After some investigation, I can see that most of the time is spent on file searching. This is because we have: SSTATE_PATHSPEC = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*" And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as sub-directories. This essentially means that all sub-directories are searched. This would take a long time, especially when run for the first time. I made some changes to output the time and the logs are as below. $ bitbake glibc -c cleansstate WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds WARNING: glibc-2.34-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst* WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds I figured that unlike git, we do have knowledge on our sstate objects. It does not seem necessary to use hash value as sub directory. So I changed the sstate directory hierarchy to use ${PN}/taskname/ as sub directories, and here's the result. $ bitbake libgcc -c cleansstate WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst* WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.020630598068237305 seconds WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst* WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0011608600616455078 seconds WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst* WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007557868957519531 seconds WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst* WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0013995170593261719 seconds WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst* WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007488727569580078 seconds WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst* WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0005896091461181641 seconds WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst* WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125 seconds It's much faster. In addition, the sub dirs now give more info, which should potentially make sstate cache easier to manage. Attached is the patch to quickly try things out. Hope to hear your opinions. Best Regards, Chen Qi