| Message ID | 20240522092817.1250809-1-richard.purdie@linuxfoundation.org |
|---|---|
| State | Accepted, archived |
| Commit | b84eec5c4cbf4b39d6712800dd0d2fe5337721cb |
| Headers | show |
| Series | [1/3] base: Switch UNPACKDIR to a subdir of WORKDIR | expand |
> -----Original Message----- > From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Richard Purdie > Sent: den 22 maj 2024 11:28 > To: openembedded-core@lists.openembedded.org > Subject: [OE-core] [PATCH 1/3] base: Switch UNPACKDIR to a subdir of WORKDIR > > Change do_unpack to unpack files to a subdirectory of WORKDIR instead of WORKDIR > itself. There are several good reasons for this but it is mainly about being able > to isolate the output of the unpack task and tell the files apart from other things > which are created in workdir (logs, sysroots, temp dirs and more). > > This means that when the do_unpack task reruns, we can clean UNPACKDIR and know > we have a standard point to start builds from. > > It also makes code in tools like devtool and recipetool easier. > > To reduce the impact to users, if a subdirectory under UNPACKDIR matches > the first subdirectory under WORKDIR of S, that directory is moved into position > inside WORKDIR. This preserves the behaviour of S = "${WORKDIR}/git", > S = "${WORKDIR}/${BPN}" and other commonly used source directory setups. > > The directory is moved since sadly many autotools based projects can't cope with > symlinks in their paths. > > The patch also updates reproducible and SOURCE_DATE_EPOCH handling to > match the new potential source locations. We can get rid of the horrible > list of hardcoded directories in WORKDIR to ignore from that code. > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > meta/classes-global/base.bbclass | 28 +++++++++++++++++++++++----- > meta/conf/bitbake.conf | 2 +- > meta/lib/oe/reproducible.py | 19 ++++++++++--------- > 3 files changed, 34 insertions(+), 15 deletions(-) > > diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass > index 066f3848f7c..13e91b24a3b 100644 > --- a/meta/classes-global/base.bbclass > +++ b/meta/classes-global/base.bbclass > @@ -153,20 +153,38 @@ python base_do_fetch() { > } > > addtask unpack after do_fetch > -do_unpack[dirs] = "${UNPACKDIR}" > - > -do_unpack[cleandirs] = "${@d.getVar('S') if os.path.normpath(d.getVar('S')) != os.path.normpath(d.getVar('WORKDIR')) else os.path.join('${S}', 'patches')}" > +do_unpack[cleandirs] = "${UNPACKDIR}" > > python base_do_unpack() { > + import shutil > + > + sourcedir = d.getVar('S') > + # Intentionally keep SOURCE_BASEDIR internal to the task just for SDE > + d.setVar("SOURCE_BASEDIR", sourcedir) > + > src_uri = (d.getVar('SRC_URI') or "").split() > if not src_uri: > return > > + basedir = None > + unpackdir = d.getVar('UNPACKDIR') > + workdir = d.getVar('WORKDIR') > + if sourcedir.startswith(workdir) and not sourcedir.startswith(unpackdir): > + basedir = sourcedir.replace(workdir, '').strip("/").split('/')[0] > + if basedir: > + bb.utils.remove(workdir + '/' + basedir, True) > + d.setVar("SOURCE_BASEDIR", workdir + '/' + basedir) > + > try: > fetcher = bb.fetch2.Fetch(src_uri, d) > fetcher.unpack(d.getVar('UNPACKDIR')) > except bb.fetch2.BBFetchException as e: > bb.fatal("Bitbake Fetcher Error: " + repr(e)) > + > + if basedir and os.path.exists(unpackdir + '/' + basedir): > + # Compatibility magic to ensure ${WORKDIR}/git and ${WORKDIR}/${BP} > + # as often used in S work as expected. > + shutil.move(unpackdir + '/' + basedir, workdir + '/' + basedir) > } > > SSTATETASKS += "do_deploy_source_date_epoch" > @@ -199,8 +217,8 @@ addtask do_deploy_source_date_epoch_setscene > addtask do_deploy_source_date_epoch before do_configure after do_patch > > python create_source_date_epoch_stamp() { > - # Version: 1 > - source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('S')) > + # Version: 2 > + source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('SOURCE_BASEDIR') or d.getVar('S')) > oe.reproducible.epochfile_write(source_date_epoch, d.getVar('SDE_FILE'), d) > } > do_unpack[postfuncs] += "create_source_date_epoch_stamp" The following code is part of the anonymous python() function in base.bbclass: if os.path.normpath(d.getVar("WORKDIR")) != os.path.normpath(d.getVar("S")): d.appendVar("PSEUDO_IGNORE_PATHS", ",${S}") if os.path.normpath(d.getVar("WORKDIR")) != os.path.normpath(d.getVar("B")): d.appendVar("PSEUDO_IGNORE_PATHS", ",${B}") Since it is (or will be) an error to have S = "${WORKDIR}", it should now be possible to move the addition of ${S} to PSEUDO_IGNORE_PATHS to bitbake.conf. I also believe ${UNPACKDIR} should be added to PSEUDO_IGNORE_PATHS. And on a related note, would it make sense to make B = ${WORKDIR} an error too? In which case the addition of ${B} to PSEUDO_IGNORE_PATHS could also be moved to bitbake.conf. > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index b2c500d8739..75c850760f6 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -405,7 +405,7 @@ STAMP = "${STAMPS_DIR}/${MULTIMACH_TARGET_SYS}/${PN}/${PV}" > STAMPCLEAN = "${STAMPS_DIR}/${MULTIMACH_TARGET_SYS}/${PN}/*-*" > BASE_WORKDIR ?= "${TMPDIR}/work" > WORKDIR = "${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}/${PN}/${PV}" > -UNPACKDIR ??= "${WORKDIR}" > +UNPACKDIR ??= "${WORKDIR}/sources-unpack" > T = "${WORKDIR}/temp" > D = "${WORKDIR}/image" > S = "${WORKDIR}/${BP}" > diff --git a/meta/lib/oe/reproducible.py b/meta/lib/oe/reproducible.py > index a9f717159e3..1957c974347 100644 > --- a/meta/lib/oe/reproducible.py > +++ b/meta/lib/oe/reproducible.py > @@ -75,10 +75,10 @@ def get_source_date_epoch_from_known_files(d, sourcedir): > return source_date_epoch > > def find_git_folder(d, sourcedir): > - # First guess: WORKDIR/git > + # First guess: UNPACKDIR/git > # This is the default git fetcher unpack path > - workdir = d.getVar('WORKDIR') > - gitpath = os.path.join(workdir, "git/.git") > + unpackdir = d.getVar('UNPACKDIR') > + gitpath = os.path.join(unpackdir, "git/.git") > if os.path.isdir(gitpath): > return gitpath > > @@ -88,15 +88,16 @@ def find_git_folder(d, sourcedir): > return gitpath > > # Perhaps there was a subpath or destsuffix specified. > - # Go looking in the WORKDIR > - exclude = set(["build", "image", "license-destdir", "patches", "pseudo", > - "recipe-sysroot", "recipe-sysroot-native", "sysroot-destdir", "temp"]) > - for root, dirs, files in os.walk(workdir, topdown=True): > - dirs[:] = [d for d in dirs if d not in exclude] > + # Go looking in the UNPACKDIR > + for root, dirs, files in os.walk(unpackdir, topdown=True): > if '.git' in dirs: > return os.path.join(root, ".git") > > - bb.warn("Failed to find a git repository in WORKDIR: %s" % workdir) > + for root, dirs, files in os.walk(sourcedir, topdown=True): > + if '.git' in dirs: > + return os.path.join(root, ".git") > + > + bb.warn("Failed to find a git repository in UNPACKDIR: %s" % unpackdir) > return None > > def get_source_date_epoch_from_git(d, sourcedir): > -- > 2.40.1 //Peter
On Wed, 2024-05-22 at 13:00 +0000, Peter Kjellerstedt wrote: > > -----Original Message----- > > From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Richard Purdie > > Sent: den 22 maj 2024 11:28 > > To: openembedded-core@lists.openembedded.org > > Subject: [OE-core] [PATCH 1/3] base: Switch UNPACKDIR to a subdir of WORKDIR > > > > Change do_unpack to unpack files to a subdirectory of WORKDIR instead of WORKDIR > > itself. There are several good reasons for this but it is mainly about being able > > to isolate the output of the unpack task and tell the files apart from other things > > which are created in workdir (logs, sysroots, temp dirs and more). > > > > This means that when the do_unpack task reruns, we can clean UNPACKDIR and know > > we have a standard point to start builds from. > > > > It also makes code in tools like devtool and recipetool easier. > > > > To reduce the impact to users, if a subdirectory under UNPACKDIR matches > > the first subdirectory under WORKDIR of S, that directory is moved into position > > inside WORKDIR. This preserves the behaviour of S = "${WORKDIR}/git", > > S = "${WORKDIR}/${BPN}" and other commonly used source directory setups. > > > > The directory is moved since sadly many autotools based projects can't cope with > > symlinks in their paths. > > > > The patch also updates reproducible and SOURCE_DATE_EPOCH handling to > > match the new potential source locations. We can get rid of the horrible > > list of hardcoded directories in WORKDIR to ignore from that code. > > > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > > --- > > meta/classes-global/base.bbclass | 28 +++++++++++++++++++++++----- > > meta/conf/bitbake.conf | 2 +- > > meta/lib/oe/reproducible.py | 19 ++++++++++--------- > > 3 files changed, 34 insertions(+), 15 deletions(-) > > > > diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass > > index 066f3848f7c..13e91b24a3b 100644 > > --- a/meta/classes-global/base.bbclass > > +++ b/meta/classes-global/base.bbclass > > @@ -153,20 +153,38 @@ python base_do_fetch() { > > } > > > > addtask unpack after do_fetch > > -do_unpack[dirs] = "${UNPACKDIR}" > > - > > -do_unpack[cleandirs] = "${@d.getVar('S') if os.path.normpath(d.getVar('S')) != os.path.normpath(d.getVar('WORKDIR')) else os.path.join('${S}', 'patches')}" > > +do_unpack[cleandirs] = "${UNPACKDIR}" > > > > python base_do_unpack() { > > + import shutil > > + > > + sourcedir = d.getVar('S') > > + # Intentionally keep SOURCE_BASEDIR internal to the task just for SDE > > + d.setVar("SOURCE_BASEDIR", sourcedir) > > + > > src_uri = (d.getVar('SRC_URI') or "").split() > > if not src_uri: > > return > > > > + basedir = None > > + unpackdir = d.getVar('UNPACKDIR') > > + workdir = d.getVar('WORKDIR') > > + if sourcedir.startswith(workdir) and not sourcedir.startswith(unpackdir): > > + basedir = sourcedir.replace(workdir, '').strip("/").split('/')[0] > > + if basedir: > > + bb.utils.remove(workdir + '/' + basedir, True) > > + d.setVar("SOURCE_BASEDIR", workdir + '/' + basedir) > > + > > try: > > fetcher = bb.fetch2.Fetch(src_uri, d) > > fetcher.unpack(d.getVar('UNPACKDIR')) > > except bb.fetch2.BBFetchException as e: > > bb.fatal("Bitbake Fetcher Error: " + repr(e)) > > + > > + if basedir and os.path.exists(unpackdir + '/' + basedir): > > + # Compatibility magic to ensure ${WORKDIR}/git and ${WORKDIR}/${BP} > > + # as often used in S work as expected. > > + shutil.move(unpackdir + '/' + basedir, workdir + '/' + basedir) > > } > > > > SSTATETASKS += "do_deploy_source_date_epoch" > > @@ -199,8 +217,8 @@ addtask do_deploy_source_date_epoch_setscene > > addtask do_deploy_source_date_epoch before do_configure after do_patch > > > > python create_source_date_epoch_stamp() { > > - # Version: 1 > > - source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('S')) > > + # Version: 2 > > + source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('SOURCE_BASEDIR') or d.getVar('S')) > > oe.reproducible.epochfile_write(source_date_epoch, d.getVar('SDE_FILE'), d) > > } > > do_unpack[postfuncs] += "create_source_date_epoch_stamp" > > The following code is part of the anonymous python() function in base.bbclass: > > if os.path.normpath(d.getVar("WORKDIR")) != os.path.normpath(d.getVar("S")): > d.appendVar("PSEUDO_IGNORE_PATHS", ",${S}") > if os.path.normpath(d.getVar("WORKDIR")) != os.path.normpath(d.getVar("B")): > d.appendVar("PSEUDO_IGNORE_PATHS", ",${B}") > > Since it is (or will be) an error to have S = "${WORKDIR}", it should now be > possible to move the addition of ${S} to PSEUDO_IGNORE_PATHS to bitbake.conf. Yes, we should be able to start finding and cleaning up things like this. > I also believe ${UNPACKDIR} should be added to PSEUDO_IGNORE_PATHS. Why? I don't think we want UNPACKDIR there, we only need S. > And on a related note, would it make sense to make B = ${WORKDIR} an error too? > In which case the addition of ${B} to PSEUDO_IGNORE_PATHS could also be moved > to bitbake.conf. I'd hope no one is doing that so yes, I'd also like to move that. We can make that an error too. Cheers, Richard
On Wed, May 22, 2024 at 3:09 PM Richard Purdie via lists.openembedded.org <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote: > > On Wed, 2024-05-22 at 13:00 +0000, Peter Kjellerstedt wrote: > > > -----Original Message----- > > > From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Richard Purdie > > > Sent: den 22 maj 2024 11:28 > > > To: openembedded-core@lists.openembedded.org > > > Subject: [OE-core] [PATCH 1/3] base: Switch UNPACKDIR to a subdir of WORKDIR > > > > > > Change do_unpack to unpack files to a subdirectory of WORKDIR instead of WORKDIR > > > itself. There are several good reasons for this but it is mainly about being able > > > to isolate the output of the unpack task and tell the files apart from other things > > > which are created in workdir (logs, sysroots, temp dirs and more). > > > > > > This means that when the do_unpack task reruns, we can clean UNPACKDIR and know > > > we have a standard point to start builds from. > > > > > > It also makes code in tools like devtool and recipetool easier. > > > > > > To reduce the impact to users, if a subdirectory under UNPACKDIR matches > > > the first subdirectory under WORKDIR of S, that directory is moved into position > > > inside WORKDIR. This preserves the behaviour of S = "${WORKDIR}/git", > > > S = "${WORKDIR}/${BPN}" and other commonly used source directory setups. > > > > > > The directory is moved since sadly many autotools based projects can't cope with > > > symlinks in their paths. > > > > > > The patch also updates reproducible and SOURCE_DATE_EPOCH handling to > > > match the new potential source locations. We can get rid of the horrible > > > list of hardcoded directories in WORKDIR to ignore from that code. > > > > > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > > > --- > > > meta/classes-global/base.bbclass | 28 +++++++++++++++++++++++----- > > > meta/conf/bitbake.conf | 2 +- > > > meta/lib/oe/reproducible.py | 19 ++++++++++--------- > > > 3 files changed, 34 insertions(+), 15 deletions(-) > > > > > > diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass > > > index 066f3848f7c..13e91b24a3b 100644 > > > --- a/meta/classes-global/base.bbclass > > > +++ b/meta/classes-global/base.bbclass > > > @@ -153,20 +153,38 @@ python base_do_fetch() { > > > } > > > > > > addtask unpack after do_fetch > > > -do_unpack[dirs] = "${UNPACKDIR}" > > > - > > > -do_unpack[cleandirs] = "${@d.getVar('S') if os.path.normpath(d.getVar('S')) != os.path.normpath(d.getVar('WORKDIR')) else os.path.join('${S}', 'patches')}" > > > +do_unpack[cleandirs] = "${UNPACKDIR}" > > > > > > python base_do_unpack() { > > > + import shutil > > > + > > > + sourcedir = d.getVar('S') > > > + # Intentionally keep SOURCE_BASEDIR internal to the task just for SDE > > > + d.setVar("SOURCE_BASEDIR", sourcedir) > > > + > > > src_uri = (d.getVar('SRC_URI') or "").split() > > > if not src_uri: > > > return > > > > > > + basedir = None > > > + unpackdir = d.getVar('UNPACKDIR') > > > + workdir = d.getVar('WORKDIR') > > > + if sourcedir.startswith(workdir) and not sourcedir.startswith(unpackdir): > > > + basedir = sourcedir.replace(workdir, '').strip("/").split('/')[0] > > > + if basedir: > > > + bb.utils.remove(workdir + '/' + basedir, True) > > > + d.setVar("SOURCE_BASEDIR", workdir + '/' + basedir) > > > + > > > try: > > > fetcher = bb.fetch2.Fetch(src_uri, d) > > > fetcher.unpack(d.getVar('UNPACKDIR')) > > > except bb.fetch2.BBFetchException as e: > > > bb.fatal("Bitbake Fetcher Error: " + repr(e)) > > > + > > > + if basedir and os.path.exists(unpackdir + '/' + basedir): > > > + # Compatibility magic to ensure ${WORKDIR}/git and ${WORKDIR}/${BP} > > > + # as often used in S work as expected. > > > + shutil.move(unpackdir + '/' + basedir, workdir + '/' + basedir) > > > } > > > > > > SSTATETASKS += "do_deploy_source_date_epoch" > > > @@ -199,8 +217,8 @@ addtask do_deploy_source_date_epoch_setscene > > > addtask do_deploy_source_date_epoch before do_configure after do_patch > > > > > > python create_source_date_epoch_stamp() { > > > - # Version: 1 > > > - source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('S')) > > > + # Version: 2 > > > + source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('SOURCE_BASEDIR') or d.getVar('S')) > > > oe.reproducible.epochfile_write(source_date_epoch, d.getVar('SDE_FILE'), d) > > > } > > > do_unpack[postfuncs] += "create_source_date_epoch_stamp" > > > > The following code is part of the anonymous python() function in base.bbclass: > > > > if os.path.normpath(d.getVar("WORKDIR")) != os.path.normpath(d.getVar("S")): > > d.appendVar("PSEUDO_IGNORE_PATHS", ",${S}") > > if os.path.normpath(d.getVar("WORKDIR")) != os.path.normpath(d.getVar("B")): > > d.appendVar("PSEUDO_IGNORE_PATHS", ",${B}") > > > > Since it is (or will be) an error to have S = "${WORKDIR}", it should now be > > possible to move the addition of ${S} to PSEUDO_IGNORE_PATHS to bitbake.conf. > > Yes, we should be able to start finding and cleaning up things like > this. > > > I also believe ${UNPACKDIR} should be added to PSEUDO_IGNORE_PATHS. > > Why? I don't think we want UNPACKDIR there, we only need S. There were recently couple issues with pseudo when the source file from something not unpacked under ${S} confuses pseudo when it sees the same file unpacked by do_unpack and then linked from package/usr/src/debug e.g.: path mismatch [3 links]: ino 214373575 db 'opencv/4.5.5-r0/package/usr/src/debug/lib32-opencv/4.5.5-r0/contrib/modules/intensity_transform/src/bimef.cpp' req 'opencv/4.5.5-r0/contrib/modules/intensity_transform/src/bimef.cpp'. which can be usually easily fixed by moving the additional files (or other repositories) to be unpacked/checked out as subdirectory under ${S} to make them ignored by pseudo as well. As in: https://git.openembedded.org/meta-openembedded/commit/?h=kirkstone&id=da98a75f37830e69ded0207cc6c73182ab00ec50 which moved ${WORKDIR}/contrib to ${WORKDIR}/git/contrib to avoid this issue. But now with clearer separation thanks to ${UNPACKDIR} this whouldn't be needed if we add whole UNPACKDIR to PSEUDO_IGNORE_PATHS. Is there any case when UNPACKDIR needs to be tracked by pseudo? > > And on a related note, would it make sense to make B = ${WORKDIR} an error too? > > In which case the addition of ${B} to PSEUDO_IGNORE_PATHS could also be moved > > to bitbake.conf. > > I'd hope no one is doing that so yes, I'd also like to move that. We > can make that an error too. > > Cheers, > > Richard > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#199750): https://lists.openembedded.org/g/openembedded-core/message/199750 > Mute This Topic: https://lists.openembedded.org/mt/106239726/3617156 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [martin.jansa@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Wed, 2024-05-22 at 15:19 +0200, Martin Jansa wrote: > On Wed, May 22, 2024 at 3:09 PM Richard Purdie via > > > The following code is part of the anonymous python() function in base.bbclass: > > > > > > if os.path.normpath(d.getVar("WORKDIR")) != os.path.normpath(d.getVar("S")): > > > d.appendVar("PSEUDO_IGNORE_PATHS", ",${S}") > > > if os.path.normpath(d.getVar("WORKDIR")) != os.path.normpath(d.getVar("B")): > > > d.appendVar("PSEUDO_IGNORE_PATHS", ",${B}") > > > > > > Since it is (or will be) an error to have S = "${WORKDIR}", it should now be > > > possible to move the addition of ${S} to PSEUDO_IGNORE_PATHS to bitbake.conf. > > > > Yes, we should be able to start finding and cleaning up things like > > this. > > > > > I also believe ${UNPACKDIR} should be added to PSEUDO_IGNORE_PATHS. > > > > Why? I don't think we want UNPACKDIR there, we only need S. > > There were recently couple issues with pseudo when the source file > from something not unpacked under ${S} confuses pseudo when it sees > the same file unpacked by do_unpack and then linked from > package/usr/src/debug e.g.: > > path mismatch [3 links]: ino 214373575 db > 'opencv/4.5.5-r0/package/usr/src/debug/lib32-opencv/4.5.5-r0/contrib/modules/intensity_transform/src/bimef.cpp' > req 'opencv/4.5.5-r0/contrib/modules/intensity_transform/src/bimef.cpp'. > > which can be usually easily fixed by moving the additional files (or > other repositories) to be unpacked/checked out as subdirectory under > ${S} to make them ignored by pseudo as well. > > As in: > https://git.openembedded.org/meta-openembedded/commit/?h=kirkstone&id=da98a75f37830e69ded0207cc6c73182ab00ec50 > which moved ${WORKDIR}/contrib to ${WORKDIR}/git/contrib to avoid this issue. > > But now with clearer separation thanks to ${UNPACKDIR} this whouldn't > be needed if we add whole UNPACKDIR to PSEUDO_IGNORE_PATHS. > > Is there any case when UNPACKDIR needs to be tracked by pseudo? I'm thinking about this backwards, too many things in parallel. We could add UNPACKDIR to there, yes. You'd really hope that isn't needed though in general :/. It may be time to think about switching the paths list to the ones we want pseudo to cover rather than ignore though, which the creation of UNPACKDIR and dropping S = WORKDIR further helps with. Cheers, Richard
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass index 066f3848f7c..13e91b24a3b 100644 --- a/meta/classes-global/base.bbclass +++ b/meta/classes-global/base.bbclass @@ -153,20 +153,38 @@ python base_do_fetch() { } addtask unpack after do_fetch -do_unpack[dirs] = "${UNPACKDIR}" - -do_unpack[cleandirs] = "${@d.getVar('S') if os.path.normpath(d.getVar('S')) != os.path.normpath(d.getVar('WORKDIR')) else os.path.join('${S}', 'patches')}" +do_unpack[cleandirs] = "${UNPACKDIR}" python base_do_unpack() { + import shutil + + sourcedir = d.getVar('S') + # Intentionally keep SOURCE_BASEDIR internal to the task just for SDE + d.setVar("SOURCE_BASEDIR", sourcedir) + src_uri = (d.getVar('SRC_URI') or "").split() if not src_uri: return + basedir = None + unpackdir = d.getVar('UNPACKDIR') + workdir = d.getVar('WORKDIR') + if sourcedir.startswith(workdir) and not sourcedir.startswith(unpackdir): + basedir = sourcedir.replace(workdir, '').strip("/").split('/')[0] + if basedir: + bb.utils.remove(workdir + '/' + basedir, True) + d.setVar("SOURCE_BASEDIR", workdir + '/' + basedir) + try: fetcher = bb.fetch2.Fetch(src_uri, d) fetcher.unpack(d.getVar('UNPACKDIR')) except bb.fetch2.BBFetchException as e: bb.fatal("Bitbake Fetcher Error: " + repr(e)) + + if basedir and os.path.exists(unpackdir + '/' + basedir): + # Compatibility magic to ensure ${WORKDIR}/git and ${WORKDIR}/${BP} + # as often used in S work as expected. + shutil.move(unpackdir + '/' + basedir, workdir + '/' + basedir) } SSTATETASKS += "do_deploy_source_date_epoch" @@ -199,8 +217,8 @@ addtask do_deploy_source_date_epoch_setscene addtask do_deploy_source_date_epoch before do_configure after do_patch python create_source_date_epoch_stamp() { - # Version: 1 - source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('S')) + # Version: 2 + source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('SOURCE_BASEDIR') or d.getVar('S')) oe.reproducible.epochfile_write(source_date_epoch, d.getVar('SDE_FILE'), d) } do_unpack[postfuncs] += "create_source_date_epoch_stamp" diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index b2c500d8739..75c850760f6 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -405,7 +405,7 @@ STAMP = "${STAMPS_DIR}/${MULTIMACH_TARGET_SYS}/${PN}/${PV}" STAMPCLEAN = "${STAMPS_DIR}/${MULTIMACH_TARGET_SYS}/${PN}/*-*" BASE_WORKDIR ?= "${TMPDIR}/work" WORKDIR = "${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}/${PN}/${PV}" -UNPACKDIR ??= "${WORKDIR}" +UNPACKDIR ??= "${WORKDIR}/sources-unpack" T = "${WORKDIR}/temp" D = "${WORKDIR}/image" S = "${WORKDIR}/${BP}" diff --git a/meta/lib/oe/reproducible.py b/meta/lib/oe/reproducible.py index a9f717159e3..1957c974347 100644 --- a/meta/lib/oe/reproducible.py +++ b/meta/lib/oe/reproducible.py @@ -75,10 +75,10 @@ def get_source_date_epoch_from_known_files(d, sourcedir): return source_date_epoch def find_git_folder(d, sourcedir): - # First guess: WORKDIR/git + # First guess: UNPACKDIR/git # This is the default git fetcher unpack path - workdir = d.getVar('WORKDIR') - gitpath = os.path.join(workdir, "git/.git") + unpackdir = d.getVar('UNPACKDIR') + gitpath = os.path.join(unpackdir, "git/.git") if os.path.isdir(gitpath): return gitpath @@ -88,15 +88,16 @@ def find_git_folder(d, sourcedir): return gitpath # Perhaps there was a subpath or destsuffix specified. - # Go looking in the WORKDIR - exclude = set(["build", "image", "license-destdir", "patches", "pseudo", - "recipe-sysroot", "recipe-sysroot-native", "sysroot-destdir", "temp"]) - for root, dirs, files in os.walk(workdir, topdown=True): - dirs[:] = [d for d in dirs if d not in exclude] + # Go looking in the UNPACKDIR + for root, dirs, files in os.walk(unpackdir, topdown=True): if '.git' in dirs: return os.path.join(root, ".git") - bb.warn("Failed to find a git repository in WORKDIR: %s" % workdir) + for root, dirs, files in os.walk(sourcedir, topdown=True): + if '.git' in dirs: + return os.path.join(root, ".git") + + bb.warn("Failed to find a git repository in UNPACKDIR: %s" % unpackdir) return None def get_source_date_epoch_from_git(d, sourcedir):
Change do_unpack to unpack files to a subdirectory of WORKDIR instead of WORKDIR itself. There are several good reasons for this but it is mainly about being able to isolate the output of the unpack task and tell the files apart from other things which are created in workdir (logs, sysroots, temp dirs and more). This means that when the do_unpack task reruns, we can clean UNPACKDIR and know we have a standard point to start builds from. It also makes code in tools like devtool and recipetool easier. To reduce the impact to users, if a subdirectory under UNPACKDIR matches the first subdirectory under WORKDIR of S, that directory is moved into position inside WORKDIR. This preserves the behaviour of S = "${WORKDIR}/git", S = "${WORKDIR}/${BPN}" and other commonly used source directory setups. The directory is moved since sadly many autotools based projects can't cope with symlinks in their paths. The patch also updates reproducible and SOURCE_DATE_EPOCH handling to match the new potential source locations. We can get rid of the horrible list of hardcoded directories in WORKDIR to ignore from that code. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> --- meta/classes-global/base.bbclass | 28 +++++++++++++++++++++++----- meta/conf/bitbake.conf | 2 +- meta/lib/oe/reproducible.py | 19 ++++++++++--------- 3 files changed, 34 insertions(+), 15 deletions(-)