| Message ID | 20251120114041.193806-1-changqing.li@windriver.com |
|---|---|
| State | New |
| Headers | show |
| Series | bitbake.conf: add new variable FIXED_SOURCE_DATE_EPOCH | expand |
On Thu, 20 Nov 2025 at 12:40, Changqing Li via lists.openembedded.org <changqing.li=windriver.com@lists.openembedded.org> wrote: > > From: Changqing Li <changqing.li@windriver.com> > > Here is SRC_URI of jquery: > SRC_URI = "\ > https://code.jquery.com/${BP}.js;name=js;subdir=${BP} \ > https://code.jquery.com/${BP}.min.js;name=min;subdir=${BP} \ > https://code.jquery.com/${BP}.min.map;name=map;subdir=${BP} \ > " > When ${BP}.js/${BP}.min.js/${BP}.min.map are downloaded in a git repo, > and this git repo is cloned locally as a PREMIRRORS. In this case, mtime > of these files will be current time when checking out. I don't understand. Is SRC_URI tweaked in a .bbappend to something else? Can you provide a way to observe the issue? The bar for adding new variables to bitbake.conf is high, and we should strive to avoid that. Alex
On Thu, 2025-11-20 at 19:40 +0800, Changqing Li via lists.openembedded.org wrote: > From: Changqing Li <changqing.li@windriver.com> > > Here is SRC_URI of jquery: > SRC_URI = "\ > https://code.jquery.com/${BP}.js;name=js;subdir=${BP} \ > https://code.jquery.com/${BP}.min.js;name=min;subdir=${BP} \ > https://code.jquery.com/${BP}.min.map;name=map;subdir=${BP} \ > " > When ${BP}.js/${BP}.min.js/${BP}.min.map are downloaded in a git repo, > and this git repo is cloned locally as a PREMIRRORS. In this case, mtime > of these files will be current time when checking out. For this recipe, > source_date_epoch will get by function > get_source_date_epoch_from_youngest_file, mtime of > ${BP}.js/${BP}.min.js/${BP}.min.map are not stable, so cause the > generated package not reproducible. > > A new variable FIXED_SOURCE_DATE_EPOCH is added to support this case. > By default, FIXED_SOURCE_DATE_EPOCH is 0, and source_date_epoch works > as before. If user build in above case, FIXED_SOURCE_DATE_EPOCH set to 1 > in recipe jquery can ensure fixed source date epoch used by this recipe > to support reproducible build. > > Signed-off-by: Changqing Li <changqing.li@windriver.com> > --- > meta/conf/bitbake.conf | 1 + > meta/lib/oe/reproducible.py | 13 ++++++++----- > 2 files changed, 9 insertions(+), 5 deletions(-) We need to take a step back here and think about what is going on. The current design is that mtimes of files are clamped at an upper value which is determined by the the logic you're changing. There is an assumption that the mtimes don't affect the build, the build process will create files and therefore we will always have to have some kind of ignore threshold as the creation times of built components will vary. As you've pointed out, git can under some circumstances mean that files don't have the correct mtime ordering which causes components to rebuild, particularly around flex/bison. We need to find those cases and ensure the right mtime ordering as you've done in some patches. What I don't think is reasonable is changing the git fetcher to try and have stable mtimes, or adding new variables without a really strong explanation of why the existing code can't work. Cheers, Richard
On 11/20/25 19:46, Alexander Kanavin via lists.openembedded.org wrote: > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On Thu, 20 Nov 2025 at 12:40, Changqing Li via lists.openembedded.org > <changqing.li=windriver.com@lists.openembedded.org> wrote: >> From: Changqing Li<changqing.li@windriver.com> >> >> Here is SRC_URI of jquery: >> SRC_URI = "\ >> https://code.jquery.com/${BP}.js;name=js;subdir=${BP} \ >> https://code.jquery.com/${BP}.min.js;name=min;subdir=${BP} \ >> https://code.jquery.com/${BP}.min.map;name=map;subdir=${BP} \ >> " >> When ${BP}.js/${BP}.min.js/${BP}.min.map are downloaded in a git repo, >> and this git repo is cloned locally as a PREMIRRORS. In this case, mtime >> of these files will be current time when checking out. > I don't understand. Is SRC_URI tweaked in a .bbappend to something > else? Can you provide a way to observe the issue? No, there is no .bbappend. Here is how to reproduce it: we have a git repo, which contains downloades sources, which contains jquery's sources: ${BP}.js, ${BP}.min.js, ${BP}.min.map . Create project under path A, and clone source_mirror under A/sources_mirror, and set A/sources_mirror as the first PREMIRRORS Create another project under path B, and clone source_mirror under B/sources_mirror, and set B/sources_mirror as the first PREMIRRORS In this way, since jquery sources are fetched by git, the files mtime is the current time when the files are checking out. So files mtime in project A is different with project B. Then bitabke jquery under A and B get different rpm package, the different part is "Build Date", which is get from source_date_epoch. According to logic of get_source_date_epoch, source_date_epoch of project will set to the youngest file of the sources. in this case, unstable files mtime cause unstable source_date_epoch. //Changqing > > The bar for adding new variables to bitbake.conf is high, and we > should strive to avoid that. > > Alex > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#226615):https://lists.openembedded.org/g/openembedded-core/message/226615 > Mute This Topic:https://lists.openembedded.org/mt/116389229/3616873 > Group Owner:openembedded-core+owner@lists.openembedded.org > Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [changqing.li@windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On 21 Nov 2025, at 05:46, Changqing Li via lists.openembedded.org <changqing.li=windriver.com@lists.openembedded.org> wrote: > No, there is no .bbappend. Here is how to reproduce it: > we have a git repo, which contains downloades sources, which contains jquery's sources: ${BP}.js, ${BP}.min.js, ${BP}.min.map . > Create project under path A, and clone source_mirror under A/sources_mirror, and set A/sources_mirror as the first PREMIRRORS > Create another project under path B, and clone source_mirror under B/sources_mirror, and set B/sources_mirror as the first PREMIRRORS > In this way, since jquery sources are fetched by git, the files mtime is the current time when the files are checking out. So files mtime in project A is different with project B. > Then bitabke jquery under A and B get different rpm package, the different part is "Build Date", which is get from source_date_epoch. > According to logic of get_source_date_epoch, source_date_epoch of project will set to the youngest file of the sources. > in this case, unstable files mtime cause unstable source_date_epoch. I agree with others on this thread: the fetcher expects that mtimes are preserved, but putting the mirror into a git repository for distribution doesn’t preserve mtimes. The problem here is using git, not the fetcher. This is a relatively niche problem because most upstreams are tarballs or git repos, so the problem doesn’t affect them. There are tools to deal with this problem, for example metastore (https://github.com/przemoc/metastore) which tracks timestamps. (also, we’re only shipping jquery for diffoscope, we can improve the packaging here) Ross
On 12/4/25 22:00, Ross Burton wrote: > CAUTION: This email comes from a non Wind River email account! > Do not click links or open attachments unless you recognize the sender and know the content is safe. > > On 21 Nov 2025, at 05:46, Changqing Li via lists.openembedded.org<changqing.li=windriver.com@lists.openembedded.org> wrote: >> No, there is no .bbappend. Here is how to reproduce it: >> we have a git repo, which contains downloades sources, which contains jquery's sources: ${BP}.js, ${BP}.min.js, ${BP}.min.map . >> Create project under path A, and clone source_mirror under A/sources_mirror, and set A/sources_mirror as the first PREMIRRORS >> Create another project under path B, and clone source_mirror under B/sources_mirror, and set B/sources_mirror as the first PREMIRRORS >> In this way, since jquery sources are fetched by git, the files mtime is the current time when the files are checking out. So files mtime in project A is different with project B. >> Then bitabke jquery under A and B get different rpm package, the different part is "Build Date", which is get from source_date_epoch. >> According to logic of get_source_date_epoch, source_date_epoch of project will set to the youngest file of the sources. >> in this case, unstable files mtime cause unstable source_date_epoch. > I agree with others on this thread: the fetcher expects that mtimes are preserved, but putting the mirror into a git repository for distribution doesn’t preserve mtimes. The problem here is using git, not the fetcher. This is a relatively niche problem because most upstreams are tarballs or git repos, so the problem doesn’t affect them. > > There are tools to deal with this problem, for example metastore (https://github.com/przemoc/metastore) which tracks timestamps. > > (also, we’re only shipping jquery for diffoscope, we can improve the packaging here) Thanks. There is another patch for this issue. https://git.openembedded.org/openembedded-core/commit/?h=master-next&id=7a47c786e98c16c384c6872a662dd3142e7ee9d6 //Changqing > > Ross
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 5406e542db..2371567da3 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -680,6 +680,7 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}" # A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE SOURCE_DATE_EPOCH_FALLBACK ??= "1302044400" REPRODUCIBLE_TIMESTAMP_ROOTFS ??= "1520598896" +FIXED_SOURCE_DATE_EPOCH ?= "0" ################################################################## # Settings used by bitbake-layers. diff --git a/meta/lib/oe/reproducible.py b/meta/lib/oe/reproducible.py index 0270024a83..8ed5e2b458 100644 --- a/meta/lib/oe/reproducible.py +++ b/meta/lib/oe/reproducible.py @@ -158,11 +158,14 @@ def fixed_source_date_epoch(d): return 0 def get_source_date_epoch(d, sourcedir): - return ( - get_source_date_epoch_from_git(d, sourcedir) or - get_source_date_epoch_from_youngest_file(d, sourcedir) or - fixed_source_date_epoch(d) # Last resort - ) + if d.getVar('FIXED_SOURCE_DATE_EPOCH') == '0': + return ( + get_source_date_epoch_from_git(d, sourcedir) or + get_source_date_epoch_from_youngest_file(d, sourcedir) or + fixed_source_date_epoch(d) # Last resort + ) + else: + return fixed_source_date_epoch(d) def epochfile_read(epochfile, d): cached, efile = d.getVar('__CACHED_SOURCE_DATE_EPOCH') or (None, None)