diff mbox series

[V3] go.bbclass: set buildid to empty string to improve reproducibility

Message ID 20260108060842.3340603-1-changqing.li@windriver.com
State Under Review
Headers show
Series [V3] go.bbclass: set buildid to empty string to improve reproducibility | expand

Commit Message

Changqing Li Jan. 8, 2026, 6:08 a.m. UTC 
From: Changqing Li <changqing.li@windriver.com>

Go packages and binaries are stamped with build IDs that record both the
action ID, which is a hash of the inputs to the action that produced the
packages or binary, and the content ID, which is a hash of the action
output, namely the archive or binary itself, Refer [1].

And action ID include hash of modroot, which will include build path,
so this make go package not reproducible.
Refer [2], keying off module path instead of module root directory is a TODO.

[snip of log]
HASH[moduleIndex]: "go1.25.3"
HASH[moduleIndex]: "modroot /build-a/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot-native/usr/lib/go/src/cmd\n"
HASH[moduleIndex]: "package go1.25.3 go index v2 /build-a/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot-native/usr/lib/go/src/cmd/buildid\n"
HASH[moduleIndex]: "file buildid.go 2025-10-13 16:08:43 +0000 UTC 1704\n"
HASH[moduleIndex]: "file doc.go 2025-10-13 16:08:43 +0000 UTC 558\n"
HASH[moduleIndex]: 007b9fe2edd5b3232f5c98ae6c46e80a435141cb627ba5418c5314c0cbf4df7b

Report this issue to upstream, refer [3]
Workaround the reproducible by setting buildid to empty, refer [4]

Refer:
[1] https://github.com/golang/go/blob/master/src/cmd/go/internal/work/buildid.go#L26
[2] https://github.com/golang/go/blob/master/src/cmd/go/internal/modindex/read.go#L70
[1] https://github.com/golang/go/issues/77086
[2] https://github.com/golang/go/issues/34186

Signed-off-by: Changqing Li <changqing.li@windriver.com>
---
 meta/classes-recipe/go.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Comments

Alexander Kanavin Jan. 8, 2026, 9:47 a.m. UTC | #1 
There's go-helloworld in oe-core, which is reproducible as it is. How
can the non-reproducibility that this patch is fixing be observed? Can
we improve our tests for it?

Alex

On Thu, 8 Jan 2026 at 07:08, Changqing Li via lists.openembedded.org
<changqing.li=windriver.com@lists.openembedded.org> wrote:
>
> From: Changqing Li <changqing.li@windriver.com>
>
> Go packages and binaries are stamped with build IDs that record both the
> action ID, which is a hash of the inputs to the action that produced the
> packages or binary, and the content ID, which is a hash of the action
> output, namely the archive or binary itself, Refer [1].
>
> And action ID include hash of modroot, which will include build path,
> so this make go package not reproducible.
> Refer [2], keying off module path instead of module root directory is a TODO.
>
> [snip of log]
> HASH[moduleIndex]: "go1.25.3"
> HASH[moduleIndex]: "modroot /build-a/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot-native/usr/lib/go/src/cmd\n"
> HASH[moduleIndex]: "package go1.25.3 go index v2 /build-a/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot-native/usr/lib/go/src/cmd/buildid\n"
> HASH[moduleIndex]: "file buildid.go 2025-10-13 16:08:43 +0000 UTC 1704\n"
> HASH[moduleIndex]: "file doc.go 2025-10-13 16:08:43 +0000 UTC 558\n"
> HASH[moduleIndex]: 007b9fe2edd5b3232f5c98ae6c46e80a435141cb627ba5418c5314c0cbf4df7b
>
> Report this issue to upstream, refer [3]
> Workaround the reproducible by setting buildid to empty, refer [4]
>
> Refer:
> [1] https://github.com/golang/go/blob/master/src/cmd/go/internal/work/buildid.go#L26
> [2] https://github.com/golang/go/blob/master/src/cmd/go/internal/modindex/read.go#L70
> [1] https://github.com/golang/go/issues/77086
> [2] https://github.com/golang/go/issues/34186
>
> Signed-off-by: Changqing Li <changqing.li@windriver.com>
> ---
>  meta/classes-recipe/go.bbclass | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/go.bbclass b/meta/classes-recipe/go.bbclass
> index b540471ba2..1c43b36fc0 100644
> --- a/meta/classes-recipe/go.bbclass
> +++ b/meta/classes-recipe/go.bbclass
> @@ -51,11 +51,12 @@ GO_RPATH:class-native = "${@'-r ${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE
>  GO_RPATH_LINK:class-native = "${@'-Wl,-rpath-link=${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE}_dynlink' if d.getVar('GO_DYNLINK') else ''}"
>  GO_EXTLDFLAGS ?= "${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS} ${GO_RPATH_LINK} ${LDFLAGS}"
>  GO_LINKMODE ?= ""
> +GO_BUILDID ?= "-buildid="
>  GO_EXTRA_LDFLAGS ?= ""
>  GO_LINUXLOADER ?= "-I ${@get_linuxloader(d)}"
>  # Use system loader. If uninative is used, the uninative loader will be patched automatically
>  GO_LINUXLOADER:class-native = ""
> -GO_LDFLAGS ?= '-ldflags="${GO_RPATH} ${GO_LINKMODE} ${GO_LINUXLOADER} ${GO_EXTRA_LDFLAGS} -extldflags '${GO_EXTLDFLAGS}'"'
> +GO_LDFLAGS ?= '-ldflags="${GO_RPATH} ${GO_LINKMODE} ${GO_LINUXLOADER} ${GO_EXTRA_LDFLAGS} ${GO_BUILDID} -extldflags '${GO_EXTLDFLAGS}'"'
>  export GOBUILDFLAGS ?= "-v ${GO_LDFLAGS} -trimpath"
>  export GOPATH_OMIT_IN_ACTIONID ?= "1"
>  export GOPTESTBUILDFLAGS ?= "${GOBUILDFLAGS} -c"
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#229038): https://lists.openembedded.org/g/openembedded-core/message/229038
> Mute This Topic: https://lists.openembedded.org/mt/117148599/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Changqing Li Jan. 12, 2026, 7:39 a.m. UTC | #2 
On 1/8/26 17:47, Alexander Kanavin wrote:
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> There's go-helloworld in oe-core, which is reproducible as it is. How
> can the non-reproducibility that this patch is fixing be observed? Can
> we improve our tests for it?

This moduleindex info only be used when GO111MODULE=on,  and 
thego-helloworld is compiled with

GO111MODULE=off, so it is reproducible as it is. To improve test, we 
properly need to enable GO111MODULE


This issue can be easily reproduced by "bitbake buildah"  with different 
path on a host. There are also other

recipes under meta-virtualization have this issue. I have also send a 
series of patches to meta-virtualization.

Bruce prefer not set to empty,  but to a value like git hash that values 
for us.

SRCREV seems like a proper value to use, but maybe not all the recipes 
set this value SRCREV,  is it possible that

a recipe use a tarball and SRC_URI[sha256sum]?

and some recipes like  nerdctl,  it set multiple SRCREV,  set a value in 
go.bbclass will make the logic here complicated.

SRCREV_nerdcli = "497c7cf74d09bf1ddf2678382360ca61e6faebac"

SRCREV_sys = "74cfc93a99be6ca6f193856132e6799065b071af"
...


//Sandy

>
> Alex
>
> On Thu, 8 Jan 2026 at 07:08, Changqing Li via lists.openembedded.org
> <changqing.li=windriver.com@lists.openembedded.org> wrote:
>> From: Changqing Li<changqing.li@windriver.com>
>>
>> Go packages and binaries are stamped with build IDs that record both the
>> action ID, which is a hash of the inputs to the action that produced the
>> packages or binary, and the content ID, which is a hash of the action
>> output, namely the archive or binary itself, Refer [1].
>>
>> And action ID include hash of modroot, which will include build path,
>> so this make go package not reproducible.
>> Refer [2], keying off module path instead of module root directory is a TODO.
>>
>> [snip of log]
>> HASH[moduleIndex]: "go1.25.3"
>> HASH[moduleIndex]: "modroot /build-a/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot-native/usr/lib/go/src/cmd\n"
>> HASH[moduleIndex]: "package go1.25.3 go index v2 /build-a/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot-native/usr/lib/go/src/cmd/buildid\n"
>> HASH[moduleIndex]: "file buildid.go 2025-10-13 16:08:43 +0000 UTC 1704\n"
>> HASH[moduleIndex]: "file doc.go 2025-10-13 16:08:43 +0000 UTC 558\n"
>> HASH[moduleIndex]: 007b9fe2edd5b3232f5c98ae6c46e80a435141cb627ba5418c5314c0cbf4df7b
>>
>> Report this issue to upstream, refer [3]
>> Workaround the reproducible by setting buildid to empty, refer [4]
>>
>> Refer:
>> [1]https://github.com/golang/go/blob/master/src/cmd/go/internal/work/buildid.go#L26
>> [2]https://github.com/golang/go/blob/master/src/cmd/go/internal/modindex/read.go#L70
>> [1]https://github.com/golang/go/issues/77086
>> [2]https://github.com/golang/go/issues/34186
>>
>> Signed-off-by: Changqing Li<changqing.li@windriver.com>
>> ---
>>   meta/classes-recipe/go.bbclass | 3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/classes-recipe/go.bbclass b/meta/classes-recipe/go.bbclass
>> index b540471ba2..1c43b36fc0 100644
>> --- a/meta/classes-recipe/go.bbclass
>> +++ b/meta/classes-recipe/go.bbclass
>> @@ -51,11 +51,12 @@ GO_RPATH:class-native = "${@'-r ${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE
>>   GO_RPATH_LINK:class-native ="${@'-Wl,-rpath-link=${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE}_dynlink' 
>> if d.getVar('GO_DYNLINK') else ''}"
>>   GO_EXTLDFLAGS ?= "${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS} ${GO_RPATH_LINK} ${LDFLAGS}"
>>   GO_LINKMODE ?= ""
>> +GO_BUILDID ?= "-buildid="
>>   GO_EXTRA_LDFLAGS ?= ""
>>   GO_LINUXLOADER ?= "-I ${@get_linuxloader(d)}"
>>   # Use system loader. If uninative is used, the uninative loader will be patched automatically
>>   GO_LINUXLOADER:class-native = ""
>> -GO_LDFLAGS ?= '-ldflags="${GO_RPATH} ${GO_LINKMODE} ${GO_LINUXLOADER} ${GO_EXTRA_LDFLAGS} -extldflags '${GO_EXTLDFLAGS}'"'
>> +GO_LDFLAGS ?= '-ldflags="${GO_RPATH} ${GO_LINKMODE} ${GO_LINUXLOADER} ${GO_EXTRA_LDFLAGS} ${GO_BUILDID} -extldflags '${GO_EXTLDFLAGS}'"'
>>   export GOBUILDFLAGS ?= "-v ${GO_LDFLAGS} -trimpath"
>>   export GOPATH_OMIT_IN_ACTIONID ?= "1"
>>   export GOPTESTBUILDFLAGS ?= "${GOBUILDFLAGS} -c"
>> --
>> 2.34.1
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#229038):https://lists.openembedded.org/g/openembedded-core/message/229038
>> Mute This Topic:https://lists.openembedded.org/mt/117148599/1686489
>> Group Owner:openembedded-core+owner@lists.openembedded.org
>> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
Alexander Kanavin Jan. 12, 2026, 9:33 a.m. UTC | #3 
On Mon, 12 Jan 2026 at 08:40, Changqing Li <changqing.li@windriver.com> wrote:
> There's go-helloworld in oe-core, which is reproducible as it is. How
> can the non-reproducibility that this patch is fixing be observed? Can
> we improve our tests for it?
>
> This moduleindex info only be used when GO111MODULE=on,  and the go-helloworld is compiled with
>
> GO111MODULE=off, so it is reproducible as it is. To improve test, we properly need to enable GO111MODULE

Can you do this please, and add the change to the patchset?

Alex
Changqing Li Jan. 13, 2026, 1:36 a.m. UTC | #4 
On 1/12/26 17:33, Alexander Kanavin wrote:
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> On Mon, 12 Jan 2026 at 08:40, Changqing Li <changqing.li@windriver.com> wrote:
>> There's go-helloworld in oe-core, which is reproducible as it is. How
>> can the non-reproducibility that this patch is fixing be observed? Can
>> we improve our tests for it?
>>
>> This moduleindex info only be used when GO111MODULE=on,  and the go-helloworld is compiled with
>>
>> GO111MODULE=off, so it is reproducible as it is. To improve test, we properly need to enable GO111MODULE
> Can you do this please, and add the change to the patchset?

OK, I will change this in V4 patch.

//changqing

>
> Alex
Changqing Li Jan. 14, 2026, 6:51 a.m. UTC | #5 
On 1/13/26 09:36, Changqing Li via lists.openembedded.org wrote:
>
> On 1/12/26 17:33, Alexander Kanavin wrote:
>> CAUTION: This email comes from a non Wind River email account!
>> Do not click links or open attachments unless you recognize the 
>> sender and know the content is safe.
>>
>> On Mon, 12 Jan 2026 at 08:40, Changqing Li 
>> <changqing.li@windriver.com> wrote:
>>> There's go-helloworld in oe-core, which is reproducible as it is. How
>>> can the non-reproducibility that this patch is fixing be observed? Can
>>> we improve our tests for it?
>>>
>>> This moduleindex info only be used when GO111MODULE=on,  and the 
>>> go-helloworld is compiled with
>>>
>>> GO111MODULE=off, so it is reproducible as it is. To improve test, we 
>>> properly need to enable GO111MODULE
>> Can you do this please, and add the change to the patchset?
>
> OK, I will change this in V4 patch.

I don't  include this patch in V4,  because I am considering add another 
example go program, which enable GO111MODULE

and also enable cgo, so we can cover this reproducibility issue and this 
one:

https://git.openembedded.org/openembedded-core/commit/?id=0642d2323072f561a4d0eeb9266213387b2997fc

So another patch will be sent later after above example program is ready.

Only change one minor typo issue in V4.

Regards

Changqing

>
> //changqing
>
>>
>> Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#229221): https://lists.openembedded.org/g/openembedded-core/message/229221
> Mute This Topic: https://lists.openembedded.org/mt/117148599/3616873
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [changqing.li@windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
diff mbox series

Patch

diff --git a/meta/classes-recipe/go.bbclass b/meta/classes-recipe/go.bbclass
index b540471ba2..1c43b36fc0 100644
--- a/meta/classes-recipe/go.bbclass
+++ b/meta/classes-recipe/go.bbclass
@@ -51,11 +51,12 @@  GO_RPATH:class-native = "${@'-r ${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE
 GO_RPATH_LINK:class-native = "${@'-Wl,-rpath-link=${STAGING_LIBDIR_NATIVE}/go/pkg/${TARGET_GOTUPLE}_dynlink' if d.getVar('GO_DYNLINK') else ''}"
 GO_EXTLDFLAGS ?= "${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS} ${GO_RPATH_LINK} ${LDFLAGS}"
 GO_LINKMODE ?= ""
+GO_BUILDID ?= "-buildid="
 GO_EXTRA_LDFLAGS ?= ""
 GO_LINUXLOADER ?= "-I ${@get_linuxloader(d)}"
 # Use system loader. If uninative is used, the uninative loader will be patched automatically
 GO_LINUXLOADER:class-native = ""
-GO_LDFLAGS ?= '-ldflags="${GO_RPATH} ${GO_LINKMODE} ${GO_LINUXLOADER} ${GO_EXTRA_LDFLAGS} -extldflags '${GO_EXTLDFLAGS}'"'
+GO_LDFLAGS ?= '-ldflags="${GO_RPATH} ${GO_LINKMODE} ${GO_LINUXLOADER} ${GO_EXTRA_LDFLAGS} ${GO_BUILDID} -extldflags '${GO_EXTLDFLAGS}'"'
 export GOBUILDFLAGS ?= "-v ${GO_LDFLAGS} -trimpath"
 export GOPATH_OMIT_IN_ACTIONID ?= "1"
 export GOPTESTBUILDFLAGS ?= "${GOBUILDFLAGS} -c"