[1/2] fetch2: Add gomod fetcher

From: Christian Lindeberg <christian.lindeberg@axis.com>

Add a go module fetcher for downloading module dependencies to the
module cache from a module proxy. The fetcher can be used with the
go-mod class in OE-Core.

A module dependency can be specified with:

  SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..."

Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
---
 lib/bb/fetch2/__init__.py |   4 +-
 lib/bb/fetch2/gomod.py    | 116 ++++++++++++++++++++++++++++++++++++++
 lib/bb/tests/fetch.py     |  73 ++++++++++++++++++++++++
 3 files changed, 192 insertions(+), 1 deletion(-)
 create mode 100644 lib/bb/fetch2/gomod.py

On Mon, 2024-09-02 at 11:13 +0200, Christian Lindeberg via lists.openembedded.org wrote:
> From: Christian Lindeberg <christian.lindeberg@axis.com>
> 
> Add a go module fetcher for downloading module dependencies to the
> module cache from a module proxy. The fetcher can be used with the
> go-mod class in OE-Core.
> 
> A module dependency can be specified with:
> 
>   SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..."
> 
> Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
> ---
>  lib/bb/fetch2/__init__.py |   4 +-
>  lib/bb/fetch2/gomod.py    | 116 ++++++++++++++++++++++++++++++++++++++
>  lib/bb/tests/fetch.py     |  73 ++++++++++++++++++++++++
>  3 files changed, 192 insertions(+), 1 deletion(-)
>  create mode 100644 lib/bb/fetch2/gomod.py

These changes look interesting and I'd be interested to hear review
from other people more used to dealing with go.

I see the code using GOMODCACHE in unpack(). In general the fetcher
tries to work off url parameters and we try not to have environment
variables unless they are truly global config. This looks to be a per
recipe local setting.

I guess this is something which any code using go is going to have to
have set into the environment anyway so perhaps it is ok but I'm not
sure I like the precedent.

I also noticed that there are two fetchers in the code, one which
subclasses git and one which subclasses wget, the difference between
them is a direct=git parameter in the url. I did wonder if these should
just be two url types (gomod:// and gomodvcs://)?

It may also be helpful to document in the code exactly what changes to
the underlying base fetcher are being made. I'm a bit worried about how
changes to the git or wget fetcher might break this code, particularly
as it isn't going to be heavily used/tested as far as I can tell.

Cheers,

Richard

On Mon, Sep 2, 2024 at 12:18 PM, Richard Purdie wrote:

> 
> On Mon, 2024-09-02 at 11:13 +0200, Christian Lindeberg via
> lists.openembedded.org wrote:
> 
>> From: Christian Lindeberg <christian.lindeberg@axis.com>
>> 
>> Add a go module fetcher for downloading module dependencies to the
>> module cache from a module proxy. The fetcher can be used with the
>> go-mod class in OE-Core.
>> 
>> A module dependency can be specified with:
>> 
>> SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..."
>> 
>> Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
>> ---
>> lib/bb/fetch2/__init__.py |   4 +-
>> lib/bb/fetch2/gomod.py    | 116 ++++++++++++++++++++++++++++++++++++++
>> lib/bb/tests/fetch.py     |  73 ++++++++++++++++++++++++
>> 3 files changed, 192 insertions(+), 1 deletion(-)
>> create mode 100644 lib/bb/fetch2/gomod.py
> 
> These changes look interesting and I'd be interested to hear review
> from other people more used to dealing with go.
> 
> I see the code using GOMODCACHE in unpack(). In general the fetcher
> tries to work off url parameters and we try not to have environment
> variables unless they are truly global config. This looks to be a per
> recipe local setting.
> 
> I guess this is something which any code using go is going to have to
> have set into the environment anyway so perhaps it is ok but I'm not
> sure I like the precedent.

I was trying to avoid having to define the location both in fetcher and
in go-mod.bbclass/recipe. But would using a directory relative rootdir
in unpack() and adapting go/mod.bbclass/recipe accordingly be preferred?

> 
> I also noticed that there are two fetchers in the code, one which
> subclasses git and one which subclasses wget, the difference between
> them is a direct=git parameter in the url. I did wonder if these should
> just be two url types (gomod:// and gomodvcs://)?

I started with a separate gomodgit:// scheme. Maybe that was clearer
after all, avoiding the extra parameter?

> 
> It may also be helpful to document in the code exactly what changes to
> the underlying base fetcher are being made. I'm a bit worried about how
> changes to the git or wget fetcher might break this code, particularly
> as it isn't going to be heavily used/tested as far as I can tell.

I was hoping the tests would help avoiding that the unpacking breaks.
Maybe verification of the resulting git or wget parameters would make
the assumptions more clear?

> 
> Cheers,
> 
> Richard

Thanks,

Christian

Can the mods be re-used (serving as some kind of mirror) from an 
existing download directory?

Regards,
Qi

On 9/2/24 17:13, Christian Lindeberg wrote:
> From: Christian Lindeberg <christian.lindeberg@axis.com>
>
> Add a go module fetcher for downloading module dependencies to the
> module cache from a module proxy. The fetcher can be used with the
> go-mod class in OE-Core.
>
> A module dependency can be specified with:
>
>    SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..."
>
> Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
> ---
>   lib/bb/fetch2/__init__.py |   4 +-
>   lib/bb/fetch2/gomod.py    | 116 ++++++++++++++++++++++++++++++++++++++
>   lib/bb/tests/fetch.py     |  73 ++++++++++++++++++++++++
>   3 files changed, 192 insertions(+), 1 deletion(-)
>   create mode 100644 lib/bb/fetch2/gomod.py
>
> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> index 5bf2c4b8c..f84ce5999 100644
> --- a/lib/bb/fetch2/__init__.py
> +++ b/lib/bb/fetch2/__init__.py
> @@ -1317,7 +1317,7 @@ class FetchData(object):
>   
>               if checksum_name in self.parm:
>                   checksum_expected = self.parm[checksum_name]
> -            elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate", "gs"]:
> +            elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate", "gs", "gomod"]:
>                   checksum_expected = None
>               else:
>                   checksum_expected = d.getVarFlag("SRC_URI", checksum_name)
> @@ -2088,6 +2088,7 @@ from . import npmsw
>   from . import az
>   from . import crate
>   from . import gcp
> +from . import gomod
>   
>   methods.append(local.Local())
>   methods.append(wget.Wget())
> @@ -2110,3 +2111,4 @@ methods.append(npmsw.NpmShrinkWrap())
>   methods.append(az.Az())
>   methods.append(crate.Crate())
>   methods.append(gcp.GCP())
> +methods.append(gomod.GoMod())
> diff --git a/lib/bb/fetch2/gomod.py b/lib/bb/fetch2/gomod.py
> new file mode 100644
> index 000000000..0675c87e5
> --- /dev/null
> +++ b/lib/bb/fetch2/gomod.py
> @@ -0,0 +1,116 @@
> +"""
> +BitBake 'Fetch' implementation for Go modules
> +
> +The gomod fetcher is used to download Go modules to the module cache from a
> +module proxy.
> +
> +Example SRC_URIs:
> +
> +SRC_URI = "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..."
> +
> +Required SRC_URI parameters:
> +
> +- version
> +    The version of the module.
> +
> +Optional SRC_URI parameters:
> +
> +- mod
> +    Fetch and unpack the go.mod file only instead of the complete module.
> +    The go command may need to download go.mod files for many different modules
> +    when computing the build list, and go.mod files are much smaller than
> +    module zip files.
> +    The default is "0", set mod=1 for the go.mod file only.
> +
> +- sha256sum
> +    The checksum of the module zip file, or the go.mod file in case of fetching
> +    only the go.mod file. Alternatively, set the SRC_URI varible flag for
> +    "module@version.sha256sum".
> +
> +Related variables:
> +
> +- GOMODCACHE
> +    The location of the module cache.
> +    The variable must be exported for the go command to find the downloaded
> +    module cache.
> +
> +- GO_MOD_PROXY
> +    The module proxy used by the fetcher.
> +
> +See the Go modules reference, https://go.dev/ref/mod, for more information
> +about the module cache, module proxies and version control systems.
> +"""
> +
> +import os
> +import re
> +import shutil
> +import zipfile
> +
> +import bb
> +from bb.fetch2 import FetchError, MissingParameterError
> +from bb.fetch2.wget import Wget
> +
> +
> +def escape(path):
> +    """Escape capital letters using exclamation points."""
> +    return re.sub(r'([A-Z])', lambda m: '!' + m.group(1).lower(), path)
> +
> +
> +class GoMod(Wget):
> +    """Class to fetch Go modules from a Go module proxy via wget"""
> +
> +    def supports(self, ud, d):
> +        """Check to see if a given URL is for this fetcher."""
> +        return ud.type == 'gomod'
> +
> +    def urldata_init(self, ud, d):
> +        """Set up to download the module from the module proxy."""
> +
> +        moddir = d.getVar('GOMODCACHE')
> +        if not moddir:
> +            raise FetchError("The module cache location is not specified in the"
> +                             " GOMODCACHE environment variable.")
> +        proxy = d.getVar('GO_MOD_PROXY') or 'proxy.golang.org'
> +
> +        if 'version' not in ud.parm:
> +            raise MissingParameterError('version', ud.url)
> +
> +        module = ud.host + ud.path
> +        ud.parm['module'] = module
> +        path = escape(module + '/@v/' + ud.parm['version'])
> +        if ud.parm.get('mod', '0') == '1':
> +            path += '.mod'
> +        else:
> +            path += '.zip'
> +            ud.parm['unpack'] = '0'
> +        ud.url = bb.fetch2.encodeurl(
> +            ('https', proxy, '/' + path, None, None, None))
> +        ud.parm['downloadfilename'] = path
> +        ud.parm['subdir'] = os.path.join(moddir, 'cache', 'download',
> +                                         os.path.dirname(path))
> +        name = f"{module}@{ud.parm['version']}"
> +        if d.getVarFlag('SRC_URI', name + '.sha256sum'):
> +            ud.parm['name'] = name
> +        super().urldata_init(ud, d)
> +
> +    def unpack(self, ud, rootdir, d):
> +        """Unpack the module in the module cache."""
> +
> +        # Unpack the module zip file or go.mod file
> +        super().unpack(ud, rootdir, d)
> +
> +        if ud.localpath.endswith('.zip'):
> +            # Unpack the go.mod file from the zip file
> +            module = ud.parm['module']
> +            unpackdir = ud.parm['subdir']
> +            name = os.path.basename(ud.localpath).rsplit('.', 1)[0] + '.mod'
> +            bb.note(f"Unpacking {name} to {unpackdir}/")
> +            with zipfile.ZipFile(ud.localpath) as zf:
> +                with open(os.path.join(unpackdir, name), mode='wb') as mf:
> +                    try:
> +                        f = module + '@' + ud.parm['version'] + '/go.mod'
> +                        shutil.copyfileobj(zf.open(f), mf)
> +                    except KeyError:
> +                        # If the module does not have a go.mod file, synthesize
> +                        # one containing only a module statement.
> +                        mf.write(f'module {module}\n'.encode())
> diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
> index 2ef206343..652907af5 100644
> --- a/lib/bb/tests/fetch.py
> +++ b/lib/bb/tests/fetch.py
> @@ -3390,3 +3390,76 @@ class FetchPremirroronlyBrokenTarball(FetcherTest):
>               fetcher.download()
>           output = "".join(logs.output)
>           self.assertFalse(" not a git repository (or any parent up to mount point /)" in output)
> +
> +class GoModTest(FetcherTest):
> +
> +    @skipIfNoNetwork()
> +    def test_gomod_url(self):
> +        self.d.setVar('GOMODCACHE', os.path.join(self.unpackdir, 'pkg/mod'))
> +
> +        urls = ['gomod://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;'
> +                'sha256sum=9bb69aea32f1d59711701f9562d66432c9c0374205e5009d1d1a62f03fb4fdad']
> +
> +        fetcher = bb.fetch2.Fetch(urls, self.d)
> +        ud = fetcher.ud[urls[0]]
> +        self.assertEqual(ud.url, 'https://proxy.golang.org/github.com/%21azure/azure-sdk-for-go/sdk/storage/azblob/%40v/v1.0.0.zip')
> +        self.assertNotIn('name', ud.parm)
> +
> +        fetcher.download()
> +        fetcher.unpack(self.unpackdir)
> +        downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
> +        self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.zip')))
> +        self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod')))
> +
> +    @skipIfNoNetwork()
> +    def test_gomod_url_mod_only(self):
> +        self.d.setVar('GOMODCACHE', os.path.join(self.unpackdir, 'pkg/mod'))
> +
> +        urls = ['gomod://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;mod=1;'
> +                'sha256sum=7873b8544842329b4f385a3aa6cf82cc2bc8defb41a04fa5291c35fd5900e873']
> +
> +        fetcher = bb.fetch2.Fetch(urls, self.d)
> +        ud = fetcher.ud[urls[0]]
> +        self.assertEqual(ud.url, 'https://proxy.golang.org/github.com/%21azure/azure-sdk-for-go/sdk/storage/azblob/%40v/v1.0.0.mod')
> +        self.assertNotIn('name', ud.parm)
> +
> +        fetcher.download()
> +        fetcher.unpack(self.unpackdir)
> +        downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
> +        self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod')))
> +
> +    @skipIfNoNetwork()
> +    def test_gomod_url_sha256sum_varflag(self):
> +        self.d.setVar('GOMODCACHE', os.path.join(self.unpackdir, 'pkg/mod'))
> +
> +        urls = ['gomod://gopkg.in/ini.v1;version=v1.67.0']
> +        self.d.setVarFlag('SRC_URI', 'gopkg.in/ini.v1@v1.67.0.sha256sum', 'bd845dfc762a87a56e5a32a07770dc83e86976db7705d7f89c5dbafdc60b06c6')
> +
> +        fetcher = bb.fetch2.Fetch(urls, self.d)
> +        ud = fetcher.ud[urls[0]]
> +        self.assertEqual(ud.url, 'https://proxy.golang.org/gopkg.in/ini.v1/%40v/v1.67.0.zip')
> +        self.assertEqual(ud.parm['name'], 'gopkg.in/ini.v1@v1.67.0')
> +
> +        fetcher.download()
> +        fetcher.unpack(self.unpackdir)
> +        downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
> +        self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip')))
> +        self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')))
> +
> +    @skipIfNoNetwork()
> +    def test_gomod_url_no_mod_in_zip(self):
> +        self.d.setVar('GOMODCACHE', os.path.join(self.unpackdir, 'pkg/mod'))
> +
> +        urls = ['gomod://gopkg.in/ini.v1;version=v1.67.0;'
> +                'sha256sum=bd845dfc762a87a56e5a32a07770dc83e86976db7705d7f89c5dbafdc60b06c6']
> +
> +        fetcher = bb.fetch2.Fetch(urls, self.d)
> +        ud = fetcher.ud[urls[0]]
> +        self.assertEqual(ud.url, 'https://proxy.golang.org/gopkg.in/ini.v1/%40v/v1.67.0.zip')
> +        self.assertNotIn('name', ud.parm)
> +
> +        fetcher.download()
> +        fetcher.unpack(self.unpackdir)
> +        downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
> +        self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip')))
> +        self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')))
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#16532): https://lists.openembedded.org/g/bitbake-devel/message/16532
> Mute This Topic: https://lists.openembedded.org/mt/108223851/7304865
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [Qi.Chen@eng.windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

On Tue, Sep 3, 2024 at 08:19 AM, Chen Qi wrote:

> 
> Can the mods be re-used (serving as some kind of mirror) from an
> existing download directory?
> 
> Regards,
> Qi

The module zip files and go.mod files are fetched with the underlying wget
fetcher (or git fecher in the git vcs repository case) so the same principles
with download cache and mirrors apply.

Thanks,
Christian