| Message ID | 20251017102015.5296-1-adam.blank.g@gmail.com |
|---|---|
| State | Accepted, archived |
| Commit | fbf0a8c5711f56bd64ae4c9d2c1d5ba884508c9f |
| Headers | show |
| Series | doc: bitbake-user-manual: AUTOREV and SRCREV_FORMAT roles in fetching | expand |
Hi, On Fri Oct 17, 2025 at 12:20 PM CEST, Adam Blank via lists.openembedded.org wrote: > Describe the roles of AUTOREV and SRCREV_FORMAT in the fetching step. > Add their missing descriptions to the variable reference section. > Correct encountered style issues. Thanks! Next, time I think it would be better to split addition and style cleanup in different patches, to make it easier to review. See my comments below otherwise. > > Fixes [YOCTO #14498] > > Signed-off-by: Adam Blank <adam.blank.g@gmail.com> > --- > .../bitbake-user-manual-fetching.rst | 115 ++++++++++++------ > .../bitbake-user-manual-ref-variables.rst | 22 +++- > 2 files changed, 95 insertions(+), 42 deletions(-) > > diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst > index f357765b7..1b5fa3ae1 100644 > --- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst > +++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst > @@ -42,7 +42,7 @@ The instantiation of the fetch class is usually followed by:: > rootdir = l.getVar('UNPACKDIR') > fetcher.unpack(rootdir) > > -This code unpacks the downloaded files to the specified by ``UNPACKDIR``. > +This code unpacks the downloaded files to the specified :term:`UNPACKDIR`. > > .. note:: > > @@ -51,7 +51,7 @@ This code unpacks the downloaded files to the specified by ``UNPACKDIR``. > examine the OpenEmbedded class file ``base.bbclass`` > . > > -The :term:`SRC_URI` and ``UNPACKDIR`` variables are not hardcoded into the > +The :term:`SRC_URI` and :term:`UNPACKDIR` variables are not hardcoded into the > fetcher, since those fetcher methods can be (and are) called with > different variable names. In OpenEmbedded for example, the shared state > (sstate) code uses the fetch module to fetch the sstate files. > @@ -159,19 +159,25 @@ URLs except Git URLs, BitBake uses the common ``unpack`` method. > A number of parameters exist that you can specify within the URL to > govern the behavior of the unpack stage: > > -- *unpack:* Controls whether the URL components are unpacked. If set to > +- *"unpack":* Controls whether the URL components are unpacked. If set to > "1", which is the default, the components are unpacked. If set to > "0", the unpack stage leaves the file alone. This parameter is useful > when you want an archive to be copied in and not be unpacked. > > -- *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether > +- *"dos":* Applies to ``.zip`` and ``.jar`` files and specifies whether > to use DOS line ending conversion on text files. > > -- *striplevel:* Strip specified number of leading components (levels) > - from file names on extraction > +- *"striplevel":* Strip specified number of leading components (levels) > + from file names on extraction. > > -- *subdir:* Unpacks the specific URL to the specified subdirectory > - within the root directory. > +- *"subdir":* Unpacks the specific URL to the specified subdirectory > + within the specified root directory. This path can be further modified > + by fetcher specific parameters. I'm not sure I get what you mean here. Could you give a short example? > + > +- *"name":* Assigns a name to a given component of the :term:`SRC_URI`. > + This component is later referenced by this name when specifying its > + :term:`SRCREV` or :term:`SRC_URI` checksum, or to correctly place its > + revision in the package version string with aid of :term:`SRCREV_FORMAT`. > > The unpack call automatically decompresses and extracts files with ".Z", > ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and > @@ -246,21 +252,19 @@ Some example URLs are as follows:: > introduce ambiguity when parsing URLs that also contain semi-colons, > for example:: > > - SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" > - > + SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" > > Such URLs should should be modified by replacing semi-colons with '&' > characters:: > > - SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" > - > + SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" > > In most cases this should work. Treating semi-colons and '&' in > queries identically is recommended by the World Wide Web Consortium > (W3C). Note that due to the nature of the URL, you may have to > specify the name of the downloaded file as well:: > > - SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" > + SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" > > > .. _cvs-fetcher: > @@ -398,16 +402,16 @@ This fetcher supports the following parameters: > > .. note:: > > - When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs > - from the one that is typically passed to ``git clone`` command and provided > - by the Git server to fetch from. For example, the URL returned by GitLab > - server for ``mesa`` when cloning over SSH is > - ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in > - :term:`SRC_URI` is the following:: > + When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs > + from the one that is typically passed to ``git clone`` command and provided > + by the Git server to fetch from. For example, the URL returned by GitLab > + server for ``mesa`` when cloning over SSH is > + ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in > + :term:`SRC_URI` is the following:: > > - SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." > + SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." > > - Note the ``:`` character changed for a ``/`` before the path to the project. > + Note the ``:`` character changed for a ``/`` before the path to the project. > > - *"nocheckout":* Tells the fetcher to not checkout source code when > unpacking when set to "1". Set this option for the URL where there is > @@ -453,7 +457,7 @@ This fetcher supports the following parameters: > By default, the path is ``git/``. > > - *"usehead":* Enables local ``git://`` URLs to use the current branch > - HEAD as the revision for use with ``AUTOREV``. The "usehead" > + HEAD as the revision for use with :term:`AUTOREV`. The "usehead" > parameter implies no branch and only works when the transfer protocol > is ``file://``. > > @@ -525,10 +529,10 @@ The fetcher uses the ``rcleartool`` or > > Following are options for the :term:`SRC_URI` statement: > > -- *vob*: The name, which must include the prepending "/" character, > +- *"vob":*: The name, which must include the prepending "/" character, > of the ClearCase VOB. This option is required. > > -- *module*: The module, which must include the prepending "/" > +- *"module":*: The module, which must include the prepending "/" > character, in the selected VOB. > > .. note:: > @@ -540,7 +544,7 @@ Following are options for the :term:`SRC_URI` statement: > > load /example_vob/example_module > > -- *proto*: The protocol, which can be either ``http`` or ``https``. > +- *"proto"*: The protocol, which can be either ``http`` or ``https``. > > By default, the fetcher creates a configuration specification. If you > want this specification written to an area other than the default, use > @@ -549,9 +553,9 @@ the specification is written. > > .. note:: > > - the SRCREV loses its functionality if you specify this variable. However, > - SRCREV is still used to label the archive after a fetch even though it does > - not define what is fetched. > + the :term:`SRCREV` loses its functionality if you specify this variable. > + However, :term:`SRCREV` is still used to label the archive after a fetch even > + though it does not define what is fetched. > > Here are a couple of other behaviors worth mentioning: > > @@ -606,21 +610,23 @@ password, and fetches a Revision based on a Label:: > > .. note:: > > - You should always set S to "${UNPACKDIR}/p4" in your recipe. > + Your recipe should always set S as shown in the above example:: s/above/following ? > + > + S = "${UNPACKDIR}/p4" > > By default, the fetcher strips the depot location from the local file paths. In > the above example, the content of ``example-depot/main/source/`` will be placed > in ``${UNPACKDIR}/p4``. For situations where preserving parts of the remote depot > paths locally is desirable, the fetcher supports two parameters: > > -- *"module":* > - The top-level depot location or directory to fetch. The value of this > - parameter can also point to a single file within the depot, in which case > - the local file path will include the module path. > -- *"remotepath":* > - When used with the value "``keep``", the fetcher will mirror the full depot > - paths locally for the specified location, even in combination with the > - ``module`` parameter. > +- *"module":* > + The top-level depot location or directory to fetch. The value of this > + parameter can also point to a single file within the depot, in which case > + the local file path will include the module path. > +- *"remotepath":* > + When used with the value "``keep``", the fetcher will mirror the full depot > + paths locally for the specified location, even in combination with the > + ``module`` parameter. > > Here is an example use of the the ``module`` parameter:: > > @@ -843,4 +849,37 @@ submodules. However, you might find the code helpful and readable. > Auto Revisions > ============== > > -We need to document ``AUTOREV`` and :term:`SRCREV_FORMAT` here. > +For recipes which need to use the latest revision of their source code, > +the way to achieve it is to use :term:`AUTOREV` as the value of the > +source code repository's :term:`SRCREV`:: Suggestion: simplify the sentence to: """ For recipes which need to use the latest revision of their source code, the way to achieve it is to set :term:`SRCREV` to :term:`AUTOREV` in the recipe:: """ > + > + SRCREV = "${AUTOREV}" > + > +In this case, to properly show the revision information in the package's version > +string, :term:`PV` should be set to include the ``+`` sign. I'm quite sure this logic is not part of Bitbake but OE-Core: https://git.openembedded.org/openembedded-core/tree/meta/classes-global/package.bbclass#n317 I'm not sure having the "+" in PV adds anything for standalone Bitbake, so I'd remove this sentence. > + > +.. note:: > + > + With :term:`AUTOREV`, BitBake will always need to take the additional step of > + querying the remore repository to retrieve the latest available revision. s/remore/remote > + > + Also, recipes using it don't benefit from the caching mechanism. I just made a quick test: - Set SRCREV = "${AUTOREV}" for the procps recipe in OE-Core. - bitbake -c unpack procps -> unpacks the source code - bitbake -c unpack procps -> "Attempted 3 tasks of which 3 didn't need to be rerun and all succeeded." So I think bitbake does not rebuild when the remote hasn't changed, and recipes do benefit from the sstate-cache? > +Multiple Source Control Repositories > +==================================== > + > +For some recipes it is necessary to make use of more than one > +version controlled source code repository. It such case, the recipe s/It/In/ > +must provide BitBake with information about how it should include > +all those SCM versions in its package version string, instead of its s/all those SCM versions/the different SCM revisions/ would sound better IMO. > +usual approach with a single :term:`SRCREV`. > + > +For this purpose, the recipe must set the :term:`SRCREV_FORMAT` > +variable. One notable example is the OpenEmbedded ``kernel-yocto.bbclass``:: It would be nice to keep example free of OpenEmbedded references. Can you convert it to another example, without mentioning it? > + > + SRCREV_FORMAT ?= "meta_machine" > + > +The value given to :term:`SRCREV_FORMAT` references names, which were Suggestion: """ The value given to :term:`SRCREV_FORMAT` references names separated by underscores ("_"), which are assigned... """ > +assigned using the ``name`` parameter to those components of :term:`SRC_URI`, Suggestion: """ assigned using the ``name`` parameter in an :term:`SRC_URI` definition. """ > +which represent the version controlled source code repositories. In the above > +example, the :term:`SRC_URI` contained two URLs named "meta" and "machine". IMO, it is important to give an example of the corresponding SRC_URI definition here. Users reading this will understand better with the full picture. > diff --git a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > index 810f88689..9256c20b6 100644 > --- a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > +++ b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > @@ -36,6 +36,14 @@ overview of their function and contents. > when specified allows for the Git binary from the host to be used > rather than building ``git-native``. > > + :term:`AUTOREV` > + This is a special variable used during fetching. When :term:`SRCREV` is > + set to the value of this variable, it means that the latest revision > + from the version controlled source code repository should be used. Suggestion: """ ...set to the value of this variable, the latest revision from the version controlled source code repository is used. """ > + Here is an example:: """ It should be set as follows: """ There's no other way of using AUTOREV I think, so it's not really an example. > + > + SRCREV = "${AUTOREV}" > + > :term:`AZ_SAS` > Azure Storage Shared Access Signature, when using the > :ref:`Azure Storage fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` > @@ -1622,7 +1630,8 @@ overview of their function and contents. > > - ``name``: Specifies a name to be used for association with > :term:`SRC_URI` checksums or :term:`SRCREV` when you have more than one > - file or git repository specified in :term:`SRC_URI`. For example:: > + file or source control repository specified in :term:`SRC_URI`. > + For example:: > > SRC_URI = "git://example.com/foo.git;branch=main;name=first \ > git://example.com/bar.git;branch=main;name=second \ > @@ -1635,7 +1644,8 @@ overview of their function and contents. > - ``subdir``: Places the file (or extracts its contents) into the > specified subdirectory. This option is useful for unusual tarballs > or other archives that do not have their files already in a > - subdirectory within the archive. > + subdirectory within the archive. This path can be further modified > + by fetcher specific parameters. Ditto: I'm not sure I get what you mean here. Could you give a short example maybe? > > - ``subpath``: Limits the checkout to a specific subpath of the > tree when using the Git fetcher is used. > @@ -1657,11 +1667,11 @@ overview of their function and contents. > identifier and not just a tag. > > :term:`SRCREV_FORMAT` > - Helps construct valid :term:`SRCREV` values when > + Helps construct a valid package version string when I'm not sure why this changed. This variable does help constructing multiple SRCREV values. Package version strings are just a by-product of using it, no? > multiple source controlled URLs are used in > :term:`SRC_URI`. > > - The system needs help constructing these values under these > + The system needs help constructing this value under these > circumstances. Each component in the :term:`SRC_URI` is assigned a name > and these are referenced in the :term:`SRCREV_FORMAT` variable. Consider > an example with URLs named "machine" and "meta". In this case, > @@ -1697,3 +1707,7 @@ overview of their function and contents. > :term:`TOPDIR` > Points to the build directory. BitBake automatically sets this > variable. > + > + :term:`UNPACKDIR` > + The directory into which recipe's downloads are unpacked during its > + build process. Thanks, Antonin
On Fri, 17 Oct 2025 at 18:06, Antonin Godard <antonin.godard@bootlin.com> wrote: > Hi, > > On Fri Oct 17, 2025 at 12:20 PM CEST, Adam Blank via > lists.openembedded.org wrote: > > Describe the roles of AUTOREV and SRCREV_FORMAT in the fetching step. > > Add their missing descriptions to the variable reference section. > > Correct encountered style issues. > > Thanks! Next, time I think it would be better to split addition and style > cleanup in different patches, to make it easier to review. See my comments > below > otherwise. > Yes, sure, I'll upload the next version as two separate changes. > > > > Fixes [YOCTO #14498] > > > > Signed-off-by: Adam Blank <adam.blank.g@gmail.com> > > --- > > .../bitbake-user-manual-fetching.rst | 115 ++++++++++++------ > > .../bitbake-user-manual-ref-variables.rst | 22 +++- > > 2 files changed, 95 insertions(+), 42 deletions(-) > > > > diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst > b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst > > index f357765b7..1b5fa3ae1 100644 > > --- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst > > +++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst > > @@ -42,7 +42,7 @@ The instantiation of the fetch class is usually > followed by:: > > rootdir = l.getVar('UNPACKDIR') > > fetcher.unpack(rootdir) > > > > -This code unpacks the downloaded files to the specified by > ``UNPACKDIR``. > > +This code unpacks the downloaded files to the specified > :term:`UNPACKDIR`. > > > > .. note:: > > > > @@ -51,7 +51,7 @@ This code unpacks the downloaded files to the > specified by ``UNPACKDIR``. > > examine the OpenEmbedded class file ``base.bbclass`` > > . > > > > -The :term:`SRC_URI` and ``UNPACKDIR`` variables are not hardcoded into > the > > +The :term:`SRC_URI` and :term:`UNPACKDIR` variables are not hardcoded > into the > > fetcher, since those fetcher methods can be (and are) called with > > different variable names. In OpenEmbedded for example, the shared state > > (sstate) code uses the fetch module to fetch the sstate files. > > @@ -159,19 +159,25 @@ URLs except Git URLs, BitBake uses the common > ``unpack`` method. > > A number of parameters exist that you can specify within the URL to > > govern the behavior of the unpack stage: > > > > -- *unpack:* Controls whether the URL components are unpacked. If set to > > +- *"unpack":* Controls whether the URL components are unpacked. If set > to > > "1", which is the default, the components are unpacked. If set to > > "0", the unpack stage leaves the file alone. This parameter is useful > > when you want an archive to be copied in and not be unpacked. > > > > -- *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether > > +- *"dos":* Applies to ``.zip`` and ``.jar`` files and specifies whether > > to use DOS line ending conversion on text files. > > > > -- *striplevel:* Strip specified number of leading components (levels) > > - from file names on extraction > > +- *"striplevel":* Strip specified number of leading components (levels) > > + from file names on extraction. > > > > -- *subdir:* Unpacks the specific URL to the specified subdirectory > > - within the root directory. > > +- *"subdir":* Unpacks the specific URL to the specified subdirectory > > + within the specified root directory. This path can be further > modified > > + by fetcher specific parameters. > > I'm not sure I get what you mean here. Could you give a short example? > > Sure, this is what the Git fetcher does for example (those are the actual code fragments): subdir = ud.parm.get("subdir") destdir = os.path.join(destdir, subdir) destdir = = os.path.join(destdir, destsuffix) As you can see, the Git specific 'destsuffix' will be daisy-chained with the more generic 'subdir'. > > + > > +- *"name":* Assigns a name to a given component of the :term:`SRC_URI`. > > + This component is later referenced by this name when specifying its > > + :term:`SRCREV` or :term:`SRC_URI` checksum, or to correctly place its > > + revision in the package version string with aid of > :term:`SRCREV_FORMAT`. > > > > The unpack call automatically decompresses and extracts files with ".Z", > > ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and > > @@ -246,21 +252,19 @@ Some example URLs are as follows:: > > introduce ambiguity when parsing URLs that also contain semi-colons, > > for example:: > > > > - SRC_URI = " > http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" > > - > > + SRC_URI = " > http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" > > > > Such URLs should should be modified by replacing semi-colons with '&' > > characters:: > > > > - SRC_URI = " > http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" > > - > > + SRC_URI = " > http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" > > > > In most cases this should work. Treating semi-colons and '&' in > > queries identically is recommended by the World Wide Web Consortium > > (W3C). Note that due to the nature of the URL, you may have to > > specify the name of the downloaded file as well:: > > > > - SRC_URI = " > http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2 > " > > + SRC_URI = " > http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2 > " > > > > > > .. _cvs-fetcher: > > @@ -398,16 +402,16 @@ This fetcher supports the following parameters: > > > > .. note:: > > > > - When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` > differs > > - from the one that is typically passed to ``git clone`` command and > provided > > - by the Git server to fetch from. For example, the URL returned by > GitLab > > - server for ``mesa`` when cloning over SSH is > > - ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the > expected URL in > > - :term:`SRC_URI` is the following:: > > + When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` > differs > > + from the one that is typically passed to ``git clone`` command > and provided > > + by the Git server to fetch from. For example, the URL returned by > GitLab > > + server for ``mesa`` when cloning over SSH is > > + ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the > expected URL in > > + :term:`SRC_URI` is the following:: > > > > - SRC_URI = "git:// > git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." > > + SRC_URI = "git:// > git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." > > > > - Note the ``:`` character changed for a ``/`` before the path to > the project. > > + Note the ``:`` character changed for a ``/`` before the path to > the project. > > > > - *"nocheckout":* Tells the fetcher to not checkout source code when > > unpacking when set to "1". Set this option for the URL where there is > > @@ -453,7 +457,7 @@ This fetcher supports the following parameters: > > By default, the path is ``git/``. > > > > - *"usehead":* Enables local ``git://`` URLs to use the current branch > > - HEAD as the revision for use with ``AUTOREV``. The "usehead" > > + HEAD as the revision for use with :term:`AUTOREV`. The "usehead" > > parameter implies no branch and only works when the transfer protocol > > is ``file://``. > > > > @@ -525,10 +529,10 @@ The fetcher uses the ``rcleartool`` or > > > > Following are options for the :term:`SRC_URI` statement: > > > > -- *vob*: The name, which must include the prepending "/" character, > > +- *"vob":*: The name, which must include the prepending "/" character, > > of the ClearCase VOB. This option is required. > > > > -- *module*: The module, which must include the prepending "/" > > +- *"module":*: The module, which must include the prepending "/" > > character, in the selected VOB. > > > > .. note:: > > @@ -540,7 +544,7 @@ Following are options for the :term:`SRC_URI` > statement: > > > > load /example_vob/example_module > > > > -- *proto*: The protocol, which can be either ``http`` or ``https``. > > +- *"proto"*: The protocol, which can be either ``http`` or ``https``. > > > > By default, the fetcher creates a configuration specification. If you > > want this specification written to an area other than the default, use > > @@ -549,9 +553,9 @@ the specification is written. > > > > .. note:: > > > > - the SRCREV loses its functionality if you specify this variable. > However, > > - SRCREV is still used to label the archive after a fetch even though > it does > > - not define what is fetched. > > + the :term:`SRCREV` loses its functionality if you specify this > variable. > > + However, :term:`SRCREV` is still used to label the archive after a > fetch even > > + though it does not define what is fetched. > > > > Here are a couple of other behaviors worth mentioning: > > > > @@ -606,21 +610,23 @@ password, and fetches a Revision based on a Label:: > > > > .. note:: > > > > - You should always set S to "${UNPACKDIR}/p4" in your recipe. > > + Your recipe should always set S as shown in the above example:: > > s/above/following ? > > > + > > + S = "${UNPACKDIR}/p4" > > > > By default, the fetcher strips the depot location from the local file > paths. In > > the above example, the content of ``example-depot/main/source/`` will > be placed > > in ``${UNPACKDIR}/p4``. For situations where preserving parts of the > remote depot > > paths locally is desirable, the fetcher supports two parameters: > > > > -- *"module":* > > - The top-level depot location or directory to fetch. The value of > this > > - parameter can also point to a single file within the depot, in > which case > > - the local file path will include the module path. > > -- *"remotepath":* > > - When used with the value "``keep``", the fetcher will mirror the > full depot > > - paths locally for the specified location, even in combination with > the > > - ``module`` parameter. > > +- *"module":* > > + The top-level depot location or directory to fetch. The value of this > > + parameter can also point to a single file within the depot, in which > case > > + the local file path will include the module path. > > +- *"remotepath":* > > + When used with the value "``keep``", the fetcher will mirror the > full depot > > + paths locally for the specified location, even in combination with > the > > + ``module`` parameter. > > > > Here is an example use of the the ``module`` parameter:: > > > > @@ -843,4 +849,37 @@ submodules. However, you might find the code > helpful and readable. > > Auto Revisions > > ============== > > > > -We need to document ``AUTOREV`` and :term:`SRCREV_FORMAT` here. > > +For recipes which need to use the latest revision of their source code, > > +the way to achieve it is to use :term:`AUTOREV` as the value of the > > +source code repository's :term:`SRCREV`:: > > Suggestion: simplify the sentence to: > > """ > For recipes which need to use the latest revision of their source code, > the way to achieve it is to set :term:`SRCREV` to :term:`AUTOREV` in the > recipe:: > """ > > I beg to differ. As a non-native English speaker, I would understand this to mean 'SRCREV = "AUTOREV"', which is a different thing entirely. In my opinion those few extra words are a small cost compared to the comprehensibility they provide. > > + > > + SRCREV = "${AUTOREV}" > > + > > +In this case, to properly show the revision information in the > package's version > > +string, :term:`PV` should be set to include the ``+`` sign. > > I'm quite sure this logic is not part of Bitbake but OE-Core: > > https://git.openembedded.org/openembedded-core/tree/meta/classes-global/package.bbclass#n317 > > I'm not sure having the "+" in PV adds anything for standalone Bitbake, so > I'd > remove this sentence. > > Sure, no problem. > > + > > +.. note:: > > + > > + With :term:`AUTOREV`, BitBake will always need to take the > additional step of > > + querying the remore repository to retrieve the latest available > revision. > > s/remore/remote > > > + > > + Also, recipes using it don't benefit from the caching mechanism. > > I just made a quick test: > > - Set SRCREV = "${AUTOREV}" for the procps recipe in OE-Core. > - bitbake -c unpack procps -> unpacks the source code > - bitbake -c unpack procps -> "Attempted 3 tasks of which 3 didn't need to > be > rerun and all succeeded." > > So I think bitbake does not rebuild when the remote hasn't changed, and > recipes > do benefit from the sstate-cache? > > I did the same test, but adding the following line: python() { bb.error("Reparsing %s" % d.getVar('PN')) } Having this line, calling 'bitbake procs' shows that it gets reparsed every time in the case where it uses AUTOREV. Regardless, my intention was to hint to the user, that the effects of the following code can be expected (again, taken from the Git fetcher): def _latest_revision(self, ud, d, name): bb.fetch2.mark_recipe_nocache(d) I don't know what (if any) would be a better choice of words. Perhaps this? """ don't benefit from all the caching mechanisms """ > > +Multiple Source Control Repositories > > +==================================== > > + > > +For some recipes it is necessary to make use of more than one > > +version controlled source code repository. It such case, the recipe > > s/It/In/ > > > +must provide BitBake with information about how it should include > > +all those SCM versions in its package version string, instead of its > > s/all those SCM versions/the different SCM revisions/ > > would sound better IMO. > > > +usual approach with a single :term:`SRCREV`. > > + > > +For this purpose, the recipe must set the :term:`SRCREV_FORMAT` > > +variable. One notable example is the OpenEmbedded > ``kernel-yocto.bbclass``:: > > It would be nice to keep example free of OpenEmbedded references. Can you > convert it to another example, without mentioning it? > > Of course. I was merely attempting to stick to the 'convention' from this file, where base.bbclass was referenced. > > + > > + SRCREV_FORMAT ?= "meta_machine" > > > + > > +The value given to :term:`SRCREV_FORMAT` references names, which were > > Suggestion: > > """ > The value given to :term:`SRCREV_FORMAT` references names separated by > underscores ("_"), which are assigned... > """ > > Hmmm. What would be the grounds for that? There is no such limitation in the code. In fact, even such abomination gets through: 'SRCREV_FORMAT = "adam@machine@meta@blank"' and successfully produces a package: Image--5.15.173+git0+adam@fa3f7978db@567f0adb9d@blank-r0-beetroot -20251018125101.bin > > +assigned using the ``name`` parameter to those components of > :term:`SRC_URI`, > > Suggestion: > > """ > assigned using the ``name`` parameter in an :term:`SRC_URI` definition. > """ > > > +which represent the version controlled source code repositories. In the > above > > +example, the :term:`SRC_URI` contained two URLs named "meta" and > "machine". > > IMO, it is important to give an example of the corresponding SRC_URI > definition > here. Users reading this will understand better with the full picture. > > Sure thing. > > diff --git > a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > > index 810f88689..9256c20b6 100644 > > --- a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > > +++ b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > > @@ -36,6 +36,14 @@ overview of their function and contents. > > when specified allows for the Git binary from the host to be used > > rather than building ``git-native``. > > > > + :term:`AUTOREV` > > + This is a special variable used during fetching. When > :term:`SRCREV` is > > + set to the value of this variable, it means that the latest > revision > > + from the version controlled source code repository should be used. > > Suggestion: > > """ > ...set to the value of this variable, the latest revision from the version > controlled source code repository is used. > """ > > > + Here is an example:: > > """ > It should be set as follows: > """ > > There's no other way of using AUTOREV I think, so it's not really an > example. > > > + > > + SRCREV = "${AUTOREV}" > > + > > :term:`AZ_SAS` > > Azure Storage Shared Access Signature, when using the > > :ref:`Azure Storage fetcher > <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` > > @@ -1622,7 +1630,8 @@ overview of their function and contents. > > > > - ``name``: Specifies a name to be used for association with > > :term:`SRC_URI` checksums or :term:`SRCREV` when you have more > than one > > - file or git repository specified in :term:`SRC_URI`. For > example:: > > + file or source control repository specified in :term:`SRC_URI`. > > + For example:: > > > > SRC_URI = "git://example.com/foo.git;branch=main;name=first > \ > > git:// > example.com/bar.git;branch=main;name=second \ > > @@ -1635,7 +1644,8 @@ overview of their function and contents. > > - ``subdir``: Places the file (or extracts its contents) into the > > specified subdirectory. This option is useful for unusual > tarballs > > or other archives that do not have their files already in a > > - subdirectory within the archive. > > + subdirectory within the archive. This path can be further > modified > > + by fetcher specific parameters. > > Ditto: I'm not sure I get what you mean here. Could you give a short > example > maybe? > > > > > - ``subpath``: Limits the checkout to a specific subpath of the > > tree when using the Git fetcher is used. > > @@ -1657,11 +1667,11 @@ overview of their function and contents. > > identifier and not just a tag. > > > > :term:`SRCREV_FORMAT` > > - Helps construct valid :term:`SRCREV` values when > > + Helps construct a valid package version string when > > I'm not sure why this changed. This variable does help constructing > multiple > SRCREV values. Package version strings are just a by-product of using it, > no? > > Perhaps it is again a point of view of a non-native English speaker, but it sounded false to my ear. The way it was phrased suggested, that SRCREV_FORMAT is in some way a prerequisite or an input to the final value of some SRCREV* variables. If fact though, it is used solely as a formatter, into which Bitbake will plug the already pre-existing values of 'named' SRCREV variables, to provide a string which will later be used (outside of BitBake) for PKGV. > > multiple source controlled URLs are used in > > :term:`SRC_URI`. > > > > - The system needs help constructing these values under these > > + The system needs help constructing this value under these > > circumstances. Each component in the :term:`SRC_URI` is assigned > a name > > and these are referenced in the :term:`SRCREV_FORMAT` variable. > Consider > > an example with URLs named "machine" and "meta". In this case, > > @@ -1697,3 +1707,7 @@ overview of their function and contents. > > :term:`TOPDIR` > > Points to the build directory. BitBake automatically sets this > > variable. > > + > > + :term:`UNPACKDIR` > > + The directory into which recipe's downloads are unpacked during > its > > + build process. > > Thanks, > Antonin > > -- > Antonin Godard, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > >
On Sat Oct 18, 2025 at 3:18 PM CEST, Adam Blank wrote: [...] >> > -- *subdir:* Unpacks the specific URL to the specified subdirectory >> > - within the root directory. >> > +- *"subdir":* Unpacks the specific URL to the specified subdirectory >> > + within the specified root directory. This path can be further >> modified >> > + by fetcher specific parameters. >> >> I'm not sure I get what you mean here. Could you give a short example? >> >> > Sure, this is what the Git fetcher does for example (those are the actual > code fragments): > subdir = ud.parm.get("subdir") > destdir = os.path.join(destdir, subdir) > destdir = = os.path.join(destdir, destsuffix) > As you can see, the Git specific 'destsuffix' will be daisy-chained with > the more generic 'subdir'. Ah I see, ok, thanks. [...] >> > Auto Revisions >> > ============== >> > >> > -We need to document ``AUTOREV`` and :term:`SRCREV_FORMAT` here. >> > +For recipes which need to use the latest revision of their source code, >> > +the way to achieve it is to use :term:`AUTOREV` as the value of the >> > +source code repository's :term:`SRCREV`:: >> >> Suggestion: simplify the sentence to: >> >> """ >> For recipes which need to use the latest revision of their source code, >> the way to achieve it is to set :term:`SRCREV` to :term:`AUTOREV` in the >> recipe:: >> """ >> >> > I beg to differ. As a non-native English speaker, I would understand this > to mean > 'SRCREV = "AUTOREV"', which is a different thing entirely. In my opinion > those few extra > words are a small cost compared to the comprehensibility they provide. I understand it your way now which is clearer. >> > + >> > + SRCREV = "${AUTOREV}" >> > + >> > +In this case, to properly show the revision information in the >> package's version >> > +string, :term:`PV` should be set to include the ``+`` sign. >> >> I'm quite sure this logic is not part of Bitbake but OE-Core: >> >> https://git.openembedded.org/openembedded-core/tree/meta/classes-global/package.bbclass#n317 >> >> I'm not sure having the "+" in PV adds anything for standalone Bitbake, so >> I'd >> remove this sentence. >> >> > Sure, no problem. > > >> > + >> > +.. note:: >> > + >> > + With :term:`AUTOREV`, BitBake will always need to take the >> additional step of >> > + querying the remore repository to retrieve the latest available >> revision. >> >> s/remore/remote >> >> > + >> > + Also, recipes using it don't benefit from the caching mechanism. >> >> I just made a quick test: >> >> - Set SRCREV = "${AUTOREV}" for the procps recipe in OE-Core. >> - bitbake -c unpack procps -> unpacks the source code >> - bitbake -c unpack procps -> "Attempted 3 tasks of which 3 didn't need to >> be >> rerun and all succeeded." >> >> So I think bitbake does not rebuild when the remote hasn't changed, and >> recipes >> do benefit from the sstate-cache? >> >> > I did the same test, but adding the following line: > python() { > bb.error("Reparsing %s" % d.getVar('PN')) > } > Having this line, calling 'bitbake procs' shows that it gets reparsed every > time in the case > where it uses AUTOREV. > > Regardless, my intention was to hint to the user, that the effects of the > following code can be > expected (again, taken from the Git fetcher): > def _latest_revision(self, ud, d, name): > bb.fetch2.mark_recipe_nocache(d) > > I don't know what (if any) would be a better choice of words. Perhaps this? > """ > don't benefit from all the caching mechanisms > """ I see, I assumed sstate-cache. I think what is correct to say is: "this recipe is not part of the parsing-time cache and so is parsed everytime BitBake is run." > >> > +Multiple Source Control Repositories >> > +==================================== >> > + >> > +For some recipes it is necessary to make use of more than one >> > +version controlled source code repository. It such case, the recipe >> >> s/It/In/ >> >> > +must provide BitBake with information about how it should include >> > +all those SCM versions in its package version string, instead of its >> >> s/all those SCM versions/the different SCM revisions/ >> >> would sound better IMO. >> >> > +usual approach with a single :term:`SRCREV`. >> > + >> > +For this purpose, the recipe must set the :term:`SRCREV_FORMAT` >> > +variable. One notable example is the OpenEmbedded >> ``kernel-yocto.bbclass``:: >> >> It would be nice to keep example free of OpenEmbedded references. Can you >> convert it to another example, without mentioning it? >> >> > Of course. I was merely attempting to stick to the 'convention' from this > file, where base.bbclass > was referenced. > > >> > + >> > + SRCREV_FORMAT ?= "meta_machine" >> >> > + >> > +The value given to :term:`SRCREV_FORMAT` references names, which were >> >> Suggestion: >> >> """ >> The value given to :term:`SRCREV_FORMAT` references names separated by >> underscores ("_"), which are assigned... >> """ >> >> > Hmmm. What would be the grounds for that? There is no such limitation in > the code. > In fact, even such abomination gets through: > 'SRCREV_FORMAT = "adam@machine@meta@blank"' and successfully produces a > package: > Image--5.15.173+git0+adam@fa3f7978db@567f0adb9d@blank-r0-beetroot > -20251018125101.bin I wasn't aware other characters could be used to separate entried here. Thanks! >> > +assigned using the ``name`` parameter to those components of >> :term:`SRC_URI`, >> >> Suggestion: >> >> """ >> assigned using the ``name`` parameter in an :term:`SRC_URI` definition. >> """ >> >> > +which represent the version controlled source code repositories. In the >> above >> > +example, the :term:`SRC_URI` contained two URLs named "meta" and >> "machine". >> >> IMO, it is important to give an example of the corresponding SRC_URI >> definition >> here. Users reading this will understand better with the full picture. >> >> > Sure thing. > > >> > diff --git >> a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst >> b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst >> > index 810f88689..9256c20b6 100644 >> > --- a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst >> > +++ b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst >> > @@ -36,6 +36,14 @@ overview of their function and contents. >> > when specified allows for the Git binary from the host to be used >> > rather than building ``git-native``. >> > >> > + :term:`AUTOREV` >> > + This is a special variable used during fetching. When >> :term:`SRCREV` is >> > + set to the value of this variable, it means that the latest >> revision >> > + from the version controlled source code repository should be used. >> >> Suggestion: >> >> """ >> ...set to the value of this variable, the latest revision from the version >> controlled source code repository is used. >> """ >> >> > + Here is an example:: >> >> """ >> It should be set as follows: >> """ >> >> There's no other way of using AUTOREV I think, so it's not really an >> example. >> >> > + >> > + SRCREV = "${AUTOREV}" >> > + >> > :term:`AZ_SAS` >> > Azure Storage Shared Access Signature, when using the >> > :ref:`Azure Storage fetcher >> <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` >> > @@ -1622,7 +1630,8 @@ overview of their function and contents. >> > >> > - ``name``: Specifies a name to be used for association with >> > :term:`SRC_URI` checksums or :term:`SRCREV` when you have more >> than one >> > - file or git repository specified in :term:`SRC_URI`. For >> example:: >> > + file or source control repository specified in :term:`SRC_URI`. >> > + For example:: >> > >> > SRC_URI = "git://example.com/foo.git;branch=main;name=first >> \ >> > git:// >> example.com/bar.git;branch=main;name=second \ >> > @@ -1635,7 +1644,8 @@ overview of their function and contents. >> > - ``subdir``: Places the file (or extracts its contents) into the >> > specified subdirectory. This option is useful for unusual >> tarballs >> > or other archives that do not have their files already in a >> > - subdirectory within the archive. >> > + subdirectory within the archive. This path can be further >> modified >> > + by fetcher specific parameters. >> >> Ditto: I'm not sure I get what you mean here. Could you give a short >> example >> maybe? >> >> > >> > - ``subpath``: Limits the checkout to a specific subpath of the >> > tree when using the Git fetcher is used. >> > @@ -1657,11 +1667,11 @@ overview of their function and contents. >> > identifier and not just a tag. >> > >> > :term:`SRCREV_FORMAT` >> > - Helps construct valid :term:`SRCREV` values when >> > + Helps construct a valid package version string when >> >> I'm not sure why this changed. This variable does help constructing >> multiple >> SRCREV values. Package version strings are just a by-product of using it, >> no? >> >> > Perhaps it is again a point of view of a non-native English speaker, but it > sounded false to > my ear. The way it was phrased suggested, that SRCREV_FORMAT is in some way > a > prerequisite or an input to the final value of some SRCREV* variables. If > fact though, it is used solely as > a formatter, into which Bitbake will plug the already pre-existing values > of 'named' > SRCREV variables, to provide a string which will later be used (outside of > BitBake) for PKGV. I see that clearly now, understood. Thanks, Antonin
Thanks for your comments. I've already sent this change in the form of 3 separate patches. Patch 1/3 shall be discarded as pointed out by Alex Kanavin. Patch 2/3 (the main one) will be updated with your comments from this topic. Patch 3/3 is purely for "style and completeness". Thanks, Adam On Wed, 22 Oct 2025 at 09:31, Antonin Godard <antonin.godard@bootlin.com> wrote: > On Sat Oct 18, 2025 at 3:18 PM CEST, Adam Blank wrote: > [...] > >> > -- *subdir:* Unpacks the specific URL to the specified subdirectory > >> > - within the root directory. > >> > +- *"subdir":* Unpacks the specific URL to the specified subdirectory > >> > + within the specified root directory. This path can be further > >> modified > >> > + by fetcher specific parameters. > >> > >> I'm not sure I get what you mean here. Could you give a short example? > >> > >> > > Sure, this is what the Git fetcher does for example (those are the actual > > code fragments): > > subdir = ud.parm.get("subdir") > > destdir = os.path.join(destdir, subdir) > > destdir = = os.path.join(destdir, destsuffix) > > As you can see, the Git specific 'destsuffix' will be daisy-chained with > > the more generic 'subdir'. > > Ah I see, ok, thanks. > > [...] > >> > Auto Revisions > >> > ============== > >> > > >> > -We need to document ``AUTOREV`` and :term:`SRCREV_FORMAT` here. > >> > +For recipes which need to use the latest revision of their source > code, > >> > +the way to achieve it is to use :term:`AUTOREV` as the value of the > >> > +source code repository's :term:`SRCREV`:: > >> > >> Suggestion: simplify the sentence to: > >> > >> """ > >> For recipes which need to use the latest revision of their source code, > >> the way to achieve it is to set :term:`SRCREV` to :term:`AUTOREV` in the > >> recipe:: > >> """ > >> > >> > > I beg to differ. As a non-native English speaker, I would understand this > > to mean > > 'SRCREV = "AUTOREV"', which is a different thing entirely. In my opinion > > those few extra > > words are a small cost compared to the comprehensibility they provide. > > I understand it your way now which is clearer. > > >> > + > >> > + SRCREV = "${AUTOREV}" > >> > + > >> > +In this case, to properly show the revision information in the > >> package's version > >> > +string, :term:`PV` should be set to include the ``+`` sign. > >> > >> I'm quite sure this logic is not part of Bitbake but OE-Core: > >> > >> > https://git.openembedded.org/openembedded-core/tree/meta/classes-global/package.bbclass#n317 > >> > >> I'm not sure having the "+" in PV adds anything for standalone Bitbake, > so > >> I'd > >> remove this sentence. > >> > >> > > Sure, no problem. > > > > > >> > + > >> > +.. note:: > >> > + > >> > + With :term:`AUTOREV`, BitBake will always need to take the > >> additional step of > >> > + querying the remore repository to retrieve the latest available > >> revision. > >> > >> s/remore/remote > >> > >> > + > >> > + Also, recipes using it don't benefit from the caching mechanism. > >> > >> I just made a quick test: > >> > >> - Set SRCREV = "${AUTOREV}" for the procps recipe in OE-Core. > >> - bitbake -c unpack procps -> unpacks the source code > >> - bitbake -c unpack procps -> "Attempted 3 tasks of which 3 didn't need > to > >> be > >> rerun and all succeeded." > >> > >> So I think bitbake does not rebuild when the remote hasn't changed, and > >> recipes > >> do benefit from the sstate-cache? > >> > >> > > I did the same test, but adding the following line: > > python() { > > bb.error("Reparsing %s" % d.getVar('PN')) > > } > > Having this line, calling 'bitbake procs' shows that it gets reparsed > every > > time in the case > > where it uses AUTOREV. > > > > Regardless, my intention was to hint to the user, that the effects of the > > following code can be > > expected (again, taken from the Git fetcher): > > def _latest_revision(self, ud, d, name): > > bb.fetch2.mark_recipe_nocache(d) > > > > I don't know what (if any) would be a better choice of words. Perhaps > this? > > """ > > don't benefit from all the caching mechanisms > > """ > > I see, I assumed sstate-cache. I think what is correct to say is: "this > recipe > is not part of the parsing-time cache and so is parsed everytime BitBake > is run." > > > > >> > +Multiple Source Control Repositories > >> > +==================================== > >> > + > >> > +For some recipes it is necessary to make use of more than one > >> > +version controlled source code repository. It such case, the recipe > >> > >> s/It/In/ > >> > >> > +must provide BitBake with information about how it should include > >> > +all those SCM versions in its package version string, instead of its > >> > >> s/all those SCM versions/the different SCM revisions/ > >> > >> would sound better IMO. > >> > >> > +usual approach with a single :term:`SRCREV`. > >> > + > >> > +For this purpose, the recipe must set the :term:`SRCREV_FORMAT` > >> > +variable. One notable example is the OpenEmbedded > >> ``kernel-yocto.bbclass``:: > >> > >> It would be nice to keep example free of OpenEmbedded references. Can > you > >> convert it to another example, without mentioning it? > >> > >> > > Of course. I was merely attempting to stick to the 'convention' from this > > file, where base.bbclass > > was referenced. > > > > > >> > + > >> > + SRCREV_FORMAT ?= "meta_machine" > >> > >> > + > >> > +The value given to :term:`SRCREV_FORMAT` references names, which were > >> > >> Suggestion: > >> > >> """ > >> The value given to :term:`SRCREV_FORMAT` references names separated by > >> underscores ("_"), which are assigned... > >> """ > >> > >> > > Hmmm. What would be the grounds for that? There is no such limitation in > > the code. > > In fact, even such abomination gets through: > > 'SRCREV_FORMAT = "adam@machine@meta@blank"' and successfully produces a > > package: > > Image--5.15.173+git0+adam@fa3f7978db@567f0adb9d@blank-r0-beetroot > > -20251018125101.bin > > I wasn't aware other characters could be used to separate entried here. > Thanks! > > >> > +assigned using the ``name`` parameter to those components of > >> :term:`SRC_URI`, > >> > >> Suggestion: > >> > >> """ > >> assigned using the ``name`` parameter in an :term:`SRC_URI` definition. > >> """ > >> > >> > +which represent the version controlled source code repositories. In > the > >> above > >> > +example, the :term:`SRC_URI` contained two URLs named "meta" and > >> "machine". > >> > >> IMO, it is important to give an example of the corresponding SRC_URI > >> definition > >> here. Users reading this will understand better with the full picture. > >> > >> > > Sure thing. > > > > > >> > diff --git > >> a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > >> b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > >> > index 810f88689..9256c20b6 100644 > >> > --- a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > >> > +++ b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst > >> > @@ -36,6 +36,14 @@ overview of their function and contents. > >> > when specified allows for the Git binary from the host to be > used > >> > rather than building ``git-native``. > >> > > >> > + :term:`AUTOREV` > >> > + This is a special variable used during fetching. When > >> :term:`SRCREV` is > >> > + set to the value of this variable, it means that the latest > >> revision > >> > + from the version controlled source code repository should be > used. > >> > >> Suggestion: > >> > >> """ > >> ...set to the value of this variable, the latest revision from the > version > >> controlled source code repository is used. > >> """ > >> > >> > + Here is an example:: > >> > >> """ > >> It should be set as follows: > >> """ > >> > >> There's no other way of using AUTOREV I think, so it's not really an > >> example. > >> > >> > + > >> > + SRCREV = "${AUTOREV}" > >> > + > >> > :term:`AZ_SAS` > >> > Azure Storage Shared Access Signature, when using the > >> > :ref:`Azure Storage fetcher > >> <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` > >> > @@ -1622,7 +1630,8 @@ overview of their function and contents. > >> > > >> > - ``name``: Specifies a name to be used for association with > >> > :term:`SRC_URI` checksums or :term:`SRCREV` when you have > more > >> than one > >> > - file or git repository specified in :term:`SRC_URI`. For > >> example:: > >> > + file or source control repository specified in > :term:`SRC_URI`. > >> > + For example:: > >> > > >> > SRC_URI = "git:// > example.com/foo.git;branch=main;name=first > >> \ > >> > git:// > >> example.com/bar.git;branch=main;name=second \ > >> > @@ -1635,7 +1644,8 @@ overview of their function and contents. > >> > - ``subdir``: Places the file (or extracts its contents) into > the > >> > specified subdirectory. This option is useful for unusual > >> tarballs > >> > or other archives that do not have their files already in a > >> > - subdirectory within the archive. > >> > + subdirectory within the archive. This path can be further > >> modified > >> > + by fetcher specific parameters. > >> > >> Ditto: I'm not sure I get what you mean here. Could you give a short > >> example > >> maybe? > >> > >> > > >> > - ``subpath``: Limits the checkout to a specific subpath of > the > >> > tree when using the Git fetcher is used. > >> > @@ -1657,11 +1667,11 @@ overview of their function and contents. > >> > identifier and not just a tag. > >> > > >> > :term:`SRCREV_FORMAT` > >> > - Helps construct valid :term:`SRCREV` values when > >> > + Helps construct a valid package version string when > >> > >> I'm not sure why this changed. This variable does help constructing > >> multiple > >> SRCREV values. Package version strings are just a by-product of using > it, > >> no? > >> > >> > > Perhaps it is again a point of view of a non-native English speaker, but > it > > sounded false to > > my ear. The way it was phrased suggested, that SRCREV_FORMAT is in some > way > > a > > prerequisite or an input to the final value of some SRCREV* variables. If > > fact though, it is used solely as > > a formatter, into which Bitbake will plug the already pre-existing values > > of 'named' > > SRCREV variables, to provide a string which will later be used (outside > of > > BitBake) for PKGV. > > I see that clearly now, understood. > > Thanks, > Antonin > > -- > Antonin Godard, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > >
diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst index f357765b7..1b5fa3ae1 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst @@ -42,7 +42,7 @@ The instantiation of the fetch class is usually followed by:: rootdir = l.getVar('UNPACKDIR') fetcher.unpack(rootdir) -This code unpacks the downloaded files to the specified by ``UNPACKDIR``. +This code unpacks the downloaded files to the specified :term:`UNPACKDIR`. .. note:: @@ -51,7 +51,7 @@ This code unpacks the downloaded files to the specified by ``UNPACKDIR``. examine the OpenEmbedded class file ``base.bbclass`` . -The :term:`SRC_URI` and ``UNPACKDIR`` variables are not hardcoded into the +The :term:`SRC_URI` and :term:`UNPACKDIR` variables are not hardcoded into the fetcher, since those fetcher methods can be (and are) called with different variable names. In OpenEmbedded for example, the shared state (sstate) code uses the fetch module to fetch the sstate files. @@ -159,19 +159,25 @@ URLs except Git URLs, BitBake uses the common ``unpack`` method. A number of parameters exist that you can specify within the URL to govern the behavior of the unpack stage: -- *unpack:* Controls whether the URL components are unpacked. If set to +- *"unpack":* Controls whether the URL components are unpacked. If set to "1", which is the default, the components are unpacked. If set to "0", the unpack stage leaves the file alone. This parameter is useful when you want an archive to be copied in and not be unpacked. -- *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether +- *"dos":* Applies to ``.zip`` and ``.jar`` files and specifies whether to use DOS line ending conversion on text files. -- *striplevel:* Strip specified number of leading components (levels) - from file names on extraction +- *"striplevel":* Strip specified number of leading components (levels) + from file names on extraction. -- *subdir:* Unpacks the specific URL to the specified subdirectory - within the root directory. +- *"subdir":* Unpacks the specific URL to the specified subdirectory + within the specified root directory. This path can be further modified + by fetcher specific parameters. + +- *"name":* Assigns a name to a given component of the :term:`SRC_URI`. + This component is later referenced by this name when specifying its + :term:`SRCREV` or :term:`SRC_URI` checksum, or to correctly place its + revision in the package version string with aid of :term:`SRCREV_FORMAT`. The unpack call automatically decompresses and extracts files with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and @@ -246,21 +252,19 @@ Some example URLs are as follows:: introduce ambiguity when parsing URLs that also contain semi-colons, for example:: - SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" - + SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" Such URLs should should be modified by replacing semi-colons with '&' characters:: - SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" - + SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" In most cases this should work. Treating semi-colons and '&' in queries identically is recommended by the World Wide Web Consortium (W3C). Note that due to the nature of the URL, you may have to specify the name of the downloaded file as well:: - SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" + SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" .. _cvs-fetcher: @@ -398,16 +402,16 @@ This fetcher supports the following parameters: .. note:: - When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs - from the one that is typically passed to ``git clone`` command and provided - by the Git server to fetch from. For example, the URL returned by GitLab - server for ``mesa`` when cloning over SSH is - ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in - :term:`SRC_URI` is the following:: + When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs + from the one that is typically passed to ``git clone`` command and provided + by the Git server to fetch from. For example, the URL returned by GitLab + server for ``mesa`` when cloning over SSH is + ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in + :term:`SRC_URI` is the following:: - SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." + SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." - Note the ``:`` character changed for a ``/`` before the path to the project. + Note the ``:`` character changed for a ``/`` before the path to the project. - *"nocheckout":* Tells the fetcher to not checkout source code when unpacking when set to "1". Set this option for the URL where there is @@ -453,7 +457,7 @@ This fetcher supports the following parameters: By default, the path is ``git/``. - *"usehead":* Enables local ``git://`` URLs to use the current branch - HEAD as the revision for use with ``AUTOREV``. The "usehead" + HEAD as the revision for use with :term:`AUTOREV`. The "usehead" parameter implies no branch and only works when the transfer protocol is ``file://``. @@ -525,10 +529,10 @@ The fetcher uses the ``rcleartool`` or Following are options for the :term:`SRC_URI` statement: -- *vob*: The name, which must include the prepending "/" character, +- *"vob":*: The name, which must include the prepending "/" character, of the ClearCase VOB. This option is required. -- *module*: The module, which must include the prepending "/" +- *"module":*: The module, which must include the prepending "/" character, in the selected VOB. .. note:: @@ -540,7 +544,7 @@ Following are options for the :term:`SRC_URI` statement: load /example_vob/example_module -- *proto*: The protocol, which can be either ``http`` or ``https``. +- *"proto"*: The protocol, which can be either ``http`` or ``https``. By default, the fetcher creates a configuration specification. If you want this specification written to an area other than the default, use @@ -549,9 +553,9 @@ the specification is written. .. note:: - the SRCREV loses its functionality if you specify this variable. However, - SRCREV is still used to label the archive after a fetch even though it does - not define what is fetched. + the :term:`SRCREV` loses its functionality if you specify this variable. + However, :term:`SRCREV` is still used to label the archive after a fetch even + though it does not define what is fetched. Here are a couple of other behaviors worth mentioning: @@ -606,21 +610,23 @@ password, and fetches a Revision based on a Label:: .. note:: - You should always set S to "${UNPACKDIR}/p4" in your recipe. + Your recipe should always set S as shown in the above example:: + + S = "${UNPACKDIR}/p4" By default, the fetcher strips the depot location from the local file paths. In the above example, the content of ``example-depot/main/source/`` will be placed in ``${UNPACKDIR}/p4``. For situations where preserving parts of the remote depot paths locally is desirable, the fetcher supports two parameters: -- *"module":* - The top-level depot location or directory to fetch. The value of this - parameter can also point to a single file within the depot, in which case - the local file path will include the module path. -- *"remotepath":* - When used with the value "``keep``", the fetcher will mirror the full depot - paths locally for the specified location, even in combination with the - ``module`` parameter. +- *"module":* + The top-level depot location or directory to fetch. The value of this + parameter can also point to a single file within the depot, in which case + the local file path will include the module path. +- *"remotepath":* + When used with the value "``keep``", the fetcher will mirror the full depot + paths locally for the specified location, even in combination with the + ``module`` parameter. Here is an example use of the the ``module`` parameter:: @@ -843,4 +849,37 @@ submodules. However, you might find the code helpful and readable. Auto Revisions ============== -We need to document ``AUTOREV`` and :term:`SRCREV_FORMAT` here. +For recipes which need to use the latest revision of their source code, +the way to achieve it is to use :term:`AUTOREV` as the value of the +source code repository's :term:`SRCREV`:: + + SRCREV = "${AUTOREV}" + +In this case, to properly show the revision information in the package's version +string, :term:`PV` should be set to include the ``+`` sign. + +.. note:: + + With :term:`AUTOREV`, BitBake will always need to take the additional step of + querying the remore repository to retrieve the latest available revision. + + Also, recipes using it don't benefit from the caching mechanism. + +Multiple Source Control Repositories +==================================== + +For some recipes it is necessary to make use of more than one +version controlled source code repository. It such case, the recipe +must provide BitBake with information about how it should include +all those SCM versions in its package version string, instead of its +usual approach with a single :term:`SRCREV`. + +For this purpose, the recipe must set the :term:`SRCREV_FORMAT` +variable. One notable example is the OpenEmbedded ``kernel-yocto.bbclass``:: + + SRCREV_FORMAT ?= "meta_machine" + +The value given to :term:`SRCREV_FORMAT` references names, which were +assigned using the ``name`` parameter to those components of :term:`SRC_URI`, +which represent the version controlled source code repositories. In the above +example, the :term:`SRC_URI` contained two URLs named "meta" and "machine". diff --git a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst index 810f88689..9256c20b6 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst +++ b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst @@ -36,6 +36,14 @@ overview of their function and contents. when specified allows for the Git binary from the host to be used rather than building ``git-native``. + :term:`AUTOREV` + This is a special variable used during fetching. When :term:`SRCREV` is + set to the value of this variable, it means that the latest revision + from the version controlled source code repository should be used. + Here is an example:: + + SRCREV = "${AUTOREV}" + :term:`AZ_SAS` Azure Storage Shared Access Signature, when using the :ref:`Azure Storage fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` @@ -1622,7 +1630,8 @@ overview of their function and contents. - ``name``: Specifies a name to be used for association with :term:`SRC_URI` checksums or :term:`SRCREV` when you have more than one - file or git repository specified in :term:`SRC_URI`. For example:: + file or source control repository specified in :term:`SRC_URI`. + For example:: SRC_URI = "git://example.com/foo.git;branch=main;name=first \ git://example.com/bar.git;branch=main;name=second \ @@ -1635,7 +1644,8 @@ overview of their function and contents. - ``subdir``: Places the file (or extracts its contents) into the specified subdirectory. This option is useful for unusual tarballs or other archives that do not have their files already in a - subdirectory within the archive. + subdirectory within the archive. This path can be further modified + by fetcher specific parameters. - ``subpath``: Limits the checkout to a specific subpath of the tree when using the Git fetcher is used. @@ -1657,11 +1667,11 @@ overview of their function and contents. identifier and not just a tag. :term:`SRCREV_FORMAT` - Helps construct valid :term:`SRCREV` values when + Helps construct a valid package version string when multiple source controlled URLs are used in :term:`SRC_URI`. - The system needs help constructing these values under these + The system needs help constructing this value under these circumstances. Each component in the :term:`SRC_URI` is assigned a name and these are referenced in the :term:`SRCREV_FORMAT` variable. Consider an example with URLs named "machine" and "meta". In this case, @@ -1697,3 +1707,7 @@ overview of their function and contents. :term:`TOPDIR` Points to the build directory. BitBake automatically sets this variable. + + :term:`UNPACKDIR` + The directory into which recipe's downloads are unpacked during its + build process.
Describe the roles of AUTOREV and SRCREV_FORMAT in the fetching step. Add their missing descriptions to the variable reference section. Correct encountered style issues. Fixes [YOCTO #14498] Signed-off-by: Adam Blank <adam.blank.g@gmail.com> --- .../bitbake-user-manual-fetching.rst | 115 ++++++++++++------ .../bitbake-user-manual-ref-variables.rst | 22 +++- 2 files changed, 95 insertions(+), 42 deletions(-)