diff mbox series

[1/2] poky.yaml: introduce DISTRO_LATEST_TAG

Message ID 20250409-fix-distro-dead-links-v1-1-616b62185d04@bootlin.com
State Under Review
Headers show
Series Fix broken links when building on branch tip | expand

Commit Message

Antonin Godard April 9, 2025, 9:55 a.m. UTC
Introduce the DISTRO_LATEST_TAG macro, which should always point to the
latest existing tag in the documentation, unlike DISTRO which may point
to A.B.999 to represent the tip of a branch.

This variable is needed to fix dead links in the documentation that
currently use the DISTRO macro.

Also, make DISTRO_REL_TAG use the DISTRO macro directly, to avoid
repetition, and add a DISTRO_REL_LATEST_TAG macro that has the same role
as DISTRO_LATEST_TAG but with "yocto-" prepended to it.

In set_versions.py, run the "git describe --abbrev=0 --tags
--match='yocto-*'" command to get the latest existing tag on the
currently checked out commit. Fallback to ourversion in case we didn't
find any.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
---
 documentation/poky.yaml.in    | 11 ++++++++++-
 documentation/set_versions.py | 16 +++++++++++++++-
 2 files changed, 25 insertions(+), 2 deletions(-)

Comments

Quentin Schulz April 14, 2025, 12:18 p.m. UTC | #1
Hi Antonin,

On 4/9/25 11:55 AM, Antonin Godard via lists.yoctoproject.org wrote:
> Introduce the DISTRO_LATEST_TAG macro, which should always point to the
> latest existing tag in the documentation, unlike DISTRO which may point
> to A.B.999 to represent the tip of a branch.
> 
> This variable is needed to fix dead links in the documentation that
> currently use the DISTRO macro.
> 
> Also, make DISTRO_REL_TAG use the DISTRO macro directly, to avoid
> repetition, and add a DISTRO_REL_LATEST_TAG macro that has the same role
> as DISTRO_LATEST_TAG but with "yocto-" prepended to it.
> 
> In set_versions.py, run the "git describe --abbrev=0 --tags
> --match='yocto-*'" command to get the latest existing tag on the
> currently checked out commit. Fallback to ourversion in case we didn't
> find any.
> 
> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
> ---
>   documentation/poky.yaml.in    | 11 ++++++++++-
>   documentation/set_versions.py | 16 +++++++++++++++-
>   2 files changed, 25 insertions(+), 2 deletions(-)
> 
> diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
> index 836f11454..26c21e346 100644
> --- a/documentation/poky.yaml.in
> +++ b/documentation/poky.yaml.in
> @@ -2,13 +2,22 @@
>   # Macros used in the documentation
>   #
>   
> +# The DISTRO variable represents the current docs version. It should be used
> +# when referring to the current docs version. See also DISTRO_LATEST_TAG.
>   DISTRO : "5.1"
> +# The DISTRO_LATEST_TAG represents the latest tag on the current branch. It
> +# should be used in HTTP link referring to the current docs version. In these
> +# cases, the DISTRO may point to A.B.999 which does not exist (just used to
> +# represent the latest HEAD revision on the branch). DISTRO_LATEST_TAG should
> +# always point to an existing tag.

I don't remember why/where we need A.B.999? I assume most shouldn't 
point at .999, ever?

Also, it would be nice to provide an example as to what is supposed to 
be stored in each variable based on a specific example.

e.g.

"""
# If building from master, e.g. commit 
3b50193fa0c9acf4a601aeae6e1c78d0e4a05aef
# DISTRO will be 5.2.999
# DISTRO_LATEST_TAG will be 5.1
#
# If building from a release branch, e.g. styhead (e.g. commit 
85e738e4c0e62f69699fff4bb0482ee3e3121496)
# DISTRO will be 5.1.999
# DISTRO_LATEST_TAG will be 5.1.4
#
# If building from a tag, e.g. yocto-5.1.4
# DISTRO will be 5.1.4
# DISTRO_LATEST_TAG will be 5.1.4
"""

Note that I haven't thoroughly checked the above is correct.

If that is correct, do we really want to return 5.1 for LATEST_TAG if 
building from master?

If you always want to return the latest tag for a release, it's a bit 
more involved. For example, I did the following for finding out the 
latest tag in TF-A, including lts branches:

"""
# Use most recent release, the latest annotated tag reachable from master
LAST_V_TAG=$(git describe --abbrev=0 --match "v?*.?*.?*")
# Find all tags from different branches that contain that latest tag 
reachable from master.
# This will return lts tags, of which we want to take the latest available.
# If no LTS tag, take the latest non-rc tag reachable from master.
LAST_LTS_TAG=$(git tag --sort -version:refname --contains "$LAST_V_TAG" 
'lts-v?*.?*.?*' | head -1)
TAG=${LAST_LTS_TAG:-$LAST_V_TAG}
"""

I assume we could something similar here.

> +DISTRO_LATEST_TAG : "5.1"
>   DISTRO_NAME_NO_CAP : "styhead"
>   DISTRO_NAME : "Styhead"
>   DISTRO_NAME_NO_CAP_MINUS_ONE : "scarthgap"
>   DISTRO_NAME_NO_CAP_LTS : "scarthgap"
>   YOCTO_DOC_VERSION : "5.1"
> -DISTRO_REL_TAG : "yocto-5.1"
> +DISTRO_REL_TAG : "yocto-$DISTRO;"

Separate patch for doing that (with the removal of it in the 
replacements dict in the python script).

> +DISTRO_REL_LATEST_TAG : "yocto-&DISTRO_LATEST_TAG;">   DOCCONF_VERSION : "dev"
>   BITBAKE_SERIES : ""
>   YOCTO_DL_URL : "https://downloads.yoctoproject.org/"
> diff --git a/documentation/set_versions.py b/documentation/set_versions.py
> index 5c55f470d..b94a7daad 100755
> --- a/documentation/set_versions.py
> +++ b/documentation/set_versions.py
> @@ -170,17 +170,29 @@ series = [k for k in release_series]
>   previousseries = series[series.index(ourseries)+1:] or [""]
>   lastlts = [k for k in previousseries if k in ltsseries] or "dunfell"
>   
> +latestreltag = subprocess.run(["git", "describe", "--abbrev=0", "--tags", "--match", "yocto-*"], capture_output=True, text=True).stdout
> +latestreltag = latestreltag.strip()
> +if latestreltag:
> +    if latestreltag.startswith("yocto-"):

This is guaranteed, because of how git describe --match works, no?

"""
        --match <pattern>
            Only consider tags matching the given glob(7) pattern
"""

So we can avoid the check.

> +        latesttag = latestreltag[6:]

You could use len("yocto-") instead of 6 to be more explicit then.

> +else:
> +    # fallback on the calculated version
> +    print("Did not find a tag with 'git describe', falling back to %s" % ourversion)
> +    latestreltag = "yocto-" + ourversion
> +    latesttag = ourversion
> +
>   print("Version calculated to be %s" % ourversion)
> +print("Latest release tag found is %s" % latestreltag)
>   print("Release series calculated to be %s" % ourseries)
>   
>   replacements = {
>       "DISTRO" : ourversion,
> +    "DISTRO_LATEST_TAG": latesttag,
>       "DISTRO_NAME_NO_CAP" : ourseries,
>       "DISTRO_NAME" : ourseries.capitalize(),
>       "DISTRO_NAME_NO_CAP_MINUS_ONE" : previousseries[0],
>       "DISTRO_NAME_NO_CAP_LTS" : lastlts[0],
>       "YOCTO_DOC_VERSION" : ourversion,
> -    "DISTRO_REL_TAG" : "yocto-" + ourversion,
>       "DOCCONF_VERSION" : docconfver,
>       "BITBAKE_SERIES" : bitbakeversion,
>   }
> @@ -318,3 +330,5 @@ with open('releases.rst', 'w') as f:
>               if tag == release_series[series] or tag.startswith('%s.' % release_series[series]):
>                   f.write('- :yocto_docs:`%s Documentation </%s>`\n' % (tag, tag))
>           f.write('\n')
> +
> +
> 

Not sure we need those two additional new lines?

Cheers,
Quentin
Antonin Godard April 16, 2025, 7:46 a.m. UTC | #2
Hi Quentin,

On Mon Apr 14, 2025 at 2:18 PM CEST, Quentin Schulz wrote:
> Hi Antonin,
>
> On 4/9/25 11:55 AM, Antonin Godard via lists.yoctoproject.org wrote:
>> Introduce the DISTRO_LATEST_TAG macro, which should always point to the
>> latest existing tag in the documentation, unlike DISTRO which may point
>> to A.B.999 to represent the tip of a branch.
>> 
>> This variable is needed to fix dead links in the documentation that
>> currently use the DISTRO macro.
>> 
>> Also, make DISTRO_REL_TAG use the DISTRO macro directly, to avoid
>> repetition, and add a DISTRO_REL_LATEST_TAG macro that has the same role
>> as DISTRO_LATEST_TAG but with "yocto-" prepended to it.
>> 
>> In set_versions.py, run the "git describe --abbrev=0 --tags
>> --match='yocto-*'" command to get the latest existing tag on the
>> currently checked out commit. Fallback to ourversion in case we didn't
>> find any.
>> 
>> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
>> ---
>>   documentation/poky.yaml.in    | 11 ++++++++++-
>>   documentation/set_versions.py | 16 +++++++++++++++-
>>   2 files changed, 25 insertions(+), 2 deletions(-)
>> 
>> diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
>> index 836f11454..26c21e346 100644
>> --- a/documentation/poky.yaml.in
>> +++ b/documentation/poky.yaml.in
>> @@ -2,13 +2,22 @@
>>   # Macros used in the documentation
>>   #
>>   
>> +# The DISTRO variable represents the current docs version. It should be used
>> +# when referring to the current docs version. See also DISTRO_LATEST_TAG.
>>   DISTRO : "5.1"
>> +# The DISTRO_LATEST_TAG represents the latest tag on the current branch. It
>> +# should be used in HTTP link referring to the current docs version. In these
>> +# cases, the DISTRO may point to A.B.999 which does not exist (just used to
>> +# represent the latest HEAD revision on the branch). DISTRO_LATEST_TAG should
>> +# always point to an existing tag.
>
> I don't remember why/where we need A.B.999? I assume most shouldn't 
> point at .999, ever?

The default landing page of docs.yoctoproject.org is a 999 version (tip of the
stable release branch).

> Also, it would be nice to provide an example as to what is supposed to
> be stored in each variable based on a specific example.
>
> e.g.
>
> """
> # If building from master, e.g. commit 
> 3b50193fa0c9acf4a601aeae6e1c78d0e4a05aef
> # DISTRO will be 5.2.999
> # DISTRO_LATEST_TAG will be 5.1
> #
> # If building from a release branch, e.g. styhead (e.g. commit 
> 85e738e4c0e62f69699fff4bb0482ee3e3121496)
> # DISTRO will be 5.1.999
> # DISTRO_LATEST_TAG will be 5.1.4
> #
> # If building from a tag, e.g. yocto-5.1.4
> # DISTRO will be 5.1.4
> # DISTRO_LATEST_TAG will be 5.1.4
> """
>
> Note that I haven't thoroughly checked the above is correct.

That's it. Although, as you see from set_versions.py, these variables are not to
be set manually in poky.yaml.in, and are automatically set by the script.
I could add a comment on that in poky.yaml.in.

> If that is correct, do we really want to return 5.1 for LATEST_TAG if 
> building from master?

DISTRO_LATEST_TAG is used in URLs and in most cases point to
downloads.yoctoproject.org. Now I'm not sure for master we want to point to a
dead link, or the latest tag across all possible branches, or just the previous
tag on the master branch. For simplicity I'd favor the last option, especially
because master is a dev branch so we don't expect any sort of stable information
on there.

> If you always want to return the latest tag for a release, it's a bit 
> more involved. For example, I did the following for finding out the 
> latest tag in TF-A, including lts branches:

DISTRO_LATEST_TAG only represents the current branch, and I don't think it's
worth trying to find the latest tag accross all branches, considering the master
branch is not a release branch (development branch) and so we don't expect URLs
to point to a valid link anyway.

> """
> # Use most recent release, the latest annotated tag reachable from master
> LAST_V_TAG=$(git describe --abbrev=0 --match "v?*.?*.?*")
> # Find all tags from different branches that contain that latest tag 
> reachable from master.
> # This will return lts tags, of which we want to take the latest available.
> # If no LTS tag, take the latest non-rc tag reachable from master.
> LAST_LTS_TAG=$(git tag --sort -version:refname --contains "$LAST_V_TAG" 
> 'lts-v?*.?*.?*' | head -1)
> TAG=${LAST_LTS_TAG:-$LAST_V_TAG}
> """
>
> I assume we could something similar here.
>
>> +DISTRO_LATEST_TAG : "5.1"
>>   DISTRO_NAME_NO_CAP : "styhead"
>>   DISTRO_NAME : "Styhead"
>>   DISTRO_NAME_NO_CAP_MINUS_ONE : "scarthgap"
>>   DISTRO_NAME_NO_CAP_LTS : "scarthgap"
>>   YOCTO_DOC_VERSION : "5.1"
>> -DISTRO_REL_TAG : "yocto-5.1"
>> +DISTRO_REL_TAG : "yocto-$DISTRO;"
>
> Separate patch for doing that (with the removal of it in the 
> replacements dict in the python script).
>
>> +DISTRO_REL_LATEST_TAG : "yocto-&DISTRO_LATEST_TAG;">   DOCCONF_VERSION : "dev"
>>   BITBAKE_SERIES : ""
>>   YOCTO_DL_URL : "https://downloads.yoctoproject.org/"
>> diff --git a/documentation/set_versions.py b/documentation/set_versions.py
>> index 5c55f470d..b94a7daad 100755
>> --- a/documentation/set_versions.py
>> +++ b/documentation/set_versions.py
>> @@ -170,17 +170,29 @@ series = [k for k in release_series]
>>   previousseries = series[series.index(ourseries)+1:] or [""]
>>   lastlts = [k for k in previousseries if k in ltsseries] or "dunfell"
>>   
>> +latestreltag = subprocess.run(["git", "describe", "--abbrev=0", "--tags", "--match", "yocto-*"], capture_output=True, text=True).stdout
>> +latestreltag = latestreltag.strip()
>> +if latestreltag:
>> +    if latestreltag.startswith("yocto-"):
>
> This is guaranteed, because of how git describe --match works, no?
>
> """
>         --match <pattern>
>             Only consider tags matching the given glob(7) pattern
> """
>
> So we can avoid the check.

True, I think I just felt off not checking. :)

>> +        latesttag = latestreltag[6:]
>
> You could use len("yocto-") instead of 6 to be more explicit then.
>
>> +else:
>> +    # fallback on the calculated version
>> +    print("Did not find a tag with 'git describe', falling back to %s" % ourversion)
>> +    latestreltag = "yocto-" + ourversion
>> +    latesttag = ourversion
>> +
>>   print("Version calculated to be %s" % ourversion)
>> +print("Latest release tag found is %s" % latestreltag)
>>   print("Release series calculated to be %s" % ourseries)
>>   
>>   replacements = {
>>       "DISTRO" : ourversion,
>> +    "DISTRO_LATEST_TAG": latesttag,
>>       "DISTRO_NAME_NO_CAP" : ourseries,
>>       "DISTRO_NAME" : ourseries.capitalize(),
>>       "DISTRO_NAME_NO_CAP_MINUS_ONE" : previousseries[0],
>>       "DISTRO_NAME_NO_CAP_LTS" : lastlts[0],
>>       "YOCTO_DOC_VERSION" : ourversion,
>> -    "DISTRO_REL_TAG" : "yocto-" + ourversion,
>>       "DOCCONF_VERSION" : docconfver,
>>       "BITBAKE_SERIES" : bitbakeversion,
>>   }
>> @@ -318,3 +330,5 @@ with open('releases.rst', 'w') as f:
>>               if tag == release_series[series] or tag.startswith('%s.' % release_series[series]):
>>                   f.write('- :yocto_docs:`%s Documentation </%s>`\n' % (tag, tag))
>>           f.write('\n')
>> +
>> +
>> 
>
> Not sure we need those two additional new lines?

My bad, a mistake that slipped in.

Thanks,
Antonin
Quentin Schulz April 16, 2025, 9:58 a.m. UTC | #3
Hi Antonin,

On 4/16/25 9:46 AM, Antonin Godard wrote:
> Hi Quentin,
> 
> On Mon Apr 14, 2025 at 2:18 PM CEST, Quentin Schulz wrote:
>> Hi Antonin,
>>
>> On 4/9/25 11:55 AM, Antonin Godard via lists.yoctoproject.org wrote:
>>> Introduce the DISTRO_LATEST_TAG macro, which should always point to the
>>> latest existing tag in the documentation, unlike DISTRO which may point
>>> to A.B.999 to represent the tip of a branch.
>>>
>>> This variable is needed to fix dead links in the documentation that
>>> currently use the DISTRO macro.
>>>
>>> Also, make DISTRO_REL_TAG use the DISTRO macro directly, to avoid
>>> repetition, and add a DISTRO_REL_LATEST_TAG macro that has the same role
>>> as DISTRO_LATEST_TAG but with "yocto-" prepended to it.
>>>
>>> In set_versions.py, run the "git describe --abbrev=0 --tags
>>> --match='yocto-*'" command to get the latest existing tag on the
>>> currently checked out commit. Fallback to ourversion in case we didn't
>>> find any.
>>>
>>> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
>>> ---
>>>    documentation/poky.yaml.in    | 11 ++++++++++-
>>>    documentation/set_versions.py | 16 +++++++++++++++-
>>>    2 files changed, 25 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
>>> index 836f11454..26c21e346 100644
>>> --- a/documentation/poky.yaml.in
>>> +++ b/documentation/poky.yaml.in
>>> @@ -2,13 +2,22 @@
>>>    # Macros used in the documentation
>>>    #
>>>    
>>> +# The DISTRO variable represents the current docs version. It should be used
>>> +# when referring to the current docs version. See also DISTRO_LATEST_TAG.
>>>    DISTRO : "5.1"
>>> +# The DISTRO_LATEST_TAG represents the latest tag on the current branch. It
>>> +# should be used in HTTP link referring to the current docs version. In these
>>> +# cases, the DISTRO may point to A.B.999 which does not exist (just used to
>>> +# represent the latest HEAD revision on the branch). DISTRO_LATEST_TAG should
>>> +# always point to an existing tag.
>>
>> I don't remember why/where we need A.B.999? I assume most shouldn't
>> point at .999, ever?
> 
> The default landing page of docs.yoctoproject.org is a 999 version (tip of the
> stable release branch).
> 

That essentially should be for the version number in text and in the JS 
dropdown? We should really check our use of &DISTRO; in the docs.

For example:

documentation/contributor-guide/report-defect.rst, not sure we don't 
want to use &DISTRO_LATEST_TAG; (or another variable) instead, as it'll 
always be wrong on the default landing page of the docs (always x.x.999)?

documentation/dev-manual/multiconfig.rst, I assume we also do not want 
to have x.x.999 there? I'm also not entirely sure why we explicitly list 
the version or if it's still appropriate?

documentation/dev-manual/start.rst:774 we should probably get rid of 
&DISTRO; (but we can keep &DISTRO_NAME; as that matches the code block 
under?)

documentation/dev-manual/start.rst:843 is plain wrong as the tag does 
not exist so we should be using DISTRO_LATEST_TAG. Same for
documentation/dev-manual/start.rst:854, same for 
documentation/overview-manual/development-environment.rst:459.

documentation/ref-manual/release-process.rst:15 should probably use 
DISTRO_NAME or DISTRO_LATEST_TAG? (or probably another variable which is 
only X.Y and not X.Y.Z if there's one, since X.Y.Z is a minor release).

documentation/ref-manual/release-process.rst:59 probably should be using 
DISTRO_LATEST_TAG too?

documentation/ref-manual/system-requirements.rst:58, could be using 
DISTRO_LATEST_TAG but probably not a big deal if we keep DISTRO instead?

documentation/ref-manual/system-requirements.rst:349, I woudl assume is 
completely wrong and should be DISTRO_LATEST_TAG instead? Same for 
documentation/ref-manual/system-requirements.rst:350?

documentation/ref-manual/system-requirements.rst:403,407,411,474,478,482 
need to use DISTRO_LATEST_TAG to match the name of the buildtools.

documentation/sdk-manual/appendix-obtain.rst:66,72,176 needs to match 
the toolchain you can download, so needs to be DISTRO_LATEST_TAG.

documentation/sdk-manual/appendix-obtain.rst:275 probably wants 
DISTRO_LATEST_TAG too.

documentation/sdk-manual/extensible.rst:120,124,126 probably wants 
DISTRO_LATEST_TAG too.

documentation/sdk-manual/using.rst:74,78,80,101,102,104,105,110,130,141, 
probably wants DISTRO_LATEST_TAG too.

documentation/sdk-manual/working-projects.rst:85,88,219,223,225,280 
probably wants DISTRO_LATEST_TAG too.

documentation/toaster-manual/reference.rst:161,295 DISTRO_LATEST_TAG

Nothing uses YOCTO_RELEASE_DL_URL, maybe we can remove it entirely?

>> Also, it would be nice to provide an example as to what is supposed to
>> be stored in each variable based on a specific example.
>>
>> e.g.
>>
>> """
>> # If building from master, e.g. commit
>> 3b50193fa0c9acf4a601aeae6e1c78d0e4a05aef
>> # DISTRO will be 5.2.999
>> # DISTRO_LATEST_TAG will be 5.1
>> #
>> # If building from a release branch, e.g. styhead (e.g. commit
>> 85e738e4c0e62f69699fff4bb0482ee3e3121496)
>> # DISTRO will be 5.1.999
>> # DISTRO_LATEST_TAG will be 5.1.4
>> #
>> # If building from a tag, e.g. yocto-5.1.4
>> # DISTRO will be 5.1.4
>> # DISTRO_LATEST_TAG will be 5.1.4
>> """
>>
>> Note that I haven't thoroughly checked the above is correct.
> 
> That's it. Although, as you see from set_versions.py, these variables are not to
> be set manually in poky.yaml.in, and are automatically set by the script.
> I could add a comment on that in poky.yaml.in.
> 

It's actually been bothering me a lot that we have some parts of 
poky.yaml.in that are taken verbatim and some modified before making it 
to poky.yaml.

I seem to recall we wanted the variables set by set_versions.py to still 
be able to build the docs from tarballs and not force the use of git 
repos? But it seems like we force to fetch git tags at the very 
beginning of the Python script.

So I'm wondering if we should only have variables that aren't replaced 
in poky.yaml.in and eventually have the ones set_versions.py adds in 
poky.yaml (instead of replacing for example) documented in comments to 
explain what they are and in which context(s) to use them?

>> If that is correct, do we really want to return 5.1 for LATEST_TAG if
>> building from master?
> 
> DISTRO_LATEST_TAG is used in URLs and in most cases point to
> downloads.yoctoproject.org. Now I'm not sure for master we want to point to a
> dead link, or the latest tag across all possible branches, or just the previous
> tag on the master branch. For simplicity I'd favor the last option, especially
> because master is a dev branch so we don't expect any sort of stable information
> on there.
> 

And the dev/ branch isn't the one accessed by default when going to 
docs.yoctoproject.org/ so I think we can afford not linking to the 
latest tag in the latest release branch and instead the latest tag in 
the master branch (i.e. the last release's first tag).

[...]

Cheers,
Quentin
Antonin Godard April 17, 2025, 7:21 a.m. UTC | #4
Hi Quentin,

On Wed Apr 16, 2025 at 11:58 AM CEST, Quentin Schulz wrote:
> Hi Antonin,
>
> On 4/16/25 9:46 AM, Antonin Godard wrote:
>> Hi Quentin,
>> 
>> On Mon Apr 14, 2025 at 2:18 PM CEST, Quentin Schulz wrote:
>>> Hi Antonin,
>>>
>>> On 4/9/25 11:55 AM, Antonin Godard via lists.yoctoproject.org wrote:
>>>> Introduce the DISTRO_LATEST_TAG macro, which should always point to the
>>>> latest existing tag in the documentation, unlike DISTRO which may point
>>>> to A.B.999 to represent the tip of a branch.
>>>>
>>>> This variable is needed to fix dead links in the documentation that
>>>> currently use the DISTRO macro.
>>>>
>>>> Also, make DISTRO_REL_TAG use the DISTRO macro directly, to avoid
>>>> repetition, and add a DISTRO_REL_LATEST_TAG macro that has the same role
>>>> as DISTRO_LATEST_TAG but with "yocto-" prepended to it.
>>>>
>>>> In set_versions.py, run the "git describe --abbrev=0 --tags
>>>> --match='yocto-*'" command to get the latest existing tag on the
>>>> currently checked out commit. Fallback to ourversion in case we didn't
>>>> find any.
>>>>
>>>> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
>>>> ---
>>>>    documentation/poky.yaml.in    | 11 ++++++++++-
>>>>    documentation/set_versions.py | 16 +++++++++++++++-
>>>>    2 files changed, 25 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
>>>> index 836f11454..26c21e346 100644
>>>> --- a/documentation/poky.yaml.in
>>>> +++ b/documentation/poky.yaml.in
>>>> @@ -2,13 +2,22 @@
>>>>    # Macros used in the documentation
>>>>    #
>>>>    
>>>> +# The DISTRO variable represents the current docs version. It should be used
>>>> +# when referring to the current docs version. See also DISTRO_LATEST_TAG.
>>>>    DISTRO : "5.1"
>>>> +# The DISTRO_LATEST_TAG represents the latest tag on the current branch. It
>>>> +# should be used in HTTP link referring to the current docs version. In these
>>>> +# cases, the DISTRO may point to A.B.999 which does not exist (just used to
>>>> +# represent the latest HEAD revision on the branch). DISTRO_LATEST_TAG should
>>>> +# always point to an existing tag.
>>>
>>> I don't remember why/where we need A.B.999? I assume most shouldn't
>>> point at .999, ever?
>> 
>> The default landing page of docs.yoctoproject.org is a 999 version (tip of the
>> stable release branch).
>> 
>
> That essentially should be for the version number in text and in the JS 
> dropdown? We should really check our use of &DISTRO; in the docs.
>
> For example:
>
> documentation/contributor-guide/report-defect.rst, not sure we don't 
> want to use &DISTRO_LATEST_TAG; (or another variable) instead, as it'll 
> always be wrong on the default landing page of the docs (always x.x.999)?
>
> documentation/dev-manual/multiconfig.rst, I assume we also do not want 
> to have x.x.999 there? I'm also not entirely sure why we explicitly list 
> the version or if it's still appropriate?
>
> documentation/dev-manual/start.rst:774 we should probably get rid of 
> &DISTRO; (but we can keep &DISTRO_NAME; as that matches the code block 
> under?)
>
> documentation/dev-manual/start.rst:843 is plain wrong as the tag does 
> not exist so we should be using DISTRO_LATEST_TAG. Same for
> documentation/dev-manual/start.rst:854, same for 
> documentation/overview-manual/development-environment.rst:459.
>
> documentation/ref-manual/release-process.rst:15 should probably use 
> DISTRO_NAME or DISTRO_LATEST_TAG? (or probably another variable which is 
> only X.Y and not X.Y.Z if there's one, since X.Y.Z is a minor release).
>
> documentation/ref-manual/release-process.rst:59 probably should be using 
> DISTRO_LATEST_TAG too?
>
> documentation/ref-manual/system-requirements.rst:58, could be using 
> DISTRO_LATEST_TAG but probably not a big deal if we keep DISTRO instead?
>
> documentation/ref-manual/system-requirements.rst:349, I woudl assume is 
> completely wrong and should be DISTRO_LATEST_TAG instead? Same for 
> documentation/ref-manual/system-requirements.rst:350?
>
> documentation/ref-manual/system-requirements.rst:403,407,411,474,478,482 
> need to use DISTRO_LATEST_TAG to match the name of the buildtools.
>
> documentation/sdk-manual/appendix-obtain.rst:66,72,176 needs to match 
> the toolchain you can download, so needs to be DISTRO_LATEST_TAG.
>
> documentation/sdk-manual/appendix-obtain.rst:275 probably wants 
> DISTRO_LATEST_TAG too.
>
> documentation/sdk-manual/extensible.rst:120,124,126 probably wants 
> DISTRO_LATEST_TAG too.
>
> documentation/sdk-manual/using.rst:74,78,80,101,102,104,105,110,130,141, 
> probably wants DISTRO_LATEST_TAG too.
>
> documentation/sdk-manual/working-projects.rst:85,88,219,223,225,280 
> probably wants DISTRO_LATEST_TAG too.
>
> documentation/toaster-manual/reference.rst:161,295 DISTRO_LATEST_TAG

Thanks for going through the different usages of DISTRO. You're right, with
DISTRO_LATEST_TAG now I also think it would make more sense to use it rather
than DISTRO. I will try to work on that a bit later (or feel free to send a
patch :), the initial focus for this series was to fix dead links .

> Nothing uses YOCTO_RELEASE_DL_URL, maybe we can remove it entirely?

Indeed we can remove this variable.

>>> Also, it would be nice to provide an example as to what is supposed to
>>> be stored in each variable based on a specific example.
>>>
>>> e.g.
>>>
>>> """
>>> # If building from master, e.g. commit
>>> 3b50193fa0c9acf4a601aeae6e1c78d0e4a05aef
>>> # DISTRO will be 5.2.999
>>> # DISTRO_LATEST_TAG will be 5.1
>>> #
>>> # If building from a release branch, e.g. styhead (e.g. commit
>>> 85e738e4c0e62f69699fff4bb0482ee3e3121496)
>>> # DISTRO will be 5.1.999
>>> # DISTRO_LATEST_TAG will be 5.1.4
>>> #
>>> # If building from a tag, e.g. yocto-5.1.4
>>> # DISTRO will be 5.1.4
>>> # DISTRO_LATEST_TAG will be 5.1.4
>>> """
>>>
>>> Note that I haven't thoroughly checked the above is correct.
>> 
>> That's it. Although, as you see from set_versions.py, these variables are not to
>> be set manually in poky.yaml.in, and are automatically set by the script.
>> I could add a comment on that in poky.yaml.in.
>> 
>
> It's actually been bothering me a lot that we have some parts of 
> poky.yaml.in that are taken verbatim and some modified before making it 
> to poky.yaml.
>
> I seem to recall we wanted the variables set by set_versions.py to still 
> be able to build the docs from tarballs and not force the use of git 
> repos? But it seems like we force to fetch git tags at the very 
> beginning of the Python script.

  $ make html
  ./set_versions.py
  Please run 'git fetch --tags' before building the documentation
  make: *** [Makefile:72: html] Error 1

It is indeed a bug!
Created one here: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15834

> So I'm wondering if we should only have variables that aren't replaced 
> in poky.yaml.in and eventually have the ones set_versions.py adds in 
> poky.yaml (instead of replacing for example) documented in comments to 
> explain what they are and in which context(s) to use them?

That would be a bit cleaner yes.

Maybe standards.md would be a good place? i.e.:
"Use &DISTRO; in the docs when wanting to refer to the latest possible version"
"Use &DISTRO_LATEST_TAG; in the docs when wanting to refer to the latest
existing tag"
etc.

>>> If that is correct, do we really want to return 5.1 for LATEST_TAG if
>>> building from master?
>> 
>> DISTRO_LATEST_TAG is used in URLs and in most cases point to
>> downloads.yoctoproject.org. Now I'm not sure for master we want to point to a
>> dead link, or the latest tag across all possible branches, or just the previous
>> tag on the master branch. For simplicity I'd favor the last option, especially
>> because master is a dev branch so we don't expect any sort of stable information
>> on there.
>> 
>
> And the dev/ branch isn't the one accessed by default when going to 
> docs.yoctoproject.org/ so I think we can afford not linking to the 
> latest tag in the latest release branch and instead the latest tag in 
> the master branch (i.e. the last release's first tag).
>
> [...]
>
> Cheers,
> Quentin

Thanks,
Antonin
Quentin Schulz April 18, 2025, 11:28 a.m. UTC | #5
Hi Antonin,

On 4/17/25 9:21 AM, Antonin Godard wrote:
> Hi Quentin,
> 
> On Wed Apr 16, 2025 at 11:58 AM CEST, Quentin Schulz wrote:
>> Hi Antonin,
>>
>> On 4/16/25 9:46 AM, Antonin Godard wrote:
>>> Hi Quentin,
>>>
>>> On Mon Apr 14, 2025 at 2:18 PM CEST, Quentin Schulz wrote:
>>>> Hi Antonin,
>>>>
>>>> On 4/9/25 11:55 AM, Antonin Godard via lists.yoctoproject.org wrote:
>>>>> Introduce the DISTRO_LATEST_TAG macro, which should always point to the
>>>>> latest existing tag in the documentation, unlike DISTRO which may point
>>>>> to A.B.999 to represent the tip of a branch.
>>>>>
>>>>> This variable is needed to fix dead links in the documentation that
>>>>> currently use the DISTRO macro.
>>>>>
>>>>> Also, make DISTRO_REL_TAG use the DISTRO macro directly, to avoid
>>>>> repetition, and add a DISTRO_REL_LATEST_TAG macro that has the same role
>>>>> as DISTRO_LATEST_TAG but with "yocto-" prepended to it.
>>>>>
>>>>> In set_versions.py, run the "git describe --abbrev=0 --tags
>>>>> --match='yocto-*'" command to get the latest existing tag on the
>>>>> currently checked out commit. Fallback to ourversion in case we didn't
>>>>> find any.
>>>>>
>>>>> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
>>>>> ---
>>>>>     documentation/poky.yaml.in    | 11 ++++++++++-
>>>>>     documentation/set_versions.py | 16 +++++++++++++++-
>>>>>     2 files changed, 25 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
>>>>> index 836f11454..26c21e346 100644
>>>>> --- a/documentation/poky.yaml.in
>>>>> +++ b/documentation/poky.yaml.in
>>>>> @@ -2,13 +2,22 @@
>>>>>     # Macros used in the documentation
>>>>>     #
>>>>>     
>>>>> +# The DISTRO variable represents the current docs version. It should be used
>>>>> +# when referring to the current docs version. See also DISTRO_LATEST_TAG.
>>>>>     DISTRO : "5.1"
>>>>> +# The DISTRO_LATEST_TAG represents the latest tag on the current branch. It
>>>>> +# should be used in HTTP link referring to the current docs version. In these
>>>>> +# cases, the DISTRO may point to A.B.999 which does not exist (just used to
>>>>> +# represent the latest HEAD revision on the branch). DISTRO_LATEST_TAG should
>>>>> +# always point to an existing tag.
>>>>
>>>> I don't remember why/where we need A.B.999? I assume most shouldn't
>>>> point at .999, ever?
>>>
>>> The default landing page of docs.yoctoproject.org is a 999 version (tip of the
>>> stable release branch).
>>>
>>
>> That essentially should be for the version number in text and in the JS
>> dropdown? We should really check our use of &DISTRO; in the docs.
>>
>> For example:
>>
>> documentation/contributor-guide/report-defect.rst, not sure we don't
>> want to use &DISTRO_LATEST_TAG; (or another variable) instead, as it'll
>> always be wrong on the default landing page of the docs (always x.x.999)?
>>
>> documentation/dev-manual/multiconfig.rst, I assume we also do not want
>> to have x.x.999 there? I'm also not entirely sure why we explicitly list
>> the version or if it's still appropriate?
>>
>> documentation/dev-manual/start.rst:774 we should probably get rid of
>> &DISTRO; (but we can keep &DISTRO_NAME; as that matches the code block
>> under?)
>>
>> documentation/dev-manual/start.rst:843 is plain wrong as the tag does
>> not exist so we should be using DISTRO_LATEST_TAG. Same for
>> documentation/dev-manual/start.rst:854, same for
>> documentation/overview-manual/development-environment.rst:459.
>>
>> documentation/ref-manual/release-process.rst:15 should probably use
>> DISTRO_NAME or DISTRO_LATEST_TAG? (or probably another variable which is
>> only X.Y and not X.Y.Z if there's one, since X.Y.Z is a minor release).
>>
>> documentation/ref-manual/release-process.rst:59 probably should be using
>> DISTRO_LATEST_TAG too?
>>
>> documentation/ref-manual/system-requirements.rst:58, could be using
>> DISTRO_LATEST_TAG but probably not a big deal if we keep DISTRO instead?
>>
>> documentation/ref-manual/system-requirements.rst:349, I woudl assume is
>> completely wrong and should be DISTRO_LATEST_TAG instead? Same for
>> documentation/ref-manual/system-requirements.rst:350?
>>
>> documentation/ref-manual/system-requirements.rst:403,407,411,474,478,482
>> need to use DISTRO_LATEST_TAG to match the name of the buildtools.
>>
>> documentation/sdk-manual/appendix-obtain.rst:66,72,176 needs to match
>> the toolchain you can download, so needs to be DISTRO_LATEST_TAG.
>>
>> documentation/sdk-manual/appendix-obtain.rst:275 probably wants
>> DISTRO_LATEST_TAG too.
>>
>> documentation/sdk-manual/extensible.rst:120,124,126 probably wants
>> DISTRO_LATEST_TAG too.
>>
>> documentation/sdk-manual/using.rst:74,78,80,101,102,104,105,110,130,141,
>> probably wants DISTRO_LATEST_TAG too.
>>
>> documentation/sdk-manual/working-projects.rst:85,88,219,223,225,280
>> probably wants DISTRO_LATEST_TAG too.
>>
>> documentation/toaster-manual/reference.rst:161,295 DISTRO_LATEST_TAG
> 
> Thanks for going through the different usages of DISTRO. You're right, with
> DISTRO_LATEST_TAG now I also think it would make more sense to use it rather
> than DISTRO. I will try to work on that a bit later (or feel free to send a
> patch :), the initial focus for this series was to fix dead links .
> 

I haven't checked if all links are now valid, but for sure some of our 
instructions aren't. I think they are as important :) or even more so 
because linkcheck wouldn't be able to catch those issues.

[...]

>> I seem to recall we wanted the variables set by set_versions.py to still
>> be able to build the docs from tarballs and not force the use of git
>> repos? But it seems like we force to fetch git tags at the very
>> beginning of the Python script.
> 
>    $ make html
>    ./set_versions.py
>    Please run 'git fetch --tags' before building the documentation
>    make: *** [Makefile:72: html] Error 1
> 
> It is indeed a bug!
> Created one here: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15834
> 
>> So I'm wondering if we should only have variables that aren't replaced
>> in poky.yaml.in and eventually have the ones set_versions.py adds in
>> poky.yaml (instead of replacing for example) documented in comments to
>> explain what they are and in which context(s) to use them?
> 
> That would be a bit cleaner yes.
> 
> Maybe standards.md would be a good place? i.e.:
> "Use &DISTRO; in the docs when wanting to refer to the latest possible version"
> "Use &DISTRO_LATEST_TAG; in the docs when wanting to refer to the latest
> existing tag"
> etc.
> 

If we need to restore building the docs from tarballs, then we need 
default values (and appropriate ones!) in poky.yaml.in. We could still 
explain proper usage in standards.md , though, up to you.

I would simply split the file in two with a big comment before the 
second section where we would define overridden variables?

I don't think we should be using &DISTRO; anymore? Most of the time we 
don't want to be printing .999 on the landing page of 
docs.yoctoproject.org! For the dev page, that's different.

Cheers,
Quentin
diff mbox series

Patch

diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
index 836f11454..26c21e346 100644
--- a/documentation/poky.yaml.in
+++ b/documentation/poky.yaml.in
@@ -2,13 +2,22 @@ 
 # Macros used in the documentation
 #
 
+# The DISTRO variable represents the current docs version. It should be used
+# when referring to the current docs version. See also DISTRO_LATEST_TAG.
 DISTRO : "5.1"
+# The DISTRO_LATEST_TAG represents the latest tag on the current branch. It
+# should be used in HTTP link referring to the current docs version. In these
+# cases, the DISTRO may point to A.B.999 which does not exist (just used to
+# represent the latest HEAD revision on the branch). DISTRO_LATEST_TAG should
+# always point to an existing tag.
+DISTRO_LATEST_TAG : "5.1"
 DISTRO_NAME_NO_CAP : "styhead"
 DISTRO_NAME : "Styhead"
 DISTRO_NAME_NO_CAP_MINUS_ONE : "scarthgap"
 DISTRO_NAME_NO_CAP_LTS : "scarthgap"
 YOCTO_DOC_VERSION : "5.1"
-DISTRO_REL_TAG : "yocto-5.1"
+DISTRO_REL_TAG : "yocto-$DISTRO;"
+DISTRO_REL_LATEST_TAG : "yocto-&DISTRO_LATEST_TAG;"
 DOCCONF_VERSION : "dev"
 BITBAKE_SERIES : ""
 YOCTO_DL_URL : "https://downloads.yoctoproject.org"
diff --git a/documentation/set_versions.py b/documentation/set_versions.py
index 5c55f470d..b94a7daad 100755
--- a/documentation/set_versions.py
+++ b/documentation/set_versions.py
@@ -170,17 +170,29 @@  series = [k for k in release_series]
 previousseries = series[series.index(ourseries)+1:] or [""]
 lastlts = [k for k in previousseries if k in ltsseries] or "dunfell"
 
+latestreltag = subprocess.run(["git", "describe", "--abbrev=0", "--tags", "--match", "yocto-*"], capture_output=True, text=True).stdout
+latestreltag = latestreltag.strip()
+if latestreltag:
+    if latestreltag.startswith("yocto-"):
+        latesttag = latestreltag[6:]
+else:
+    # fallback on the calculated version
+    print("Did not find a tag with 'git describe', falling back to %s" % ourversion)
+    latestreltag = "yocto-" + ourversion
+    latesttag = ourversion
+
 print("Version calculated to be %s" % ourversion)
+print("Latest release tag found is %s" % latestreltag)
 print("Release series calculated to be %s" % ourseries)
 
 replacements = {
     "DISTRO" : ourversion,
+    "DISTRO_LATEST_TAG": latesttag,
     "DISTRO_NAME_NO_CAP" : ourseries,
     "DISTRO_NAME" : ourseries.capitalize(),
     "DISTRO_NAME_NO_CAP_MINUS_ONE" : previousseries[0],
     "DISTRO_NAME_NO_CAP_LTS" : lastlts[0],
     "YOCTO_DOC_VERSION" : ourversion,
-    "DISTRO_REL_TAG" : "yocto-" + ourversion,
     "DOCCONF_VERSION" : docconfver,
     "BITBAKE_SERIES" : bitbakeversion,
 }
@@ -318,3 +330,5 @@  with open('releases.rst', 'w') as f:
             if tag == release_series[series] or tag.startswith('%s.' % release_series[series]):
                 f.write('- :yocto_docs:`%s Documentation </%s>`\n' % (tag, tag))
         f.write('\n')
+
+