Makefile/set_versions: Allow poky.yaml to be autogenerated

Use a script to generate the branch/tag information inside poky.yaml.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 documentation/Makefile                    |   1 +
 documentation/{poky.yaml => poky.yaml.in} |   0
 documentation/set_versions.py             | 102 ++++++++++++++++++++++
 3 files changed, 103 insertions(+)
 rename documentation/{poky.yaml => poky.yaml.in} (100%)
 create mode 100755 documentation/set_versions.py

Hi Richard,

Thanks for this good idea... A few issues though... See my comments below.

On 3/18/22 14:08, Richard Purdie wrote:
>
> diff --git a/documentation/Makefile b/documentation/Makefile
> index f04f381bd..bec53399c 100644
> --- a/documentation/Makefile
> +++ b/documentation/Makefile
> @@ -57,4 +57,5 @@ all: html epub latexpdf
>  # Catch-all target: route all unknown targets to Sphinx using the new
>  # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
>  %:
> +	$(SOURCEDIR)/set_versions.py
>  	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
> diff --git a/documentation/poky.yaml b/documentation/poky.yaml.in
> similarity index 100%
> rename from documentation/poky.yaml
> rename to documentation/poky.yaml.in
> diff --git a/documentation/set_versions.py b/documentation/set_versions.py
> new file mode 100755
> index 000000000..cdcfd6bb0
> --- /dev/null
> +++ b/documentation/set_versions.py
> @@ -0,0 +1,102 @@
> +#!/usr/bin/env python3
> +#
> +# Add version information to poky.yaml based upon current git branch/tags
> +#
> +# Copyright Linux Foundation
> +# Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +
> +import subprocess
> +import collections
> +import sys
> +
> +devbranch = "kirkstone"
> +#devbranch = "langdale"
> +ltsseries = ["kirkstone", "dunfell"]
> +
> +release_series = collections.OrderedDict()
> +#release_series["langdale"] = "4.1"
> +release_series["kirkstone"] = "4.0"
> +release_series["honister"] = "3.4"
> +release_series["hardknott"] = "3.3"
> +release_series["gatesgarth"] = "3.2"
> +release_series["dunfell"] = "3.1"
> +
> +ourversion = None
> +ourseries = None
> +ourbranch = None
> +
> +# Test tags exist and inform the user to fetch if not
> +try:
> +    subprocess.run(["git", "show", "yocto-3.4.2"], capture_output=True, check=True)
> +except subprocess.CalledProcessError:
> +    sys.exit("Please run 'git fetch --tags' before building the documentation")
> +
> +# Try and figure out what we are
> +tags = subprocess.run(["git", "tag", "--points-at", "HEAD"], capture_output=True, text=True).stdout
> +for t in tags.split():
> +    if t.startswith("yocto-"):
> +        ourversion = t[6:]
> +
> +if ourversion:
> +    # We're a tagged release
> +    components = ourversion.split(".")
> +    baseversion = components[0] + "." + components[1]
> +    for i in release_series:
> +        if release_series[i] == baseversion:
> +            ourseries = i
> +            ourbranch = i
> +else:
> +    # We're floating on a branch
> +    branch = subprocess.run(["git", "branch", "--show-current"], capture_output=True, text=True).stdout.strip()
> +    ourbranch = branch
> +    if branch == "master":
> +        ourseries = devbranch
> +    elif branch in release_series:
> +        ourseries = branch
> +    else:
> +        sys.exit("Unknown series for branch %s" % branch)


This last message is problematic when you create an arbitrary branch to
develop new changes, or test changes from others.
Could you just set

ourseries = devbranch

instead of exiting with an error.

> +
> +    previoustags = subprocess.run(["git", "tag", "--merged", "HEAD"], capture_output=True, text=True).stdout
> +    previoustags = [t[6:] for t in previoustags.split() if t.startswith("yocto-" + release_series[ourseries])]
> +    futuretags = subprocess.run(["git", "tag", "--merged", ourbranch], capture_output=True, text=True).stdout
> +    futuretags = [t[6:] for t in futuretags.split() if t.startswith("yocto-" + release_series[ourseries])]
> +
> +    # Append .999 against the last known version
> +    if len(previoustags) != len(futuretags):
> +        ourversion = previoustags[-1] + ".999"
> +    else:
> +        ourversion = release_series[ourseries] + ".999"
> +
> +series = [k for k in release_series]
> +previousseries = series[series.index(ourseries)+1:]
> +lastlts = [k for k in previousseries if k in ltsseries]
> +
> +print("Version calculated to be %s" % ourversion)
> +print("Release series calculated to be %s" % ourseries)
> +
> +replacements = {
> +    "DISTRO" : ourversion,
> +    "DISTRO_NAME_NO_CAP" : ourseries,
> +    "DISTRO_NAME" : ourseries.capitalize(),
> +    "DISTRO_NAME_NO_CAP_MINUS_ONE" : previousseries[0],
> +    "DISTRO_NAME_NO_CAP_LTS" : lastlts[0],
> +    "YOCTO_DOC_VERSION" : ourversion,
> +    "DISTRO_REL_TAG" : "yocto-" + ourversion,
> +}
> +
> +with open("poky.yaml.in", "r") as r, open("poky.yaml", "w") as w:
> +    lines = r.readlines()
> +    for line in lines:
> +        data = line.split(":")
> +        k = data[0].strip()
> +        if k in replacements:
> +            w.write("%s : \"%s\"\n" % (k, replacements[k]))
> +        else:
> +            w.write(line)
> +
> +print("poky.yaml generated from poky.yaml.in")


Didn't you forget to add "documentation/poky.yaml" to the .gitignore file?

Thanks again
Michael.

On Fri, 2022-03-18 at 15:46 +0100, Michael Opdenacker wrote:
> Hi Richard,
> 
> Thanks for this good idea... A few issues though... See my comments below.
> 
> On 3/18/22 14:08, Richard Purdie wrote:
> > 
> > diff --git a/documentation/Makefile b/documentation/Makefile
> > index f04f381bd..bec53399c 100644
> > --- a/documentation/Makefile
> > +++ b/documentation/Makefile
> > @@ -57,4 +57,5 @@ all: html epub latexpdf
> >  # Catch-all target: route all unknown targets to Sphinx using the new
> >  # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
> >  %:
> > +	$(SOURCEDIR)/set_versions.py
> >  	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
> > diff --git a/documentation/poky.yaml b/documentation/poky.yaml.in
> > similarity index 100%
> > rename from documentation/poky.yaml
> > rename to documentation/poky.yaml.in
> > diff --git a/documentation/set_versions.py b/documentation/set_versions.py
> > new file mode 100755
> > index 000000000..cdcfd6bb0
> > --- /dev/null
> > +++ b/documentation/set_versions.py
> > @@ -0,0 +1,102 @@
> > +#!/usr/bin/env python3
> > +#
> > +# Add version information to poky.yaml based upon current git branch/tags
> > +#
> > +# Copyright Linux Foundation
> > +# Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> > +#
> > +# SPDX-License-Identifier: MIT
> > +#
> > +
> > +
> > +import subprocess
> > +import collections
> > +import sys
> > +
> > +devbranch = "kirkstone"
> > +#devbranch = "langdale"
> > +ltsseries = ["kirkstone", "dunfell"]
> > +
> > +release_series = collections.OrderedDict()
> > +#release_series["langdale"] = "4.1"
> > +release_series["kirkstone"] = "4.0"
> > +release_series["honister"] = "3.4"
> > +release_series["hardknott"] = "3.3"
> > +release_series["gatesgarth"] = "3.2"
> > +release_series["dunfell"] = "3.1"
> > +
> > +ourversion = None
> > +ourseries = None
> > +ourbranch = None
> > +
> > +# Test tags exist and inform the user to fetch if not
> > +try:
> > +    subprocess.run(["git", "show", "yocto-3.4.2"], capture_output=True, check=True)
> > +except subprocess.CalledProcessError:
> > +    sys.exit("Please run 'git fetch --tags' before building the documentation")
> > +
> > +# Try and figure out what we are
> > +tags = subprocess.run(["git", "tag", "--points-at", "HEAD"], capture_output=True, text=True).stdout
> > +for t in tags.split():
> > +    if t.startswith("yocto-"):
> > +        ourversion = t[6:]
> > +
> > +if ourversion:
> > +    # We're a tagged release
> > +    components = ourversion.split(".")
> > +    baseversion = components[0] + "." + components[1]
> > +    for i in release_series:
> > +        if release_series[i] == baseversion:
> > +            ourseries = i
> > +            ourbranch = i
> > +else:
> > +    # We're floating on a branch
> > +    branch = subprocess.run(["git", "branch", "--show-current"], capture_output=True, text=True).stdout.strip()
> > +    ourbranch = branch
> > +    if branch == "master":
> > +        ourseries = devbranch
> > +    elif branch in release_series:
> > +        ourseries = branch
> > +    else:
> > +        sys.exit("Unknown series for branch %s" % branch)
> 
> 
> This last message is problematic when you create an arbitrary branch to
> develop new changes, or test changes from others.
> Could you just set
> 
> ourseries = devbranch
> 
> instead of exiting with an error.

If only it were that simple. It needs to resolve this back to a valid parent
branch, i.e. one in release_series.

I think we can do that with (deep breath):

git show-branch <release_series list> | grep '*' | grep -v "$(git rev-parse --abbrev-ref HEAD)" | head -n1 | sed 's/.*\[\(.*\)\].*/\1/' | sed 's/[\^~].*//'

so perhaps we update it to do that?

> Didn't you forget to add "documentation/poky.yaml" to the .gitignore file?

True. That is easier to fix.

Cheers,

Richard