diff mbox series

[yocto-autobuilder-helper,1/3] scripts/run-docs-build: cleanups and improvements

Message ID 20250915-pass-bitbake-active-releases-v1-1-a5530a97b9ef@bootlin.com
State New
Headers show
Series scripts/run-docs-build: improvements for the bitbake menu | expand

Commit Message

Antonin Godard Sept. 15, 2025, 12:19 p.m. UTC 
- Make it possible to override the shared directory to a local directory
  from environment
- Make variables that can be overridden from environment variables
  uppercase, for clarity
- Convert mkdir to mkdir -p when creating output directories
- Minor indent fixes and overall cleanups

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
---
 scripts/run-docs-build | 78 +++++++++++++++++++++++++++++---------------------
 1 file changed, 46 insertions(+), 32 deletions(-)

Comments

Quentin Schulz Sept. 15, 2025, 12:53 p.m. UTC | #1 
Hi Antonin,

On 9/15/25 2:19 PM, Antonin Godard via lists.yoctoproject.org wrote:
> - Make it possible to override the shared directory to a local directory
>    from environment
> - Make variables that can be overridden from environment variables
>    uppercase, for clarity

If you want to keep exisiting users of this script working, you can keep 
reading the lowercase variant and assign it to the uppercase variant as 
default if it isn't already set for example.

buildtools_url=${buildtools_url:-https://downloads.yoctoproject.org/releases/yocto/yocto-4.1.2/buildtools/x86_64-buildtools-docs-nativesdk-standalone-5.1-modified.sh}
BUILDTOOLS_URL=${BUILDTOOLS_URL:-$buildtools_url}

> - Convert mkdir to mkdir -p when creating output directories

Why? mkdir -p silences some errors. I assume we want to make sure the 
directory doesn't exist already so we don't reuse some stale data.

> - Minor indent fixes and overall cleanups
> 

One commit per item in this list please, so it's easier to review :)

Cheers,
Quentin
Antonin Godard Sept. 15, 2025, 2:02 p.m. UTC | #2 
On Mon Sep 15, 2025 at 2:53 PM CEST, Quentin Schulz via lists.yoctoproject.org wrote:
> Hi Antonin,
>
> On 9/15/25 2:19 PM, Antonin Godard via lists.yoctoproject.org wrote:
>> - Make it possible to override the shared directory to a local directory
>>    from environment
>> - Make variables that can be overridden from environment variables
>>    uppercase, for clarity
>
> If you want to keep exisiting users of this script working, you can keep 
> reading the lowercase variant and assign it to the uppercase variant as 
> default if it isn't already set for example.
>
> buildtools_url=${buildtools_url:-https://downloads.yoctoproject.org/releases/yocto/yocto-4.1.2/buildtools/x86_64-buildtools-docs-nativesdk-standalone-5.1-modified.sh}
> BUILDTOOLS_URL=${BUILDTOOLS_URL:-$buildtools_url}

I didn't have this in mind, I can surely do that.

>> - Convert mkdir to mkdir -p when creating output directories
>
> Why? mkdir -p silences some errors. I assume we want to make sure the 
> directory doesn't exist already so we don't reuse some stale data.

IIRC, I had issues building locally without this. I'll try again without this flag.

>> - Minor indent fixes and overall cleanups
>> 
>
> One commit per item in this list please, so it's easier to review :)

I'll try splitting it up.

Thanks,
Antonin
diff mbox series

Patch

diff --git a/scripts/run-docs-build b/scripts/run-docs-build
index a001fb2..aa1847d 100755
--- a/scripts/run-docs-build
+++ b/scripts/run-docs-build
@@ -7,53 +7,67 @@ 
 #             $3 as the path to bitbake
 #
 # Environment variables:
-#   - buildtools_url as the path to buildtools script for the docs.
-#                     Can be found here: https://autobuilder.yocto.io/pub/buildtools/
-#   - docbookarchive_url as the path to old (pre 3.1.5 and Sphinx migration) docs tarball
-#                     Can be found here: https://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-archives-20201105.tar.xz
-#   - PUBLISH (0/1) for whether the files should be rsync'ed to docs.yoctoproject.org
+#   - BUILDTOOLS_URL: path to buildtools script for the docs. Can be found here:
+#     https://autobuilder.yocto.io/pub/buildtools/
+#   - BUILDTOOLS_URL_OLD: path to the old buildtools script for docs releases
+#     older than scarthgap.
+#   - DOCBOOKARCHIVE_URL: path to old (pre 3.1.5 and Sphinx migration) docs
+#     tarball Can be found here: https://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-archives-20201105.tar.xz
+#   - SHAREDIR as the path to a shared directory that caches downloads. Default
+#     to $BASE_SHAREDDIR/cluster-downloads-cache, where BASE_SHAREDDIR is
+#     extracted from $HOME/config-local.json
+#   - PUBLISH (0/1) (default: 1): whether the files should be rsync'ed to docs.yoctoproject.org
 
 set -e
 set -u
 set -o pipefail
 set -x
 
+# Environment variables
+PUBLISH=${PUBLISH:-1}
+# Modified 5.1 tools image to include rsvg, switch to next released version
+BUILDTOOLS_URL=${BUILDTOOLS_URL:-https://downloads.yoctoproject.org/releases/yocto/yocto-4.1.2/buildtools/x86_64-buildtools-docs-nativesdk-standalone-5.1-modified.sh}
+BUILDTOOLS_URL_OLD=${BUILDTOOLS_URL_OLD:-https://downloads.yoctoproject.org/releases/yocto/yocto-4.1.2/buildtools/x86_64-buildtools-docs-nativesdk-standalone-4.1.2.sh}
+DOCBOOKARCHIVE_URL=${DOCBOOKARCHIVE_URL:-https://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-archives-20201105.tar.xz}
+SHAREDIR=${SHAREDIR:-"$(jq -r '.["BASE_SHAREDDIR"]' < $HOME/config-local.json)/cluster-downloads-cache"}
+
 builddir=$(realpath "$1")
 ypdocs=$(realpath "$2/documentation/")
 bbdocs=$(realpath "$3/doc/")
 outputdir=$builddir/output
 scriptdir="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"
-PUBLISH=${PUBLISH:-1}
-sharedir=`jq -r '.["BASE_SHAREDDIR"]' < $HOME/config-local.json`
-# Modified 5.1 tools image to include rsvg, switch to next released version
-buildtools_url=${buildtools_url:-https://downloads.yoctoproject.org/releases/yocto/yocto-4.1.2/buildtools/x86_64-buildtools-docs-nativesdk-standalone-5.1-modified.sh}
-buildtools_url_old=${buildtools_url_old:-https://downloads.yoctoproject.org/releases/yocto/yocto-4.1.2/buildtools/x86_64-buildtools-docs-nativesdk-standalone-4.1.2.sh}
-docbookarchive_url=${docbookarchive_url:-https://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-archives-20201105.tar.xz}
+# make sure SHAREDIR is absolute
+SHAREDIR=$(readlink -f "$SHAREDIR")
 
 
-buildtools_name=`basename $buildtools_url`
-buildtools_localpath=${sharedir}/cluster-downloads-cache/${buildtools_name}
+buildtools_name=`basename $BUILDTOOLS_URL`
+buildtools_localpath=${SHAREDIR}/${buildtools_name}
 if [ ! -e ${buildtools_localpath} ]; then
     mkdir -p `dirname ${buildtools_localpath}`
-    wget ${buildtools_url} -O ${buildtools_localpath}
+    wget ${BUILDTOOLS_URL} -O ${buildtools_localpath}
     chmod a+x ${buildtools_localpath}
 fi
+mkdir -p $builddir
 cd $builddir
-mkdir buildtools
-${buildtools_localpath} -y -d $builddir/buildtools
+mkdir -p buildtools
+if [ ! -d "$builddir/buildtools/sysroots" ]; then
+    ${buildtools_localpath} -y -d $builddir/buildtools
+fi
 
 
-buildtools_name_old=`basename $buildtools_url_old`
-buildtools_localpath_old=${sharedir}/cluster-downloads-cache/${buildtools_name_old}
+buildtools_name_old=`basename $BUILDTOOLS_URL_OLD`
+buildtools_localpath_old=${SHAREDIR}/${buildtools_name_old}
 if [ ! -e ${buildtools_localpath_old} ]; then
     mkdir -p `dirname ${buildtools_localpath_old}`
-    wget ${buildtools_url_old} -O ${buildtools_localpath_old}
+    wget ${BUILDTOOLS_URL_OLD} -O ${buildtools_localpath_old}
     chmod a+x ${buildtools_localpath_old}
 fi
 
 cd $builddir
-mkdir buildtools-old
-${buildtools_localpath_old} -y -d $builddir/buildtools-old
+mkdir -p buildtools-old
+if [ ! -d "$builddir/buildtools-old/sysroots" ]; then
+    ${buildtools_localpath_old} -y -d $builddir/buildtools-old
+fi
 
 
 # To build the latexpdf output we need Xetex (which can handle international characters)
@@ -84,21 +98,21 @@  fi
 export FONTCONFIG_PATH="$scriptdir/docsfontconfig"
 
 # Getting the old docbook built docs from an archive. Not rebuilding them.
-docbookarchive_localpath=${sharedir}/cluster-downloads-cache/`basename ${docbookarchive_url}`
+docbookarchive_localpath=${SHAREDIR}/`basename ${DOCBOOKARCHIVE_URL}`
 if [ ! -e ${docbookarchive_localpath} ]; then
     mkdir -p `dirname ${docbookarchive_localpath}`
-    wget ${docbookarchive_url} -O ${docbookarchive_localpath}
+    wget ${DOCBOOKARCHIVE_URL} -O ${docbookarchive_localpath}
 fi
 
-mkdir $outputdir
+mkdir -p $outputdir
 cd $outputdir
-echo Extracing old content from archive
+echo Extracting old content from archive
 tar --exclude=eclipse -xJf ${docbookarchive_localpath}
 
 $scriptdir/docs_add_banner.py
 
 cd $bbdocs
-mkdir $outputdir/bitbake
+mkdir -p $outputdir/bitbake
 
 # A decision was made to keep updating all the Sphinx generated docs for the moment,
 # even the ones corresponding to no longer supported releases
@@ -128,16 +142,16 @@  for branch in 1.46 $(git branch --remote --contains "$first_sphinx_commit" --for
 
     if [ "$branch" = "master-next" ]; then
         branch="next"
-	mkdir $outputdir/bitbake/$branch
+        mkdir -p $outputdir/bitbake/$branch
     elif [ "$branch" = "master" ]; then
         branch="dev"
-        mkdir $outputdir/bitbake/$branch
+        mkdir -p $outputdir/bitbake/$branch
     elif [ "$branch" = "$latest_branch" ]; then
         branch=""
-        mkdir $outputdir/bitbake/$latest_branch
+        mkdir -p $outputdir/bitbake/$latest_branch
         cp -r ./_build/final/* $outputdir/bitbake/$latest_branch
     else
-	mkdir $outputdir/bitbake/$branch
+        mkdir -p $outputdir/bitbake/$branch
     fi
 
     cp -r ./_build/final/* $outputdir/bitbake/$branch
@@ -222,7 +236,7 @@  for branch in "$ypdocsbranch" dunfell $(git branch --remote --contains "$first_s
 
     if [ "$branch" = "master-next" ]; then
         branch="next"
-	mkdir -p $outputdir/$branch
+        mkdir -p $outputdir/$branch
     elif [ "$branch" = "master" ]; then
         branch="dev"
         mkdir -p $outputdir/$branch
@@ -235,7 +249,7 @@  for branch in "$ypdocsbranch" dunfell $(git branch --remote --contains "$first_s
     elif [ "$branch" = "transition" ]; then
         branch=""
     else
-	mkdir -p $outputdir/$branch
+        mkdir -p $outputdir/$branch
     fi
 
     cp -r ./_build/final/* $outputdir/$branch