[auh] weeklyjob: Add Openembedded update-helper conf

- Add weeklyjob for openembedded meta
- Add upgrade-helper.in config template

auto-upgrade-helper script accept configurations to specify the layer
to be analyzed.
The .in file is modified by sed command to generate the .conf file
that match the meta to analyze.

We've chosen to use a template configuration file rather than multiple
configuration files (1 per meta). The template file contains tags
between braces that is replaced in the script using simple `sed` calls.
The meta list to be analyzed is defined via the `meta_list` variable.
This variable matches all `meta-*` directories

Signed-off-by: David Pierret <david.pierret@smile.fr>
Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
---
 upgrade-helper.in | 124 ++++++++++++++++++++++++++++++++++++++++++++++
 weeklyjob-oe.sh   |  48 ++++++++++++++++++
 2 files changed, 172 insertions(+)
 create mode 100644 upgrade-helper.in
 create mode 100644 weeklyjob-oe.sh

On Tue, 10 Oct 2023 at 16:59, David Pierret <david.pierret@smile.fr> wrote:
> We've chosen to use a template configuration file rather than multiple
> configuration files (1 per meta). The template file contains tags
> between braces that is replaced in the script using simple `sed` calls.
> The meta list to be analyzed is defined via the `meta_list` variable.
> This variable matches all `meta-*` directories

I would suggest that AUH itself is modified to accept the dynamic
layer_* parts as command line arguments. Tweaking things with sed is
difficult to understand or maintain, and can be easily avoided here.

Then the other commit for autobuilder-helper can as well be greatly
simplified and can reuse existing scripts instead of
copy-paste-tweaking them.

Alex



> Signed-off-by: David Pierret <david.pierret@smile.fr>
> Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
> ---
>  upgrade-helper.in | 124 ++++++++++++++++++++++++++++++++++++++++++++++
>  weeklyjob-oe.sh   |  48 ++++++++++++++++++
>  2 files changed, 172 insertions(+)
>  create mode 100644 upgrade-helper.in
>  create mode 100644 weeklyjob-oe.sh
>
> diff --git a/upgrade-helper.in b/upgrade-helper.in
> new file mode 100644
> index 0000000..4eedef1
> --- /dev/null
> +++ b/upgrade-helper.in
> @@ -0,0 +1,124 @@
> +#
> +# This file is your upgrade-helper configuration files and is where all local user
> +# settings specifically for AUH are placed. The comments in this file provide some
> +# guidelines to the options a new user to AUH tool might want to change.
> +#
> +# This file should be placed inside your BUILDDIR/upgrade-helper after initializing
> +# the OE build environment (this location can be overriden with -c command line option).
> +#
> +# Lines starting with the '#' character are commented out and in some cases the
> +# default values are provided as comments to show people example syntax. Enabling
> +# the option is a question of removing the # character and making any change to the
> +# variable as required. For boolean settings, anything not 'yes' is treated as 'no'.
> +
> +[maintainer_override]
> +# e-mail message for recipe upgrades will go to john.doe instead of jane.doe, etc
> +# see also the global_maintainer_override option
> +#jane.doe@doe.com=john.doe@doe.com
> +#johhny.bravo@bravo.com=john.doe@doe.com
> +
> +[settings]
> +# SMTP server that is used to send e-mails with patches and other information.
> +# If you are running AUH locally, you do not need to set this up, as AUH
> +# saves everything to BUILDDIR/upgrade-helper/<timestamp>, and does not attempt
> +# to send email messages (unless explicitly asked with -e command line option).
> +# If no port is specified, port 25 is assumed.
> +#smtp=smtp.my-server.com:25
> +
> +# from whom should the e-mails be sent (mandatory if --send-emails is passed).
> +# Also sets the email address of the author of automated commits.
> +#from=uh@not.set
> +
> +# If enabled, emails for all recipe upgrades will go to john.doe,
> +# except when recipes are owned by specific maintainer_override entries above.
> +#global_maintainer_override=john.doe@doe.com
> +
> +# who should be CCd with all upgrade emails (optional)
> +#cc_recipients=john.doe@doe.com
> +
> +# who should get the status mail with statistics, at the end (mandatory if
> +# --send-emails is passed)
> +#status_recipients=john.doe@doe.com
> +
> +# Only recipes belonging to maintainers in whitelist will be attempted
> +# Insert your own e-mail address here together with any other maintainers' addresses
> +# whose recipes you wish to take care of. This restricts the set of recipes that AUH
> +# will attempt when it is run with 'all' option.
> +#maintainers_whitelist=jane.doe@doe.com john.doe@doe.com johhny.bravo@bravo.com
> +
> +# recipes in blacklist will be skipped (applies only when 'all' or no recipe is
> +# passed; does not apply when layer_mode is enabled).
> +#blacklist=python glibc gcc
> +
> +# specify the directory where work (patches) will be saved
> +# (optional; default is BUILDDIR/upgrade-helper/)
> +#workdir=
> +
> +# public url with AUH results to include in statistics summary (optional)
> +#publish_work_url=http://auh.somehost.com/work
> +
> +# whether to attach a tarball with AUH artefacts (patches, log, buildhistory, etc.)
> +# to the summary email
> +#summary_includes_tarball=True
> +
> +# clean sstate directory before upgrading
> +# Generally not necessary, as bitbake can handle this automatically.
> +#clean_sstate=no
> +
> +# clean tmp directory before upgrading
> +# Generally not necessary as bitbake can handle this automatically.
> +#clean_tmp=no
> +
> +# Machines to test build with.
> +# Append _libc-name to test with alternative C library implementations
> +# e.g. qemux86_musl.
> +#
> +# Buildhistory and testimages will be created only for the first
> +# machine in the list, as otherwise it adds enormously to AUH run time.
> +#
> +# AUH has a reasonable default for this, so you do not need to set your own,
> +# at least initially.
> +#
> +# Does not apply when layer_mode is enabled.
> +#machines=qemux86 qemux86-64 qemuarm qemumips qemuppc qemux86_musl
> +
> +# Enables buildhistory feature; this is useful as it produces information
> +# about what has changed in the resulting packages, compared to previous version
> +#
> +# Requires 'buildhistory' to be present in INHERIT and BUILDHISTORY_COMMIT to be set
> +# in your conf/local.conf.
> +#buildhistory=no
> +
> +# When AUH has built an upgraded recipe it then creates a commit with the upgrade.
> +# This setting specifies whether to also revert the commit. Possible values are:
> +# 'failed_to_build' (default) - revert if the recipe failed to build. This is useful
> +# to prevent further upgrades in the batch from failing, if they're dependent on the
> +# recipe.
> +# 'all' - always revert. This is useful when running automatic, unattended upgrades,
> +#  as it will effectively test each upgrade against the baseline, and will avoid the
> +# situation where one successful upgrade thwarts others due to newly introduced
> +# incompatibilities.
> +# 'never' - never revert. Use in interactive upgrade sessions, where any issues can be
> +# manually fixed.
> +#commit_revert_policy=failed_to_build
> +
> +# If enabled, build and boots a test image, and runs integration tests on it
> +# If upgraded packages have ptest support those are run as well
> +#
> +# Requires 'testimage' in INHERIT in your conf/local.conf and 'ptest' in your
> +# distro's DISTRO_FEATURES.
> +#testimage=no
> +
> +# This can be used to change the name of the test image.
> +#
> +#testimage_name=core-image-sato
> +
> +# This can be used to upgrade recipes in a specific layer,
> +# for example meta-intel, instead of upgrading oe-core recipes.
> +#
> +# When layer_mode is enabled, layer_name, layer_dir and layer_machines are
> +# mandatory. 'blacklist' setting does not apply when in layer_mode.
> +layer_mode=yes
> +layer_name={meta}
> +layer_dir={layerdir}
> +layer_machines={machine}
> diff --git a/weeklyjob-oe.sh b/weeklyjob-oe.sh
> new file mode 100644
> index 0000000..1bad6da
> --- /dev/null
> +++ b/weeklyjob-oe.sh
> @@ -0,0 +1,48 @@
> +#!/bin/bash
> +
> +# Cronjob which can be run weekly to run the upgrade helper script.
> +# Add the job in /etc/crontab like below.
> +#
> +# It will execute weekly at the same hour (8 AM).
> +#
> +# 00 8   * * 6   auh  /home/auh/bin/weeklyjob-oe.sh
> +
> +# Re-assign these to match your setup!
> +auh_dir=~/auto-upgrade-helper
> +poky_dir=~/poky
> +openembedded_dir=~/meta-openembedded
> +build_dir=~/build-tmp-auh-upgrades
> +sstate_dir=~/sstate-cache
> +
> +meta_list="${openembedded_dir}/meta-*"
> +machine_list="qemux86 qemux86-64 qemuarm qemumips qemuppc qemux86_musl"
> +
> +pushd $openembedded_dir || exit 1
> +
> +source $poky_dir/oe-init-build-env $build_dir
> +
> +# Base the upgrades on meta_openembedded master
> +git fetch origin
> +git checkout -B tmp-auh-upgrades origin/main
> +# upgrade helper need to use ?= for MACHINE
> +sed -i.bak -E 's|^(MACHINE[[:space:]]*)[?+=]+([[:space:]]*.*)$|\1?=\2|' $build_dir/conf/local.conf
> +
> +mkdir $build_dir/upgrade-helper/
> +
> +for d in $meta_list; do
> +  meta=$(basename "$d")
> +  sed -E "s|\{meta\}|$meta| ; \
> +          s|\{layerdir\}|$openembedded_dir/$meta| ; \
> +          s|\{machine\}|$machine_list|" \
> +          $auh_dir/upgrade-helper.in \
> +          > $build_dir/upgrade-helper/upgrade-helper.conf
> +  python3 $auh_dir/upgrade-helper.py -e all
> +done
> +
> +# clean up to avoid the disk filling up
> +rm -rf $build_dir/tmp/
> +rm -rf $build_dir/workspace/sources/*
> +find $sstate_dir -atime +10 -delete
> +
> +popd || exit 1
> +
> --
> 2.39.2
>

On Wed, 2023-10-11 at 11:02 +0200, Alexander Kanavin wrote:
> On Tue, 10 Oct 2023 at 16:59, David Pierret <david.pierret@smile.fr> wrote:
> > We've chosen to use a template configuration file rather than multiple
> > configuration files (1 per meta). The template file contains tags
> > between braces that is replaced in the script using simple `sed` calls.
> > The meta list to be analyzed is defined via the `meta_list` variable.
> > This variable matches all `meta-*` directories
> 
> I would suggest that AUH itself is modified to accept the dynamic
> layer_* parts as command line arguments. Tweaking things with sed is
> difficult to understand or maintain, and can be easily avoided here.
> 
> Then the other commit for autobuilder-helper can as well be greatly
> simplified and can reuse existing scripts instead of
> copy-paste-tweaking them.

I agree, the goal here is to be able to use these tools with other
layers so fixing them to be more generic and configurable rather than
hacking them is much preferred.

Cheers,

Richard