| Message ID | 20231026130340.2070302-1-david.pierret@smile.fr |
|---|---|
| State | New |
| Headers | show |
| Series | [yocto-autobuilder-helper,v3] AUH: Add Openembedded auto-update-helper with list of layer to test | expand |
This looks basically ok, but how can this be tested if needed changes
to AUH are still in review?
Running auh over meta-oe layers shouldn't be defined as steps that are
run one after the other, but as separate parallelizable jobs, e.g.
"auh-meta-oe" : {
...
"auh-meta-python" : {
...
etc. They could use a job template so you only need to set the varying
parameters (config.json has examples).
Alex
On Thu, 26 Oct 2023 at 15:06, David Pierret <david.pierret@smile.fr> wrote:
>
> - Modify the setup-auh to use repositories cloned by the auto-builder.
> Manage the case where meta-openembedded is not used.
>
> - Modify the run-auh to manage the optional additional layer.
> Only 1 layer can be provided.
>
> - Add auh-meta-openembedded job with 1 step per layer.
>
> Signed-off-by: David Pierret <david.pierret@smile.fr>
> ---
> config.json | 102 ++++++++++++++++++++++++++++++++++++++++++++--
> scripts/run-auh | 39 ++++++++++++------
> scripts/setup-auh | 34 ++++++++++------
> 3 files changed, 146 insertions(+), 29 deletions(-)
>
> diff --git a/config.json b/config.json
> index 3acb710..a168e48 100644
> --- a/config.json
> +++ b/config.json
> @@ -1416,9 +1416,99 @@
> "TEMPLATE" : "buildperf"
> },
> "auh" : {
> - "EXTRAPLAINCMDS" : [
> - "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; ${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/"
> - ]
> + "NEEDREPOS" : ["poky", "auto-upgrade-helper"],
> + "step1" : {
> + "shortname" : "setup AUH",
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}"
> + ]
> + },
> + "step2" : {
> + "shortname" : "run AUH poky",
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/"
> + ]
> + }
> + },
> + "auh-meta-openembedded" : {
> + "NEEDREPOS" : ["poky", "meta-openembedded", "auto-upgrade-helper"],
> + "step1" : {
> + "shortname" : "setup AUH",
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}"
> + ]
> + },
> + "step2" : {
> + "shortname" : "run AUH for layer meta-oe",
> +
> + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-oe"],
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-oe"
> + ]
> + },
> + "step3" : {
> + "shortname" : "run AUH for layer meta-python",
> + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-python"],
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-python"
> + ]
> + },
> + "step4" : {
> + "shortname" : "run AUH for layer meta-perl",
> + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-perl"],
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-perl"
> + ]
> + },
> + "step5" : {
> + "shortname" : "run AUH for layer meta-networking",
> + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-networking"],
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-networking"
> + ]
> + },
> + "step6" : {
> + "shortname" : "run AUH for layer meta-multimedia",
> + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-multimedia"],
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-multimedia"
> + ]
> + },
> + "step7" : {
> + "shortname" : "run AUH for layer meta-gnome",
> + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-gnome"],
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-gnome"
> + ]
> + },
> + "step8" : {
> + "shortname" : "run AUH for layer meta-xfce",
> + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-xfce"],
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-xfce"
> + ]
> + },
> + "step9" : {
> + "shortname" : "run AUH for layer meta-filesystems",
> + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-filesystems"],
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-filesystems"
> + ]
> + },
> + "step10" : {
> + "shortname" : "run AUH for layer meta-initramfs",
> + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-initramfs"],
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-initramfs"
> + ]
> + },
> + "step11" : {
> + "shortname" : "run AUH for layer meta-webserver",
> + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-webserver"],
> + "EXTRAPLAINCMDS" : [
> + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-webserver"
> + ]
> + }
> },
> "a-quick" : {
> "TEMPLATE" : "trigger-build"
> @@ -1531,6 +1621,12 @@
> "branch" : "master",
> "revision" : "HEAD",
> "no-layer-add" : true
> + },
> + "auto-upgrade-helper" : {
> + "url" : "git://git.yoctoproject.org/auto-upgrade-helper",
> + "branch" : "master",
> + "revision" : "HEAD",
> + "no-layer-add" : true
> }
> },
> "buildtools" : {
> diff --git a/scripts/run-auh b/scripts/run-auh
> index 0419dd1..f59c561 100755
> --- a/scripts/run-auh
> +++ b/scripts/run-auh
> @@ -7,28 +7,41 @@
> # Called with $1 - the directory where the setup was created
>
> if [ -z $1 ]; then
> - echo "Use: $0 auh_setup_dir [publish_dir]"
> + echo "Use: $0 [auh_setup_dir] [publish_dir] [meta_dir] [meta_list]"
> exit 1
> fi
>
> -full_dir=$(readlink -e $1)
> +build_dir=$(readlink -e $1)
>
> -auh_dir=$full_dir/auto-upgrade-helper
> -poky_dir=$full_dir/poky
> -build_dir=$full_dir/build
> -sstate_dir=$full_dir/build/sstate-cache
> +auh_dir=$build_dir/../auto-upgrade-helper
> +poky_dir=$build_dir/../poky
> +sstate_dir=$build_dir/sstate-cache
>
> pushd $poky_dir
> + # Base the upgrades on poky master
> + git fetch origin
> + git checkout -B tmp-auh-upgrades origin/master
> +popd
> +
> +LAYERS_ARGS=''
> +# if layer arguments are provided
> +if [ ! -z $3 ]; then
> + meta_dir=$3
> + meta_layer=$4
> + machine_list="qemux86 qemux86-64 qemuarm qemumips qemuppc qemux86_musl"
> +
> + pushd $meta_dir || exit 1
> + # Base the upgrades on master branch
> + git fetch origin
> + git checkout -B tmp-auh-upgrades origin/main
> + popd
>
> -# Base the upgrades on poky master
> -git fetch origin
> -git checkout -B tmp-auh-upgrades origin/master
> + LAYERS_ARGS="--layer-dir ${meta_dir} --layer-names ${meta_layer} --layer-machines ${machine_list}"
> +fi
>
> -source $poky_dir/oe-init-build-env $build_dir
> -$auh_dir/upgrade-helper.py -e all
> +# execute upgrade-helper script on all recipes
> +$auh_dir/upgrade-helper.py -e ${LAYERS_ARGS} -- all
>
> if [ -n $2 ]; then
> cp -rf $build_dir/upgrade-helper/* $2
> fi
> -
> -popd
> diff --git a/scripts/setup-auh b/scripts/setup-auh
> index d6b83fe..f8bd8fd 100755
> --- a/scripts/setup-auh
> +++ b/scripts/setup-auh
> @@ -4,26 +4,34 @@
> #
> # Initialize Auto Upgrade Helper in a directory.
> #
> -# Called with $1 - the directory to place the setup
> +# Called with $1 - the build directory
> CONFIG_DIR=`dirname $0`/auh-config
>
> if [ -z $1 ]; then
> - echo "Use: $0 target_dir"
> + echo "Use: $0 build_dir"
> exit 1
> fi
>
> -mkdir -p $1
> -pushd $1
> -
> -git clone git://git.yoctoproject.org/poky
> -pushd poky
> +# Setup git repository with name and email to be able to commit
> +# the upgrade patch.
> +# Assume repositories are cloned on ${build_dir}/../
> +pushd $1/../poky
> git config user.email auh@yoctoproject.org
> git config user.name "Auto Upgrade Helper"
> popd
> -git clone git://git.yoctoproject.org/auto-upgrade-helper
> -source poky/oe-init-build-env build
> -mkdir -p upgrade-helper
> -popd
> +# handle the case where meta-openembedded repository is used
> +# todo: detect any layer ?
> +if [ -d $1/../meta-openembedded ]; then
> + pushd $1/../meta-openembedded
> + git config user.email auh@yoctoproject.org
> + git config user.name "Auto Upgrade Helper"
> + popd
> +fi
> +
> +# auto-upgrade-helper doesn't need to be configured.
>
> -cp $CONFIG_DIR/upgrade-helper.conf $1/build/upgrade-helper
> -cat $CONFIG_DIR/local.conf.append >> $1/build/conf/local.conf
> +# create and copy the upgrade helper configuration file
> +mkdir -p $1/upgrade-helper
> +cp $CONFIG_DIR/upgrade-helper.conf $1/upgrade-helper
> +# amend the local.conf
> +cat $CONFIG_DIR/local.conf.append >> $1/conf/local.conf
> --
> 2.39.2
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#61489): https://lists.yoctoproject.org/g/yocto/message/61489
> Mute This Topic: https://lists.yoctoproject.org/mt/102199062/1686489
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
On Thu, 2023-10-26 at 15:53 +0200, Alexander Kanavin wrote: > This looks basically ok, but how can this be tested if needed changes > to AUH are still in review? > > Running auh over meta-oe layers shouldn't be defined as steps that are > run one after the other, but as separate parallelizable jobs, e.g. > "auh-meta-oe" : { > ... > "auh-meta-python" : { > ... > etc. They could use a job template so you only need to set the varying > parameters (config.json has examples). In general you're right, but for AUH and meta-oe, I'm ok with this being done serially given what AUH does. We don't really want 11 workers working on this at the same time? Cheers, Richard
On Thu, 26 Oct 2023 at 18:17, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > In general you're right, but for AUH and meta-oe, I'm ok with this > being done serially given what AUH does. We don't really want 11 > workers working on this at the same time? Yes, I think that's fine for a start, unless it's proven to be unbearably slow (e.g. days). To help with that I just trimmed the default auh targets so that only qemux86-64/glibc and qemuarm/musl remain - that's 2 out of previous 6: - 'qemux86 qemux86-64 qemuarm qemumips qemuppc qemux86_musl').split() + 'qemux86-64 qemuarm_musl').split() Alex
Hi David, On Thu, 2023-10-26 at 15:03 +0200, David Pierret wrote: > - Modify the setup-auh to use repositories cloned by the auto-builder. > Manage the case where meta-openembedded is not used. > > - Modify the run-auh to manage the optional additional layer. > Only 1 layer can be provided. > > - Add auh-meta-openembedded job with 1 step per layer. > > Signed-off-by: David Pierret <david.pierret@smile.fr> > --- > config.json | 102 ++++++++++++++++++++++++++++++++++++++++++++-- > scripts/run-auh | 39 ++++++++++++------ > scripts/setup-auh | 34 ++++++++++------ > 3 files changed, 146 insertions(+), 29 deletions(-) This took a while to review. Firstly, this really should be written as a set of commits with incremental changes. For example, changing the script to use poky from the autobuilder checkout instead of the local clone is one logical change. We could take and review/merge that independently of the other changes. Another logical independent change is using auto-upgrade-helper from the checkout of the autobuilder. That did highlight an issue that the autobuilder does not provide a checkout of that repository. With that and this work in mind I added auh-meta-oe here: https://git.yoctoproject.org/yocto-autobuilder2/commit/?id=47c0119b6e87f920566d2176793ef8d982cf31e2 and then added the auto-upgrade-helper repo here: https://git.yoctoproject.org/yocto-autobuilder2/commit/?id=854a60ce8bf8ce0feec370bc6782cc12aa1e29a5 which then means this patch is closer to working. I had to make some tweaks on top of the patch to make it work: https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?h=master-next&id=673745f473024020ed9efece26462e8c884dc819 where I found that: * poky in the AB context is $1/.. since it is the top level container of the layers * meta-openembedded is located at ${HELPERBUILDDIR}/../meta-openembedded * we still need to source the build env script. The AB could be configured to do that for us but currently isn't. * we need to trim the machine_list down as mentioned * the naming we're using is auh-meta-oe, not auh-meta-openembedded * I've temporarily configured to use the test-list so we don't send lots of mail to all users * layer dependencies were missing in ADDLAYER. I hate having to do that but it is is what it is right now. With that, we get this "successful" build: https://autobuilder.yoctoproject.org/typhoon/#/builders/159/builds/5 which shows the exit code handling isn't great but probably deliberate to avoid autobuilder errors. This gets us to the point where the code in the upgrade helper itself needs the layer support changing. Since I've now had to look into that code, it seems the layer options are entirely missing from the commandline options parser. I'd strongly suggest dropping layer_machines variable and just use the machines value in all cases. "layer-mode" seems pointless as you can determine if we're in "layer mode" if a layer is specified on the commandline (which needs adding). I'm hoping you could use the layer overrides to be able to identify which recipes to upgrade. Cheers, Richard
J Viele Grüße, Matthias —- Von unterwegs gesendet
Hello, Le 28/10/2023 à 10:27, Richard Purdie a écrit : > Hi David, > > On Thu, 2023-10-26 at 15:03 +0200, David Pierret wrote: >> - Modify the setup-auh to use repositories cloned by the auto-builder. >> Manage the case where meta-openembedded is not used. >> >> - Modify the run-auh to manage the optional additional layer. >> Only 1 layer can be provided. >> >> - Add auh-meta-openembedded job with 1 step per layer. >> >> Signed-off-by: David Pierret <david.pierret@smile.fr> >> --- >> config.json | 102 ++++++++++++++++++++++++++++++++++++++++++++-- >> scripts/run-auh | 39 ++++++++++++------ >> scripts/setup-auh | 34 ++++++++++------ >> 3 files changed, 146 insertions(+), 29 deletions(-) > > This took a while to review. Firstly, this really should be written as > a set of commits with incremental changes. > > For example, changing the script to use poky from the autobuilder > checkout instead of the local clone is one logical change. We could > take and review/merge that independently of the other changes. > > Another logical independent change is using auto-upgrade-helper from > the checkout of the autobuilder. That did highlight an issue that the > autobuilder does not provide a checkout of that repository.> > With that and this work in mind I added auh-meta-oe here: > > https://git.yoctoproject.org/yocto-autobuilder2/commit/?id=47c0119b6e87f920566d2176793ef8d982cf31e2 > > and then added the auto-upgrade-helper repo here: > > https://git.yoctoproject.org/yocto-autobuilder2/commit/?id=854a60ce8bf8ce0feec370bc6782cc12aa1e29a5 > > which then means this patch is closer to working. > > I had to make some tweaks on top of the patch to make it work: > > https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?h=master-next&id=673745f473024020ed9efece26462e8c884dc819 > > where I found that: > > * poky in the AB context is $1/.. since it is the top level > container of the layers > * meta-openembedded is located at ${HELPERBUILDDIR}/../meta-openembedded > * we still need to source the build env script. The AB could be > configured to do that for us but currently isn't. > * we need to trim the machine_list down as mentioned > * the naming we're using is auh-meta-oe, not auh-meta-openembedded > * I've temporarily configured to use the test-list so we don't > send lots of mail to all users > * layer dependencies were missing in ADDLAYER. I hate having to do > that but it is is what it is right now I've just sent a reworked version of this patch : https://lists.yoctoproject.org/g/yocto/message/61579 [yocto-autobuilder-helper][PATCH v4 0/2] Extend auto-upgrade helper to meta-openembedded https://lists.yoctoproject.org/g/yocto/message/61580 [yocto-autobuilder-helper][PATCH v4 1/2] config.json: Move AUH from custom scripts to config.json https://lists.yoctoproject.org/g/yocto/message/61581 [yocto-autobuilder-helper][PATCH v4 2/2] config.json: Extend AUH to meta-openembedded layers While moving stuff from the scripts to config.json, I found the scripts to be mostly redundant and choose to remove them. I've took most of RP's tweaks but not the test-list change. I hope this one will be easier to review. > With that, we get this "successful" build: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/159/builds/5 > > which shows the exit code handling isn't great but probably deliberate > to avoid autobuilder errors. > > This gets us to the point where the code in the upgrade helper itself > needs the layer support changing. > > Since I've now had to look into that code, it seems the layer options > are entirely missing from the commandline options parser. I'd strongly > suggest dropping layer_machines variable and just use the machines > value in all cases. > > "layer-mode" seems pointless as you can determine if we're in "layer > mode" if a layer is specified on the commandline (which needs adding). > > I'm hoping you could use the layer overrides to be able to identify > which recipes to upgrade. I did not find how to use layer overrides here... Maybe once AUH is more integrated into oe-core as RP suggested on IRC? Regards,
diff --git a/config.json b/config.json index 3acb710..a168e48 100644 --- a/config.json +++ b/config.json @@ -1416,9 +1416,99 @@ "TEMPLATE" : "buildperf" }, "auh" : { - "EXTRAPLAINCMDS" : [ - "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; ${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/" - ] + "NEEDREPOS" : ["poky", "auto-upgrade-helper"], + "step1" : { + "shortname" : "setup AUH", + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}" + ] + }, + "step2" : { + "shortname" : "run AUH poky", + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/" + ] + } + }, + "auh-meta-openembedded" : { + "NEEDREPOS" : ["poky", "meta-openembedded", "auto-upgrade-helper"], + "step1" : { + "shortname" : "setup AUH", + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}" + ] + }, + "step2" : { + "shortname" : "run AUH for layer meta-oe", + + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-oe"], + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-oe" + ] + }, + "step3" : { + "shortname" : "run AUH for layer meta-python", + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-python"], + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-python" + ] + }, + "step4" : { + "shortname" : "run AUH for layer meta-perl", + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-perl"], + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-perl" + ] + }, + "step5" : { + "shortname" : "run AUH for layer meta-networking", + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-networking"], + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-networking" + ] + }, + "step6" : { + "shortname" : "run AUH for layer meta-multimedia", + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-multimedia"], + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-multimedia" + ] + }, + "step7" : { + "shortname" : "run AUH for layer meta-gnome", + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-gnome"], + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-gnome" + ] + }, + "step8" : { + "shortname" : "run AUH for layer meta-xfce", + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-xfce"], + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-xfce" + ] + }, + "step9" : { + "shortname" : "run AUH for layer meta-filesystems", + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-filesystems"], + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-filesystems" + ] + }, + "step10" : { + "shortname" : "run AUH for layer meta-initramfs", + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-initramfs"], + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-initramfs" + ] + }, + "step11" : { + "shortname" : "run AUH for layer meta-webserver", + "ADDLAYER" : ["${BUILDDIR}/../meta-openembedded/meta-webserver"], + "EXTRAPLAINCMDS" : [ + "${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/ ${HELPERBUILDDIR}/meta-openembedded meta-webserver" + ] + } }, "a-quick" : { "TEMPLATE" : "trigger-build" @@ -1531,6 +1621,12 @@ "branch" : "master", "revision" : "HEAD", "no-layer-add" : true + }, + "auto-upgrade-helper" : { + "url" : "git://git.yoctoproject.org/auto-upgrade-helper", + "branch" : "master", + "revision" : "HEAD", + "no-layer-add" : true } }, "buildtools" : { diff --git a/scripts/run-auh b/scripts/run-auh index 0419dd1..f59c561 100755 --- a/scripts/run-auh +++ b/scripts/run-auh @@ -7,28 +7,41 @@ # Called with $1 - the directory where the setup was created if [ -z $1 ]; then - echo "Use: $0 auh_setup_dir [publish_dir]" + echo "Use: $0 [auh_setup_dir] [publish_dir] [meta_dir] [meta_list]" exit 1 fi -full_dir=$(readlink -e $1) +build_dir=$(readlink -e $1) -auh_dir=$full_dir/auto-upgrade-helper -poky_dir=$full_dir/poky -build_dir=$full_dir/build -sstate_dir=$full_dir/build/sstate-cache +auh_dir=$build_dir/../auto-upgrade-helper +poky_dir=$build_dir/../poky +sstate_dir=$build_dir/sstate-cache pushd $poky_dir + # Base the upgrades on poky master + git fetch origin + git checkout -B tmp-auh-upgrades origin/master +popd + +LAYERS_ARGS='' +# if layer arguments are provided +if [ ! -z $3 ]; then + meta_dir=$3 + meta_layer=$4 + machine_list="qemux86 qemux86-64 qemuarm qemumips qemuppc qemux86_musl" + + pushd $meta_dir || exit 1 + # Base the upgrades on master branch + git fetch origin + git checkout -B tmp-auh-upgrades origin/main + popd -# Base the upgrades on poky master -git fetch origin -git checkout -B tmp-auh-upgrades origin/master + LAYERS_ARGS="--layer-dir ${meta_dir} --layer-names ${meta_layer} --layer-machines ${machine_list}" +fi -source $poky_dir/oe-init-build-env $build_dir -$auh_dir/upgrade-helper.py -e all +# execute upgrade-helper script on all recipes +$auh_dir/upgrade-helper.py -e ${LAYERS_ARGS} -- all if [ -n $2 ]; then cp -rf $build_dir/upgrade-helper/* $2 fi - -popd diff --git a/scripts/setup-auh b/scripts/setup-auh index d6b83fe..f8bd8fd 100755 --- a/scripts/setup-auh +++ b/scripts/setup-auh @@ -4,26 +4,34 @@ # # Initialize Auto Upgrade Helper in a directory. # -# Called with $1 - the directory to place the setup +# Called with $1 - the build directory CONFIG_DIR=`dirname $0`/auh-config if [ -z $1 ]; then - echo "Use: $0 target_dir" + echo "Use: $0 build_dir" exit 1 fi -mkdir -p $1 -pushd $1 - -git clone git://git.yoctoproject.org/poky -pushd poky +# Setup git repository with name and email to be able to commit +# the upgrade patch. +# Assume repositories are cloned on ${build_dir}/../ +pushd $1/../poky git config user.email auh@yoctoproject.org git config user.name "Auto Upgrade Helper" popd -git clone git://git.yoctoproject.org/auto-upgrade-helper -source poky/oe-init-build-env build -mkdir -p upgrade-helper -popd +# handle the case where meta-openembedded repository is used +# todo: detect any layer ? +if [ -d $1/../meta-openembedded ]; then + pushd $1/../meta-openembedded + git config user.email auh@yoctoproject.org + git config user.name "Auto Upgrade Helper" + popd +fi + +# auto-upgrade-helper doesn't need to be configured. -cp $CONFIG_DIR/upgrade-helper.conf $1/build/upgrade-helper -cat $CONFIG_DIR/local.conf.append >> $1/build/conf/local.conf +# create and copy the upgrade helper configuration file +mkdir -p $1/upgrade-helper +cp $CONFIG_DIR/upgrade-helper.conf $1/upgrade-helper +# amend the local.conf +cat $CONFIG_DIR/local.conf.append >> $1/conf/local.conf
- Modify the setup-auh to use repositories cloned by the auto-builder. Manage the case where meta-openembedded is not used. - Modify the run-auh to manage the optional additional layer. Only 1 layer can be provided. - Add auh-meta-openembedded job with 1 step per layer. Signed-off-by: David Pierret <david.pierret@smile.fr> --- config.json | 102 ++++++++++++++++++++++++++++++++++++++++++++-- scripts/run-auh | 39 ++++++++++++------ scripts/setup-auh | 34 ++++++++++------ 3 files changed, 146 insertions(+), 29 deletions(-)