Message ID | 20231010144940.3811241-1-david.pierret@smile.fr |
---|---|
State | New |
Headers | show |
Series | [yocto-autobuilder-helper] oe-mirror: Add job to mirror openembedded sources | expand |
On Tue, 2023-10-10 at 16:49 +0200, David Pierret wrote: > - Add a job that call run-fetchuniverse > - Add script run-fetchuniverse > - loop over each LTS branches > - Download all the meta-openembedded layers sources. > - Call to oe-selftest test_yocto_source_mirror after fetching. > We are not using a second step to keep checkout consistency with potential merge. > > Signed-off-by: David Pierret <david.pierret@smile.fr> > Reviewed-by: Yoann Congal <yoann.congal@smile.fr> > --- > config.json | 13 +++++++++++ > scripts/run-fetchuniverse | 49 +++++++++++++++++++++++++++++++++++++++ > 2 files changed, 62 insertions(+) > create mode 100755 scripts/run-fetchuniverse > > diff --git a/config.json b/config.json > index 3acb710..8d59be8 100644 > --- a/config.json > +++ b/config.json > @@ -1420,6 +1420,19 @@ > "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; ${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/" > ] > }, > + "oe-mirror" : { This needs to be called something like meta-oe-mirror to differentiate it from other "OE" things. > + "SDKMACHINE" : "x86_64", > + "MACHINE" : "qemux86-64", > + "NEEDREPOS" : ["poky", "meta-openembedded"], > + > + "step1" : { > + "shortname" : "Sources pre-fetching", > + "EXTRAPLAINCMDS" : ["${SCRIPTSDIR}/run-fetchuniverse ${BUILDDIR} ${SCRIPTSDIR}"], > + "extravars" : [ > + "SOURCE_MIRROR_FETCH = '1'" > + ] > + } > + }, > "a-quick" : { > "TEMPLATE" : "trigger-build" > }, > diff --git a/scripts/run-fetchuniverse b/scripts/run-fetchuniverse > new file mode 100755 > index 0000000..9ef82de > --- /dev/null > +++ b/scripts/run-fetchuniverse > @@ -0,0 +1,49 @@ > +#!/bin/bash > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > +BUILDDIR=`realpath $1` > +SCRIPTSDIR=`realpath $2` > +ROOTDIR=$BUILDDIR/.. > + > +# > +# Fetch mirror for each LTS branch and test integrity > +# > +for branch in master mickledore langdale kirkstone dunfell; do Rather than iterate here, we should add a configuration to each branch of -helper where we want to enable this. > + # For each layer > + for repo in bitbake poky meta-openembedded ; do > + pushd $ROOTDIR/$repo || exit 1 > + # reset branch > + git reset origin/$branch --hard > + popd || exit 1 > + done > + # remove config to be re-generated by oe-init-build-env > + rm conf/local.conf > + rm conf/bblayers.conf > + rm -f conf/templateconf.cfg > + rm tmp/ -rf > + > + # got back to ROOTDIR > + pushd $ROOTDIR || exit 1 > + # generate config for current branch > + . ./poky/oe-init-build-env build > + > + # We need to add the meta-openembedded layers before fetch > + bitbake-layers add-layer ${BUILDDIR}/../meta-openembedded/meta-filesystems ${BUILDDIR}/../meta-openembedded/meta-gnome \ > + ${BUILDDIR}/../meta-openembedded/meta-initramfs ${BUILDDIR}/../meta-openembedded/meta-multimedia \ > + ${BUILDDIR}/../meta-openembedded/meta-networking ${BUILDDIR}/../meta-openembedded/meta-oe \ > + ${BUILDDIR}/../meta-openembedded/meta-perl ${BUILDDIR}/../meta-openembedded/meta-python \ > + ${BUILDDIR}/../meta-openembedded/meta-webserver ${BUILDDIR}/../meta-openembedded/meta-xfce Then hopefully we can massively simplify the script! Cheers, Richard > + # call bitbake to fetch universe > + bitbake universe -c fetch -k > + > + # call oe-selftest to execute test_yocto_source_mirror > + # We need the selftest layer > + bitbake-layers add-layer $BUILDDIR/../meta-selftest > + # ensure no previous build-st directory exist > + rm -rf ${BUILDDIR}/../build-st > + ${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror || exit 1 > + popd || exit 1 > +done > + > + > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#61306): https://lists.yoctoproject.org/g/yocto/message/61306 > Mute This Topic: https://lists.yoctoproject.org/mt/101876565/1686473 > Group Owner: yocto+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [richard.purdie@linuxfoundation.org] > -=-=-=-=-=-=-=-=-=-=-=- >
diff --git a/config.json b/config.json index 3acb710..8d59be8 100644 --- a/config.json +++ b/config.json @@ -1420,6 +1420,19 @@ "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; ${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/" ] }, + "oe-mirror" : { + "SDKMACHINE" : "x86_64", + "MACHINE" : "qemux86-64", + "NEEDREPOS" : ["poky", "meta-openembedded"], + + "step1" : { + "shortname" : "Sources pre-fetching", + "EXTRAPLAINCMDS" : ["${SCRIPTSDIR}/run-fetchuniverse ${BUILDDIR} ${SCRIPTSDIR}"], + "extravars" : [ + "SOURCE_MIRROR_FETCH = '1'" + ] + } + }, "a-quick" : { "TEMPLATE" : "trigger-build" }, diff --git a/scripts/run-fetchuniverse b/scripts/run-fetchuniverse new file mode 100755 index 0000000..9ef82de --- /dev/null +++ b/scripts/run-fetchuniverse @@ -0,0 +1,49 @@ +#!/bin/bash +# +# SPDX-License-Identifier: GPL-2.0-only +# +BUILDDIR=`realpath $1` +SCRIPTSDIR=`realpath $2` +ROOTDIR=$BUILDDIR/.. + +# +# Fetch mirror for each LTS branch and test integrity +# +for branch in master mickledore langdale kirkstone dunfell; do + # For each layer + for repo in bitbake poky meta-openembedded ; do + pushd $ROOTDIR/$repo || exit 1 + # reset branch + git reset origin/$branch --hard + popd || exit 1 + done + # remove config to be re-generated by oe-init-build-env + rm conf/local.conf + rm conf/bblayers.conf + rm -f conf/templateconf.cfg + rm tmp/ -rf + + # got back to ROOTDIR + pushd $ROOTDIR || exit 1 + # generate config for current branch + . ./poky/oe-init-build-env build + + # We need to add the meta-openembedded layers before fetch + bitbake-layers add-layer ${BUILDDIR}/../meta-openembedded/meta-filesystems ${BUILDDIR}/../meta-openembedded/meta-gnome \ + ${BUILDDIR}/../meta-openembedded/meta-initramfs ${BUILDDIR}/../meta-openembedded/meta-multimedia \ + ${BUILDDIR}/../meta-openembedded/meta-networking ${BUILDDIR}/../meta-openembedded/meta-oe \ + ${BUILDDIR}/../meta-openembedded/meta-perl ${BUILDDIR}/../meta-openembedded/meta-python \ + ${BUILDDIR}/../meta-openembedded/meta-webserver ${BUILDDIR}/../meta-openembedded/meta-xfce + # call bitbake to fetch universe + bitbake universe -c fetch -k + + # call oe-selftest to execute test_yocto_source_mirror + # We need the selftest layer + bitbake-layers add-layer $BUILDDIR/../meta-selftest + # ensure no previous build-st directory exist + rm -rf ${BUILDDIR}/../build-st + ${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror || exit 1 + popd || exit 1 +done + +