[v2] oeqa/runtime/cases: new image_upgrade test

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

New oe-selftest and associated "testimage" test to check that generated
package feeds can be used to update the latest image built
by the Yocto Project autobuilder.

Currently, only the "core-image-full-cmdline" image with IPK packages
and the "poky-altcfg" distro is tested.

Test it by running:
oe-selftest -r image_upgrade

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Suggested-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CC: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
CC: Bruce Ashfield <bruce.ashfield@gmail.com>
CC: Alexander Kanavin <alex.kanavin@gmail.com>

---
Changes in V2:

- Move meta/lib/oeqa/runtime/cases/opkg_sysupgrade.py
  to meta-selftest/lib/oeqa/runtime/cases/ipk_sysupgrade.py
  (notice that "ipk" is a new prefix, to match PACKAGE_CLASSES values)
  as it needs special setup via selftest, and not useful in standalone
  '-c testimage' runs.
  Suggested by Alexander Kanavin.

- Implement a more generic run_image_upgrade_test() which can be
  given any image type, image file path, image download URL,
  any PACKAGE_CLASSES value, and specific features.
  This way, the function running the tests is not supposed to be
  Yocto and Poky specific.

- Tested on the latest master against yocto-5.0_M3

Interested in further (specific) guidelines for making this test
more generic, depending on the environments calling it and their
configurations.
---
 .../lib/oeqa/runtime/cases/ipk_sysupgrade.py  |  68 +++++++++
 meta/lib/oeqa/selftest/cases/image_upgrade.py | 134 ++++++++++++++++++
 2 files changed, 202 insertions(+)
 create mode 100644 meta-selftest/lib/oeqa/runtime/cases/ipk_sysupgrade.py
 create mode 100755 meta/lib/oeqa/selftest/cases/image_upgrade.py

On Mon, 29 Apr 2024 at 17:22, <michael.opdenacker@bootlin.com> wrote:
> +# Version string utilities copied from yocto-autobuilder-helper/scripts/utils.py
> +
> +def get_string_from_version(version, milestone=None, rc=None):
> +    """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists,
> +    those are major releases
> +    """
> +    if len(version) == 3 and version[-1] == 0:
> +        version = version[:-1]
> +
> +    result = ".".join(list(map(str, version)))
> +    if milestone:
> +        result += "_M" + str(milestone)
> +    if rc:
> +        result += ".rc" + str(rc)
> +    return result
> +
> +def get_tag_from_version(version, milestone):
> +    if not milestone:
> +        return "yocto-" + get_string_from_version(version, milestone)
> +    return get_string_from_version(version, milestone)
> +
> +def get_version_from_string(raw_version):
> +    """ Get version as list of int from raw_version.
> +
> +    Raw version _can_ be prefixed by "yocto-",
> +    Raw version _can_ be suffixed by "_MX"
> +    Raw version _can_ be suffixed by ".rcY"
> +    """
> +    version = None
> +    milestone = None
> +    rc = None
> +    if raw_version[:6] == "yocto-":
> +        raw_version = raw_version[6:]
> +    raw_version = raw_version.split(".")
> +    if raw_version[-1][:2] == "rc":
> +        rc = int(raw_version[-1][-1])
> +        raw_version = raw_version[:-1]
> +    if raw_version[-1][-3:-1] == "_M":
> +        milestone = int(raw_version[-1][-1])
> +        raw_version = raw_version[:-1] + [raw_version[-1][:-3]]
> +    version = list(map(int, raw_version))
> +    """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists,
> +    those are major releases
> +    """
> +    if len(version) == 3 and version[-1] == 0:
> +        version = version[:-1]
> +    return version, milestone, rc

I understand that the above is merely a copy-paste, but these
functions would really benefit from having comments that contain
examples of input and output. I don't understand what they do.

> +def get_poky_latest_image_url(machine, machine_variant, image_file):
> +
> +    """Returns the URL of the latest generated image for the current branch"""
> +
> +    baseversion, milestone, _ = get_version_from_string(subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=basepath).decode('utf-8').strip())
> +    tag = get_tag_from_version(baseversion, milestone)
> +    downloads_base = "https://downloads.yoctoproject.org/releases/yocto/"
> +
> +    if milestone is not None:
> +        downloads_base += "milestones/yocto-%s" % tag
> +    else:
> +        downloads_base += tag
> +
> +    if machine_variant == "":
> +        subdir = machine
> +    else:
> +        subdir = "%s-%s" % (machine, machine_variant)
> +
> +    return "%s/machines/qemu/%s/%s" % (downloads_base, subdir, image_file)

Same here.  A comment with example input and output would help a lot.

> +class ImageIpkUpgrade(OESelftestTestCase):
> +
> +    def run_image_upgrade_test(self, image, image_path, image_url, pkg, features):
> +        """
> +        Summary: Test that generated packages can
> +        be used to upgrade an older image version.
> +        This is done by generating an image but then replacing it
> +        by an older image available at a specified location.
> +        We then run QEMU on the old image and replace the original
> +        original package feeds by our own.
> +        """
> +
> +        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
> +        features += 'PACKAGE_CLASSES = "package_%s"\n' % pkg
> +        features += 'IMAGE_CLASSES += "testimage"\n'
> +        features += 'TEST_SUITES="%s_sysupgrade"\n' % pkg
> +        self.write_config(features)
> +
> +        # Need to build a full image to build the .json file needed by QEMU.
> +        # Therefore, it is not sufficient to run only "package_write_ipk" for the image.
> +
> +        self.logger.info("Generating '%s' and package index for latest commit in this branch..." % image)
> +        bitbake(image)
> +        bitbake('package-index')
> +
> +        # Download previously generated image
> +
> +        image_full_path = '%s/%s' % (self.builddir, image_path)
> +
> +        os.remove(image_full_path)
> +        self.logger.info("Downloading image: %s..." % image_url)
> +        cmd = 'wget -O %s %s' % (image_full_path, image_url)
> +        result = runCmd(cmd)
> +        self.assertEqual(0, result.status, cmd + ' returned a non 0 status: %s' % result.output)
> +
> +        # Now run the upgrade tests on the old image
> +
> +        self.logger.info("Running upgrade tests on the downloaded image, using the package feeds generated here...")
> +        bitbake(image + ' -c testimage')
> +
> +    def run_poky_image_upgrade_test(self, distro_variant, image, machine_variant, fstype, pkg):
> +        machine = get_bb_var("MACHINE")
> +        features = 'DISTRO = "poky%s"\n' % distro_variant
> +        image_file = '%s-%s.rootfs.%s' % (image, machine, fstype)
> +        image_path = 'tmp/deploy/images/%s/%s' % (machine, image_file)
> +        image_url = get_poky_latest_image_url(machine, machine_variant, image_file)
> +        self.run_image_upgrade_test(image, image_path, image_url, pkg, features)
> +
> +    @skipIfNotQemu()
> +    def test_poky_altcfg_core_image_ext4_full_cmdline_ipk_upgrade(self):
> +        self.run_poky_image_upgrade_test("-altcfg", "core-image-full-cmdline", "alt", "ext4", "ipk")

This is a much improved code structure that can scale easily, right? :)

Alex

On Mon, 2024-04-29 at 17:55 +0200, Alexander Kanavin wrote:
> On Mon, 29 Apr 2024 at 17:22, <michael.opdenacker@bootlin.com> wrote:
> > +# Version string utilities copied from yocto-autobuilder-helper/scripts/utils.py
> > +
> > +def get_string_from_version(version, milestone=None, rc=None):
> > +    """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists,
> > +    those are major releases
> > +    """
> > +    if len(version) == 3 and version[-1] == 0:
> > +        version = version[:-1]
> > +
> > +    result = ".".join(list(map(str, version)))
> > +    if milestone:
> > +        result += "_M" + str(milestone)
> > +    if rc:
> > +        result += ".rc" + str(rc)
> > +    return result
> > +
> > +def get_tag_from_version(version, milestone):
> > +    if not milestone:
> > +        return "yocto-" + get_string_from_version(version, milestone)
> > +    return get_string_from_version(version, milestone)
> > +
> > +def get_version_from_string(raw_version):
> > +    """ Get version as list of int from raw_version.
> > +
> > +    Raw version _can_ be prefixed by "yocto-",
> > +    Raw version _can_ be suffixed by "_MX"
> > +    Raw version _can_ be suffixed by ".rcY"
> > +    """
> > +    version = None
> > +    milestone = None
> > +    rc = None
> > +    if raw_version[:6] == "yocto-":
> > +        raw_version = raw_version[6:]
> > +    raw_version = raw_version.split(".")
> > +    if raw_version[-1][:2] == "rc":
> > +        rc = int(raw_version[-1][-1])
> > +        raw_version = raw_version[:-1]
> > +    if raw_version[-1][-3:-1] == "_M":
> > +        milestone = int(raw_version[-1][-1])
> > +        raw_version = raw_version[:-1] + [raw_version[-1][:-3]]
> > +    version = list(map(int, raw_version))
> > +    """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists,
> > +    those are major releases
> > +    """
> > +    if len(version) == 3 and version[-1] == 0:
> > +        version = version[:-1]
> > +    return version, milestone, rc
> 
> I understand that the above is merely a copy-paste, but these
> functions would really benefit from having comments that contain
> examples of input and output. I don't understand what they do.

They're coming from yocto-autobuilder-helper and there are even unit
tests for these functions there :/.


> > +def get_poky_latest_image_url(machine, machine_variant, image_file):
> > +
> > +    """Returns the URL of the latest generated image for the current branch"""
> > +
> > +    baseversion, milestone, _ = get_version_from_string(subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=basepath).decode('utf-8').strip())
> > +    tag = get_tag_from_version(baseversion, milestone)
> > +    downloads_base = "https://downloads.yoctoproject.org/releases/yocto/"
> > +
> > +    if milestone is not None:
> > +        downloads_base += "milestones/yocto-%s" % tag
> > +    else:
> > +        downloads_base += tag
> > +
> > +    if machine_variant == "":
> > +        subdir = machine
> > +    else:
> > +        subdir = "%s-%s" % (machine, machine_variant)
> > +
> > +    return "%s/machines/qemu/%s/%s" % (downloads_base, subdir, image_file)
> 
> Same here.  A comment with example input and output would help a lot.
> 
> > +class ImageIpkUpgrade(OESelftestTestCase):
> > +
> > +    def run_image_upgrade_test(self, image, image_path, image_url, pkg, features):
> > +        """
> > +        Summary: Test that generated packages can
> > +        be used to upgrade an older image version.
> > +        This is done by generating an image but then replacing it
> > +        by an older image available at a specified location.
> > +        We then run QEMU on the old image and replace the original
> > +        original package feeds by our own.
> > +        """
> > +
> > +        features += 'EXTRA_IMAGE_FEATURES += "package-management"\n'
> > +        features += 'PACKAGE_CLASSES = "package_%s"\n' % pkg
> > +        features += 'IMAGE_CLASSES += "testimage"\n'
> > +        features += 'TEST_SUITES="%s_sysupgrade"\n' % pkg
> > +        self.write_config(features)
> > +
> > +        # Need to build a full image to build the .json file needed by QEMU.
> > +        # Therefore, it is not sufficient to run only "package_write_ipk" for the image.
> > +
> > +        self.logger.info("Generating '%s' and package index for latest commit in this branch..." % image)
> > +        bitbake(image)
> > +        bitbake('package-index')
> > +
> > +        # Download previously generated image
> > +
> > +        image_full_path = '%s/%s' % (self.builddir, image_path)
> > +
> > +        os.remove(image_full_path)
> > +        self.logger.info("Downloading image: %s..." % image_url)
> > +        cmd = 'wget -O %s %s' % (image_full_path, image_url)
> > +        result = runCmd(cmd)
> > +        self.assertEqual(0, result.status, cmd + ' returned a non 0 status: %s' % result.output)
> > +
> > +        # Now run the upgrade tests on the old image
> > +
> > +        self.logger.info("Running upgrade tests on the downloaded image, using the package feeds generated here...")
> > +        bitbake(image + ' -c testimage')
> > +
> > +    def run_poky_image_upgrade_test(self, distro_variant, image, machine_variant, fstype, pkg):
> > +        machine = get_bb_var("MACHINE")
> > +        features = 'DISTRO = "poky%s"\n' % distro_variant
> > +        image_file = '%s-%s.rootfs.%s' % (image, machine, fstype)
> > +        image_path = 'tmp/deploy/images/%s/%s' % (machine, image_file)
> > +        image_url = get_poky_latest_image_url(machine, machine_variant, image_file)
> > +        self.run_image_upgrade_test(image, image_path, image_url, pkg, features)
> > +
> > +    @skipIfNotQemu()
> > +    def test_poky_altcfg_core_image_ext4_full_cmdline_ipk_upgrade(self):
> > +        self.run_poky_image_upgrade_test("-altcfg", "core-image-full-cmdline", "alt", "ext4", "ipk")
> 
> This is a much improved code structure that can scale easily, right? :)

Its getting better but the last bit of this 
(run_poky_image_upgrade_test and
test_poky_altcfg_core_image_ext4_full_cmdline_ipk_upgrade) should be in
meta-yocto/meta-poky/lib/oeqa/selftest.

Cheers,

Richard