From patchwork Tue Nov 10 19:14:04 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jon Mason X-Patchwork-Id: 103604 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.392.1794338048516803905 for ; Tue, 10 Nov 2026 11:14:09 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@arm.com header.s=foss header.b=NvcGInI/; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: jon.mason@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9EC741477 for ; Tue, 10 Nov 2026 11:14:04 -0800 (PST) Received: from H24V3P4C17.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 733B03F673 for ; Tue, 10 Nov 2026 11:14:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1794338047; bh=W/tzz4k/Ta5x3IpoQtVExA252sIMEGzecBjWETqZRm8=; h=From:To:Subject:Date:From; b=NvcGInI//5u6BwQ80dALn+O0AWq0znalmYUyaMddC/ZRhBnW2f4fCaZuJiw7cH3tI XA1PRSKtYSVfhHwmcXMpyBROhGH3frM5isGZQMS8tQguJg1qgbHJaX0774WbZZzngP xhhoA93OsirZtJIGAFrS/O15C/wRq+1pyX8ERUU8= From: Jon Mason To: meta-arm@lists.yoctoproject.org Subject: [PATCH 1/3] scripts: add release management script Date: Tue, 10 Nov 2026 14:14:04 -0500 Message-ID: <20261110191406.48308-1-jon.mason@arm.com> X-Mailer: git-send-email 2.54.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Tue, 10 Nov 2026 19:14:12 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/7207 Add a Python script to simplify and consolidate the meta-arm release process, replacing several previously used standalone Bash scripts. The script supports creating GPG-signed Yocto release tags, updating existing release branches when new commits are available, and creating platform-specific release tags. Platform tags use the existing naming convention and support specifying a release date independently of when the tag is created. Release tags can be published to the upstream repository, with support for publishing individual tags or automatically identifying unpublished releases. Existing upstream tags are checked to prevent accidental overwrites. Release announcements can be generated automatically, including commit counts, changelogs, and LTS support information. Announcements can be reviewed and edited using the user’s configured editor before being sent via sendmail or mutt. A dry-run mode is enabled by default, allowing release operations to be reviewed before execution. Additional checks verify the release environment, and a unit test suite validates the release management logic. This implementation is based on original Bash scripts written by me for managing meta-arm and platform releases. ChatGPT was used to consolidate and refactor the existing functionality, translate it to Python, and assist with additional features and testing. AI-Generated: Uses OpenAI ChatGPT (GPT-6) Signed-off-by: Jon Mason --- scripts/meta-arm-release.py | 731 +++++++++++++++++++++++++++++++ scripts/test_meta_arm_release.py | 220 ++++++++++ 2 files changed, 951 insertions(+) create mode 100644 scripts/meta-arm-release.py create mode 100644 scripts/test_meta_arm_release.py diff --git a/scripts/meta-arm-release.py b/scripts/meta-arm-release.py new file mode 100644 index 000000000000..518da03fc168 --- /dev/null +++ b/scripts/meta-arm-release.py @@ -0,0 +1,731 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: 2026 Arm Limited +# SPDX-License-Identifier: MIT +"""Create, publish and announce meta-arm release tags.""" + +import argparse +from datetime import date, datetime +import os +import re +import shlex +import shutil +import subprocess +import sys +import tempfile +from dataclasses import dataclass +from email.message import EmailMessage +from pathlib import Path + +SUPPORT_URL = 'https://docs.yoctoproject.org/ref-manual/release-process.html' +VERSION_RE = re.compile(r'^(\d+)\.(\d+)(?:\.(\d+))?$') +PLATFORM_TAG_RE = re.compile(r'^[A-Z][A-Z0-9_-]*-[0-9]{4}\.[0-9]{2}\.[0-9]{2}$') +DEFAULT_LTS = ('scarthgap', 'wrynose') + + +class ReleaseError(Exception): + pass + + +def run(*argv, check=True): + result = subprocess.run(argv, text=True, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + if check and result.returncode: + raise ReleaseError(f"{' '.join(argv)}: {result.stderr.strip()}") + return result + + +def git(*args, check=True): + return run('git', *args, check=check) + + +def series_number(version): + match = VERSION_RE.fullmatch(version) + if not match or match.group(3) is not None or int(match.group(2)) > 3: + raise ReleaseError(f'Invalid initial release version: {version}') + return int(match.group(1)) * 4 + int(match.group(2)) + + +def series_from_number(number): + return f'{number // 4}.{number % 4}' + + +def previous_series(version): + number = series_number(version) + if number == 0: + raise ReleaseError('No previous release version') + return series_from_number(number - 1) + + +def next_patch(version): + match = VERSION_RE.fullmatch(version) + if not match: + raise ReleaseError(f'Invalid version: {version}') + return f'{match.group(1)}.{match.group(2)}.{int(match.group(3) or 0) + 1}' + + +def platform_tag(name, release_date=None): + """Build a platform tag, validating both platform name and calendar date.""" + if not re.fullmatch(r'[A-Za-z][A-Za-z0-9_-]*', name): + raise ReleaseError(f'Invalid platform name: {name}') + if release_date is None: + release_date = date.today().strftime('%Y.%m.%d') + try: + datetime.strptime(release_date, '%Y.%m.%d') + except ValueError as exc: + raise ReleaseError('--date must be a valid YYYY.MM.DD date') from exc + if not re.fullmatch(r'\d{4}\.\d{2}\.\d{2}', release_date): + raise ReleaseError('--date must use YYYY.MM.DD') + return f'{name.upper()}-{release_date}' + + +@dataclass +class BranchStatus: + branch: str + series: str + last_tag: str | None + commits: int | None + next_tag: str + + +class ReleaseManager: + def __init__(self, remote='yoctoproject', recipient='meta-arm@lists.yoctoproject.org', + lts=DEFAULT_LTS, execute=False, edit=False, as_of=None, force_lts=False, mailer='auto'): + self.remote = remote + self.recipient = recipient + self.lts = tuple(lts) + self.execute = execute + self.edit = edit + self.as_of = as_of + self.force_lts = force_lts + if mailer not in ('auto', 'sendmail', 'mutt'): + raise ReleaseError(f'Invalid mailer: {mailer}') + self.mailer = mailer + self._branches = None + self._series = None + + def branch_ref(self, branch): + return f'refs/remotes/{self.remote}/{branch}' + + def branches(self): + if self._branches is None: + refs = git('for-each-ref', '--format=%(refname:strip=3)', + f'refs/remotes/{self.remote}/').stdout.splitlines() + self._branches = sorted(b for b in refs if b not in ('master', 'testing') and '/' not in b) + if not self._branches: + raise ReleaseError(f'No release branches under {self.remote}') + return self._branches + + def require_branch(self, branch): + if branch not in self.branches(): + raise ReleaseError(f'Unknown or excluded upstream branch: {branch}') + + @staticmethod + def ancestor(older, newer): + return git('merge-base', '--is-ancestor', older, newer, check=False).returncode == 0 + + @staticmethod + def tag_commit(tag): + result = git('rev-parse', '--verify', '-q', f'refs/tags/{tag}^{{commit}}', check=False) + return result.stdout.strip() if result.returncode == 0 else None + + @staticmethod + def tag_object(tag): + result = git('rev-parse', '--verify', '-q', f'refs/tags/{tag}', check=False) + return result.stdout.strip() if result.returncode == 0 else None + + def numeric_tags(self): + return [tag for tag in git('tag', '-l').stdout.splitlines() if VERSION_RE.fullmatch(tag)] + + def discover_series(self): + """Use tags that lie on a branch, not inherited tags on its fork history. + + A branch's initial release can be at the fork itself; in that case, + resolve shared tags by proximity and fail closed if ambiguous. + """ + if self._series is not None: + return self._series + branches = self.branches() + tags = self.numeric_tags() + forks = {} + for branch in branches: + result = git('merge-base', self.branch_ref(branch), self.branch_ref('master')) + forks[branch] = result.stdout.strip() + + # For every series, assign it to the branch whose tip is nearest + # to its latest reachable tag. Ties cannot establish ownership. + candidates = {} + for tag in tags: + match = VERSION_RE.fullmatch(tag) + series = f'{match.group(1)}.{match.group(2)}' + for branch in branches: + if not self.ancestor(f'refs/tags/{tag}', self.branch_ref(branch)): + continue + distance = int(git('rev-list', '--count', + f'{tag}..{self.branch_ref(branch)}').stdout) + current = candidates.setdefault(series, {}) + if branch not in current or distance < current[branch][0] or ( + distance == current[branch][0] and + (int(match.group(3) or 0) > current[branch][1])): + current[branch] = (distance, int(match.group(3) or 0), tag) + + owners = {} + for series, choices in candidates.items(): + ranked = sorted((v[0], -v[1], b) for b, v in choices.items()) + if not ranked: + continue + # If multiple branches are tied, avoid guessing ownership. + if len(ranked) > 1 and ranked[0][:2] == ranked[1][:2]: + raise ReleaseError(f'Ambiguous ownership for {series}: {ranked[0][2]}, {ranked[1][2]}') + owners[series] = ranked[0][2] + + by_branch = {} + for series, branch in owners.items(): + if branch in by_branch: + raise ReleaseError(f'Multiple version series appear to belong to {branch}') + by_branch[branch] = series + + if not by_branch: + raise ReleaseError('No release version anchors found') + + # Infer only unreleased branches, after the highest established series. + missing = [b for b in branches if b not in by_branch] + if missing: + latest = max(series_number(s) for s in by_branch.values()) + ordered = sorted(missing, key=lambda b: int(git('show', '-s', '--format=%ct', forks[b]).stdout)) + for earlier, later in zip(ordered, ordered[1:]): + if not self.ancestor(forks[earlier], forks[later]) or forks[earlier] == forks[later]: + raise ReleaseError(f'Cannot order unreleased branches: {earlier}, {later}') + for branch in ordered: + latest += 1 + by_branch[branch] = series_from_number(latest) + self._series = by_branch + return by_branch + + def latest_tag(self, branch): + series = self.discover_series()[branch] + choices = [] + for tag in self.numeric_tags(): + match = VERSION_RE.fullmatch(tag) + if f'{match.group(1)}.{match.group(2)}' != series: + continue + if self.ancestor(f'refs/tags/{tag}', self.branch_ref(branch)): + choices.append((int(match.group(3) or 0), tag)) + return max(choices)[1] if choices else None + + def status(self, branch): + self.require_branch(branch) + series = self.discover_series()[branch] + last = self.latest_tag(branch) + if not last: + return BranchStatus(branch, series, None, None, f'Initial {series}') + count = int(git('rev-list', '--count', f'{last}..{self.branch_ref(branch)}').stdout) + return BranchStatus(branch, series, last, count, next_patch(last) if count else '-') + + def show_status(self, branches): + print(f'{"Branch":<18} {"Version":<9} {"Last-tag":<14} {"Commits":<10} Next-tag') + for branch in branches: + state = self.status(branch) + print(f'{branch:<18} {state.series:<9} {state.last_tag or "(none)":<14} ' + f'{str(state.commits) if state.commits is not None else "-":<10} {state.next_tag}') + + def create_tags(self, branch, version, ref, kind): + self.require_branch(branch) + if not VERSION_RE.fullmatch(version): + raise ReleaseError(f'Invalid tag version: {version}') + target = git('rev-parse', '--verify', f'{ref}^{{commit}}').stdout.strip() + if not self.ancestor(target, self.branch_ref(branch)): + raise ReleaseError(f'{ref} is not on {branch}') + for tag in (version, f'yocto-{version}'): + old = self.tag_commit(tag) + if old and old != target: + raise ReleaseError(f'{tag} already points to another commit') + print(f'{branch}: {version} -> {target[:12]} ({kind})') + for tag in (version, f'yocto-{version}'): + if self.tag_object(tag): + print(f' Exists: {tag}') + elif self.execute: + git('tag', '-s', tag, '-m', f'{kind} for Yocto Project {version} ({branch})', target) + print(f' Created: {tag}') + else: + print(f' DRY RUN: create {tag}') + self._series = None + + def release(self, branch, version): + self.require_branch(branch) + expected = self.discover_series()[branch] + if version != expected: + raise ReleaseError(f'Expected {expected} for {branch}, got {version}') + ref = self.branch_ref(branch) + if self.as_of: + ref = git('rev-list', '--first-parent', '--before', f'{self.as_of} 23:59:59', + '-1', ref).stdout.strip() + if not ref: + raise ReleaseError(f'No commit on {branch} before {self.as_of}') + self.create_tags(branch, version, ref, 'Initial release') + + def update(self, branches): + created = [] + for branch in branches: + state = self.status(branch) + if not state.last_tag: + print(f'{branch}: initial release {state.series} missing; skipping') + elif not state.commits: + print(f'{branch}: no new commits; skipping') + else: + self.create_tags(branch, state.next_tag, self.branch_ref(branch), 'Update') + created.append((branch, state.next_tag)) + return created + + def platform(self, name, ref, release_date=None): + """Create a signed platform tag without requiring release branch discovery.""" + tag = platform_tag(name, release_date) + target = git('rev-parse', '--verify', f'{ref}^{{commit}}').stdout.strip() + existing = self.tag_object(tag) + if existing: + if self.tag_commit(tag) != target: + raise ReleaseError(f'{tag} already exists at another commit') + print(f'Platform tag already exists: {tag} -> {target[:12]}') + return tag + print(f'Platform: {name}\nCommit: {target[:12]}\nTag: {tag}') + if self.execute: + message = f'{name.upper()} platform release ({tag.rsplit("-", 1)[1]})\n\nRelease of the {name} platform at commit {target}.' + git('tag', '-s', tag, '-m', message, target) + print(f'Created signed tag: {tag}') + else: + print(f'DRY RUN: Would create signed tag {tag}.') + return tag + + def tags_for(self, branches): + series_map = self.discover_series() + selected = branches or self.branches() + result = set() + for branch in selected: + self.require_branch(branch) + series = series_map[branch] + for tag in self.numeric_tags(): + match = VERSION_RE.fullmatch(tag) + if f'{match.group(1)}.{match.group(2)}' != series: + continue + if not self.ancestor(f'refs/tags/{tag}', self.branch_ref(branch)): + continue + paired = f'yocto-{tag}' + if not self.tag_commit(paired) or self.tag_commit(paired) != self.tag_commit(tag): + raise ReleaseError(f'Missing or mismatched paired tag: {paired}') + result.update((tag, paired)) + return sorted(result) + + def push(self, selections=(), all_tags=False): + """Push selected tags, or all recognized release tags, without overwrites. + + For compatibility, a selection may be an existing release branch. + Explicit tag names need not match the recognized tag patterns. + """ + if all_tags: + tags = git('tag', '-l').stdout.splitlines() + elif selections: + tags = set() + for selection in selections: + if self.tag_object(selection): + tags.add(selection) + elif selection in self.branches(): + tags.update(self.tags_for((selection,))) + else: + raise ReleaseError(f'Unknown tag or release branch: {selection}') + tags = sorted(tags) + else: + tags = [t for t in git('tag', '-l').stdout.splitlines() + if VERSION_RE.fullmatch(t) + or (t.startswith('yocto-') and VERSION_RE.fullmatch(t[6:])) + or PLATFORM_TAG_RE.fullmatch(t)] + if not tags: + print('No matching release tags') + return + remote = {} + for line in git('ls-remote', '--tags', '--refs', self.remote).stdout.splitlines(): + sha, ref = line.split() + remote[ref.removeprefix('refs/tags/')] = sha + pending = [] + published = 0 + for tag in tags: + local = self.tag_object(tag) + old = remote.get(tag) + if old is None: + pending.append(tag) + elif old == local: + published += 1 + else: + raise ReleaseError(f'Tag conflict on {self.remote}: {tag} local={local} remote={old}') + print(f'Upstream: {self.remote}\nAlready published: {published}\nNew tags: {len(pending)}') + for tag in pending: + print(f' {tag}') + if pending and self.execute: + git('push', '--atomic', self.remote, + *(f'refs/tags/{tag}:refs/tags/{tag}' for tag in pending)) + print(f'Published {len(pending)} tags.') + elif not self.execute: + print('DRY RUN: No tags pushed.') + + def lts_description(self, branch): + if branch in self.lts or self.force_lts: + opening = ('This is a Long Term Support (LTS) release. meta-arm generally aims ' + 'to follow the Yocto Project\'s LTS support lifecycle on a best-effort ' + 'basis, with maintenance primarily focused on bug fixes and security ' + 'fixes. CI testing is performed on supported configurations as resources permit.') + else: + opening = ('This is NOT a Long Term Support (LTS) release. meta-arm generally aims ' + 'to follow the Yocto Project\'s release support lifecycle on a best-effort ' + 'basis. For non-LTS releases, this means approximately seven months of ' + 'limited maintenance, primarily consisting of bug fixes and security fixes, ' + 'with minimal CI testing.') + if self.lts: + names = ', '.join(self.lts[:-1]) + (' and ' if len(self.lts) > 1 else '') + self.lts[-1] + opening += f'\n\nThe currently supported LTS branches are {names}.' + return opening + ('\n\nFor more information about the Yocto Project\'s release ' + f'lifecycle and LTS policy, see:\n{SUPPORT_URL}') + + def announcement(self, branch, version): + self.require_branch(branch) + series = self.discover_series()[branch] + if version != series and not re.fullmatch(re.escape(series) + r'\.\d+', version): + raise ReleaseError(f'{version} does not belong to {branch} ({series})') + if not self.tag_commit(version) or self.tag_commit(version) != self.tag_commit(f'yocto-{version}'): + raise ReleaseError(f'Missing or mismatched release tag pair: {version}') + if not self.ancestor(f'refs/tags/{version}', self.branch_ref(branch)): + raise ReleaseError(f'{version} is not on {branch}') + lts_text = self.lts_description(branch) + if version == series: + previous = previous_series(version) + if not self.tag_commit(previous): + raise ReleaseError(f'Missing previous initial release tag {previous}') + count = git('rev-list', '--count', f'{previous}..{version}').stdout.strip() + subject = f'meta-arm {branch} release' + body = (f'meta-arm has released the {branch} support branch, tagged as {version}.\n\n' + f'{lts_text}\n\nThis release contains {count} commits since the previous ' + f'Yocto Project release, {previous}.') + else: + patch = int(version.rsplit('.', 1)[1]) + previous = series if patch == 1 else f'{series}.{patch - 1}' + if not self.tag_commit(previous) or not self.ancestor(previous, version): + raise ReleaseError(f'Missing or non-ancestor previous tag: {previous}') + log = git('log', '--reverse', '--format=%h %s', f'{previous}..{version}').stdout.strip() + count = int(git('rev-list', '--count', f'{previous}..{version}').stdout) + if not count: + raise ReleaseError('No commits in update release') + dates = git('log', '--reverse', '--format=%cs', f'{previous}..{version}').stdout.splitlines() + first, last = dates[0], git('log', '-1', '--format=%cs', version).stdout.strip() + subject = f'meta-arm {branch} update' + body = (f'meta-arm has released an updated {branch} support tag, {version}.\n\n' + f'{lts_text}\n\nThis release contains {count} commits between {first} and {last}.\n\n' + 'Changes included in this release:\n\n' + + '\n'.join(' ' + line for line in log.splitlines())) + body += (f'\n\nThe release is available under the {version} and yocto-{version} tags.\n' + 'Both tags are GPG signed.\n\nThanks,\nThe meta-arm team') + return subject, body + + @staticmethod + def editor_command(): + # git var GIT_EDITOR already respects GIT_EDITOR, VISUAL, EDITOR and core.editor. + result = git('var', 'GIT_EDITOR', check=False) + return result.stdout.strip() if result.returncode == 0 and result.stdout.strip() else 'vi' + + def edit_message(self, subject, body): + original = f'Subject: {subject}\n\n{body}\n' + with tempfile.TemporaryDirectory(prefix='meta-arm-release-') as directory: + path = Path(directory) / 'announcement.txt' + path.write_text(original, encoding='utf-8') + editor = self.editor_command() + print(f'Opening announcement in {editor}...', flush=True) + # Interactive editors must inherit stdin, stdout, and stderr. + # Git editor settings may contain shell quoting and arguments. + try: + subprocess.run( + ['sh', '-c', f'{editor} "$1"', 'editor', str(path)], + check=True, + ) + except subprocess.CalledProcessError as exc: + raise ReleaseError('Editor exited unsuccessfully; email not sent') from exc + edited = path.read_text(encoding='utf-8') + if edited == original: + print('Email unchanged; not sending.') + return None + header, separator, text = edited.partition('\n\n') + if not separator or not header.startswith('Subject: '): + raise ReleaseError('Edited email must begin with Subject: followed by a blank line') + new_subject = header.removeprefix('Subject: ').strip() + if not new_subject or '\r' in new_subject or '\n' in new_subject or not text.strip(): + raise ReleaseError('Invalid edited email subject or body') + return new_subject, text.rstrip('\n') + + def mail_backend(self): + """Resolve the requested mailer without silently changing forced choices.""" + sendmail = shutil.which('sendmail') + if not sendmail and os.access('/usr/sbin/sendmail', os.X_OK): + sendmail = '/usr/sbin/sendmail' + mutt = shutil.which('mutt') + + if self.mailer == 'sendmail': + return sendmail + if self.mailer == 'mutt': + return mutt + return sendmail or mutt + + def send(self, subject, body): + backend = self.mail_backend() + if not backend: + raise ReleaseError(f'Requested mailer {self.mailer!r} is unavailable' if self.mailer != 'auto' + else 'Neither sendmail nor mutt is available') + if os.path.basename(backend) == 'mutt': + subprocess.run([backend, '-s', subject, '--', self.recipient], + input=body + '\n', text=True, check=True) + else: + message = EmailMessage() + message['To'] = self.recipient + message['Subject'] = subject + message.set_content(body) + subprocess.run([backend, '-t', '-i'], input=message.as_bytes(), check=True) + + def email(self, branch, version): + subject, body = self.announcement(branch, version) + if self.edit: + changed = self.edit_message(subject, body) + if changed is None: + return + subject, body = changed + print(f'\nSubject: {subject}\n\n{body}\n') + if not self.execute: + print('DRY RUN: Email not sent.') + return + for tag in (version, f'yocto-{version}'): + output = git('ls-remote', '--tags', '--refs', self.remote, f'refs/tags/{tag}').stdout + entries = dict(line.split()[::-1] for line in output.splitlines()) + if entries.get(f'refs/tags/{tag}') != self.tag_object(tag): + raise ReleaseError(f'{tag} is not published identically on {self.remote}') + self.send(subject, body) + print('Email submitted successfully.') + + def check(self): + print('meta-arm release environment\n' + '-' * 48) + errors = 0 + inside = git('rev-parse', '--is-inside-work-tree', check=False).returncode == 0 + print(f'Git repository: {"OK" if inside else "FAILED"}') + errors += not inside + url = git('remote', 'get-url', self.remote, check=False) + print(f'Upstream remote: {self.remote} ({url.stdout.strip() or "missing"})') + reachable = url.returncode == 0 and git('ls-remote', '--heads', self.remote, check=False).returncode == 0 + print(f'Upstream reachable: {"OK" if reachable else "FAILED"}') + errors += not reachable + master = git('rev-parse', '--verify', '-q', f'{self.branch_ref("master")}^{{commit}}', check=False).returncode == 0 + print(f'Master branch: {"OK" if master else "MISSING"}') + errors += not master + if master: + print(f'Release branches: {len(self.branches())}') + identity = (git('config', 'user.name', check=False).stdout.strip(), + git('config', 'user.email', check=False).stdout.strip()) + print(f'Git identity: {identity[0]} <{identity[1]}>' if all(identity) + else 'Git identity: NOT CONFIGURED') + errors += not all(identity) + signing = git('config', 'user.signingkey', check=False).stdout.strip() + print(f'GPG signing key: {signing or "Git default (not verified)"}') + mailer_path = self.mail_backend() + mailer_note = ' (forced)' if self.mailer != 'auto' else ' (auto)' + print(f'Email delivery: {(mailer_path or "Not available") + mailer_note}') + print(f'Email recipient: {self.recipient}') + print(f'Editor: {self.editor_command()}') + print(f'LTS branches: {", ".join(self.lts)}') + print('Environment ready.' if not errors else f'Environment check failed: {errors} error(s).') + return not errors + + +class FriendlyArgumentParser(argparse.ArgumentParser): + """Display the curated usage page without argparse's duplicate sections.""" + + def format_help(self): + return self.description + '\n' + + +def parser(): + """Build a CLI with a compact, example-driven help page.""" + help_text = """\ +Usage: meta-arm-release.py COMMAND [ARGUMENTS] [OPTIONS] + +Commands: + check Check the release environment + status [BRANCH...] Show release status and pending commits + release BRANCH VERSION Create initial signed release tags (x.y) + update [BRANCH...] Tag branches containing new commits (x.y.z) + tag BRANCH VERSION REF Tag a specific historical commit + platform NAME REF Create a signed platform release tag + push [TAG|BRANCH...] Publish missing release tags to upstream + email BRANCH VERSION Preview or send an announcement + all BRANCH VERSION Initial release, updates, push and announcements + +Options: + -h, --help, -? Display this help + -n, --dry-run Preview operations (default) + -x, --execute Perform requested operations + -e, --edit Edit announcements using your configured editor + --lts Treat specified release as LTS + --as-of YYYY-MM-DD Select historical first-parent commit by date + --date YYYY.MM.DD Platform tag date (default: today) + --remote NAME Upstream remote (default: yoctoproject) + --email-to ADDRESS Override announcement recipient + --mailer {auto,sendmail,mutt} Select email delivery method (default: auto) + --lts-branches "A B" Set supported LTS branches + +Examples: + meta-arm-release.py check + meta-arm-release.py status + meta-arm-release.py status scarthgap wrynose + meta-arm-release.py release blacksail 6.1 + meta-arm-release.py release blacksail 6.1 --execute + meta-arm-release.py update scarthgap whinlatter --execute + meta-arm-release.py tag wrynose 6.0 COMMIT --execute + meta-arm-release.py platform corstone1000 HEAD + meta-arm-release.py platform corstone1000 HEAD --date 2026.10.01 --execute + meta-arm-release.py push CORSTONE1000-2026.10.01 --execute + meta-arm-release.py push + meta-arm-release.py push --execute + meta-arm-release.py email blacksail 6.1 + meta-arm-release.py email blacksail 6.1 --edit + meta-arm-release.py email blacksail 6.1 --edit --execute + meta-arm-release.py email blacksail 6.1 --mailer mutt --execute + meta-arm-release.py all blacksail 6.1 --execute + +Notes: + * Dry-run is the default; --execute does not ask for confirmation. + * Email previews require no mail program; automatic delivery prefers + sendmail and falls back to mutt. Use --mailer to force a backend. + * Editor selection follows Git configuration (including VISUAL/EDITOR). + * Release branches must exist upstream; branches containing '/' are skipped. + * Existing tags are never overwritten; only missing tags are pushed. + * Push without arguments discovers Yocto and platform release tags. + * Push operations use only the configured upstream remote. + * The script never checks out branches. + +Environment variables: + META_ARM_RELEASE_REMOTE Override upstream remote + META_ARM_RELEASE_EMAIL Override announcement recipient + META_ARM_LTS_BRANCHES Space-separated LTS branches + META_ARM_MAILER auto, sendmail, or mutt +""" + p = FriendlyArgumentParser( + prog='meta-arm-release.py', + description=help_text, + formatter_class=argparse.RawDescriptionHelpFormatter, + add_help=False, + ) + p.add_argument('-h', '--help', '-?', action='help', help=argparse.SUPPRESS) + p.add_argument('--execute', '-x', action='store_true', help=argparse.SUPPRESS) + p.add_argument('--dry-run', '-n', action='store_true', help=argparse.SUPPRESS) + p.add_argument('--edit', '-e', action='store_true', help=argparse.SUPPRESS) + p.add_argument('--lts', action='store_true', help=argparse.SUPPRESS) + p.add_argument('--as-of', metavar='YYYY-MM-DD', help=argparse.SUPPRESS) + p.add_argument('--date', metavar='YYYY.MM.DD', help=argparse.SUPPRESS) + p.add_argument('--remote', default=os.getenv('META_ARM_RELEASE_REMOTE', 'yoctoproject'), + help=argparse.SUPPRESS) + p.add_argument('--email-to', default=os.getenv('META_ARM_RELEASE_EMAIL', 'meta-arm@lists.yoctoproject.org'), + help=argparse.SUPPRESS) + p.add_argument('--mailer', choices=('auto', 'sendmail', 'mutt'), + default=os.getenv('META_ARM_MAILER', 'auto'), help=argparse.SUPPRESS) + p.add_argument('--lts-branches', default=os.getenv('META_ARM_LTS_BRANCHES', 'scarthgap wrynose'), + help=argparse.SUPPRESS) + sub = p.add_subparsers(dest='command', required=True, metavar='COMMAND') + sub.add_parser('check', add_help=False) + q = sub.add_parser('status', add_help=False) + q.add_argument('branches', nargs='*') + q = sub.add_parser('release', add_help=False) + q.add_argument('branch') + q.add_argument('version') + q = sub.add_parser('update', add_help=False) + q.add_argument('branches', nargs='*') + q = sub.add_parser('tag', add_help=False) + q.add_argument('branch') + q.add_argument('version') + q.add_argument('ref') + q = sub.add_parser('platform', add_help=False) + q.add_argument('name') + q.add_argument('ref') + q = sub.add_parser('push', add_help=False) + q.add_argument('selections', nargs='*') + q = sub.add_parser('email', add_help=False) + q.add_argument('branch') + q.add_argument('version') + q = sub.add_parser('all', add_help=False) + q.add_argument('branch') + q.add_argument('version') + return p + + +def main(argv=None): + p = parser() + # Support global options either before or after the subcommand. + args = list(sys.argv[1:] if argv is None else argv) + if not args or any(token in ('-h', '--help', '-?') for token in args): + p.print_help() + return 0 + globals_with_value = ('--as-of', '--date', '--remote', '--email-to', '--lts-branches', '--mailer') + globals_flags = ('--execute', '-x', '--dry-run', '-n', '--edit', '-e', '--lts') + front, rest = [], [] + i = 0 + while i < len(args): + token = args[i] + if token in globals_with_value: + if i + 1 >= len(args): + p.error(f'{token} requires a value') + front.extend(args[i:i+2]); i += 2 + elif token in globals_flags: + front.append(token); i += 1 + else: + rest.append(token); i += 1 + ns = p.parse_args(front + rest) + if ns.email_to.find('\n') >= 0 or ns.email_to.find('\r') >= 0: + p.error('Invalid email recipient') + if ns.date and ns.command != 'platform': + p.error('--date only applies to platform') + if ns.edit and ns.command not in ('email', 'all'): + p.error('--edit only applies to email and all') + if ns.as_of and not re.fullmatch(r'\d{4}-\d{2}-\d{2}', ns.as_of): + p.error('--as-of requires YYYY-MM-DD') + manager = ReleaseManager(ns.remote, ns.email_to, ns.lts_branches.split(), + ns.execute and not ns.dry_run, ns.edit, ns.as_of, ns.lts, ns.mailer) + if ns.command == 'check': + return 0 if manager.check() else 1 + if git('rev-parse', '--is-inside-work-tree', check=False).returncode: + raise ReleaseError('Not in a Git repository') + if ns.command == 'status': + manager.show_status(ns.branches or manager.branches()) + elif ns.command == 'release': + manager.release(ns.branch, ns.version) + elif ns.command == 'update': + manager.update(ns.branches or manager.branches()) + elif ns.command == 'tag': + manager.require_branch(ns.branch) + expected = manager.discover_series()[ns.branch] + if ns.version != expected and not re.fullmatch(re.escape(expected) + r'\.\d+', ns.version): + raise ReleaseError(f'Wrong version series for {ns.branch}: {ns.version}') + manager.create_tags(ns.branch, ns.version, ns.ref, 'Release') + elif ns.command == 'platform': + manager.platform(ns.name, ns.ref, ns.date) + elif ns.command == 'push': + manager.push(ns.selections) + elif ns.command == 'email': + manager.email(ns.branch, ns.version) + elif ns.command == 'all': + manager.release(ns.branch, ns.version) + created = [(ns.branch, ns.version)] + created += manager.update([b for b in manager.branches() if b != ns.branch]) + if manager.execute: + manager.push() + for branch, version in created: + manager.email(branch, version) + else: + print('DRY RUN: No tags created, pushed or emails sent.') + return 0 + + +if __name__ == '__main__': + try: + sys.exit(main()) + except (ReleaseError, subprocess.CalledProcessError) as error: + print(f'ERROR: {error}', file=sys.stderr) + sys.exit(1) diff --git a/scripts/test_meta_arm_release.py b/scripts/test_meta_arm_release.py new file mode 100644 index 000000000000..8c0d3c39a211 --- /dev/null +++ b/scripts/test_meta_arm_release.py @@ -0,0 +1,220 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: 2026 Arm Limited +# SPDX-License-Identifier: MIT +"""Offline unit and integration tests for meta-arm-release.py.""" + +import importlib.util +import os +import subprocess +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +SCRIPT = Path(__file__).with_name('meta-arm-release.py') +spec = importlib.util.spec_from_file_location('meta_arm_release', SCRIPT) +release = importlib.util.module_from_spec(spec) +import sys +sys.modules[spec.name] = release +spec.loader.exec_module(release) + + +class VersionTests(unittest.TestCase): + def test_initial_release_sequence(self): + self.assertEqual(release.previous_series('6.0'), '5.3') + self.assertEqual(release.previous_series('6.1'), '6.0') + self.assertEqual(release.series_from_number(release.series_number('5.3') + 1), '6.0') + + def test_patch_version(self): + self.assertEqual(release.next_patch('5.0'), '5.0.1') + self.assertEqual(release.next_patch('5.0.4'), '5.0.5') + + def test_bad_version(self): + with self.assertRaises(release.ReleaseError): + release.series_number('6.4') + + +class GitFixture(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.addCleanup(self.temp.cleanup) + self.repo = Path(self.temp.name) + self.previous = Path.cwd() + os.chdir(self.repo) + self.addCleanup(os.chdir, self.previous) + self.git('init', '-q', '-b', 'master') + self.git('config', 'user.name', 'Test Maintainer') + self.git('config', 'user.email', 'test@example.com') + self.commit('initial') + self.remote = self.repo / 'upstream.git' + subprocess.check_call(['git', 'init', '-q', '--bare', str(self.remote)]) + self.git('remote', 'add', 'yoctoproject', str(self.remote)) + self.git('update-ref', 'refs/remotes/yoctoproject/master', self.git('rev-parse', 'master')) + + def git(self, *args): + return subprocess.check_output(['git', *args], text=True).strip() + + def commit(self, subject): + with (self.repo / 'history.txt').open('a') as stream: + stream.write(subject + '\n') + self.git('add', 'history.txt') + self.git('commit', '-q', '-m', subject) + + def branch(self, name): + self.git('branch', name) + self.git('update-ref', f'refs/remotes/yoctoproject/{name}', self.git('rev-parse', name)) + + def refresh(self, name): + self.git('update-ref', f'refs/remotes/yoctoproject/{name}', self.git('rev-parse', name)) + + +class ReleaseIntegrationTests(GitFixture): + def test_existing_tag_and_update(self): + self.branch('scarthgap') + self.git('checkout', '-q', 'scarthgap') + self.git('tag', '-a', '5.0', '-m', 'initial') + self.git('tag', '-a', 'yocto-5.0', '-m', 'initial') + self.commit('fix one') + self.commit('fix two') + self.refresh('scarthgap') + self.git('checkout', '-q', 'master') + manager = release.ReleaseManager() + state = manager.status('scarthgap') + self.assertEqual((state.series, state.last_tag, state.commits, state.next_tag), + ('5.0', '5.0', 2, '5.0.1')) + + def test_no_updates(self): + self.branch('scarthgap') + self.git('tag', '-a', '5.0', '-m', 'initial') + self.git('tag', '-a', 'yocto-5.0', '-m', 'initial') + state = release.ReleaseManager().status('scarthgap') + self.assertEqual(state.commits, 0) + self.assertEqual(state.next_tag, '-') + + def test_announcement_initial_count(self): + self.branch('wrynose') + self.git('tag', '-a', '6.0', '-m', 'initial') + self.git('tag', '-a', 'yocto-6.0', '-m', 'initial') + self.commit('master change') + self.branch('blacksail') + self.git('tag', '-a', '6.1', '-m', 'initial') + self.git('tag', '-a', 'yocto-6.1', '-m', 'initial') + manager = release.ReleaseManager() + subject, body = manager.announcement('blacksail', '6.1') + self.assertEqual(subject, 'meta-arm blacksail release') + self.assertIn('1 commits since the previous Yocto Project release, 6.0', body) + self.assertIn('The meta-arm team', body) + self.assertNotIn('Changes included', body) + + def test_update_announcement_lists_changes(self): + self.branch('scarthgap') + self.git('checkout', '-q', 'scarthgap') + self.git('tag', '-a', '5.0', '-m', 'initial') + self.git('tag', '-a', 'yocto-5.0', '-m', 'initial') + self.commit('fix one') + self.git('tag', '-a', '5.0.1', '-m', 'update') + self.git('tag', '-a', 'yocto-5.0.1', '-m', 'update') + self.refresh('scarthgap') + manager = release.ReleaseManager() + subject, body = manager.announcement('scarthgap', '5.0.1') + self.assertIn('fix one', body) + self.assertIn('1 commits', body) + self.assertIn('5.0.1 and yocto-5.0.1 tags', body) + + def test_push_dry_run_does_not_modify_remote(self): + self.branch('scarthgap') + self.git('tag', '-a', '5.0', '-m', 'initial') + self.git('tag', '-a', 'yocto-5.0', '-m', 'initial') + manager = release.ReleaseManager(execute=False) + with patch('builtins.print') as printer: + manager.push(['scarthgap']) + self.assertTrue(any('New tags: 2' in str(call) for call in printer.call_args_list)) + + + +class MailerTests(unittest.TestCase): + def test_auto_prefers_sendmail(self): + with patch.object(release.shutil, 'which', side_effect=lambda x: {'sendmail': '/usr/bin/sendmail', 'mutt': '/usr/bin/mutt'}.get(x)): + self.assertEqual(release.ReleaseManager(mailer='auto').mail_backend(), '/usr/bin/sendmail') + + def test_force_mutt(self): + with patch.object(release.shutil, 'which', side_effect=lambda x: {'sendmail': '/usr/bin/sendmail', 'mutt': '/usr/bin/mutt'}.get(x)): + self.assertEqual(release.ReleaseManager(mailer='mutt').mail_backend(), '/usr/bin/mutt') + + def test_force_sendmail_unavailable_does_not_fallback(self): + with patch.object(release.shutil, 'which', side_effect=lambda x: {'mutt': '/usr/bin/mutt'}.get(x)), patch.object(release.os, 'access', return_value=False): + manager = release.ReleaseManager(mailer='sendmail') + self.assertIsNone(manager.mail_backend()) + with self.assertRaisesRegex(release.ReleaseError, 'sendmail'): + manager.send('subject', 'body') + + def test_cli_override_env(self): + with patch.dict(os.environ, {'META_ARM_MAILER': 'mutt'}): + self.assertEqual(release.parser().parse_args(['--mailer', 'sendmail', 'check']).mailer, 'sendmail') + self.assertEqual(release.parser().parse_args(['check']).mailer, 'mutt') + + def test_invalid_mailer_rejected(self): + with self.assertRaises(release.ReleaseError): + release.ReleaseManager(mailer='invalid') + +if __name__ == '__main__': + unittest.main(verbosity=2) + + +class EditorTests(unittest.TestCase): + def test_editor_inherits_terminal_streams(self): + manager = release.ReleaseManager(edit=True) + with patch.object(manager, 'editor_command', return_value='vim -c "set spell"'): + with patch.object(release.subprocess, 'run') as invoke: + result = manager.edit_message('Subject', 'Body') + self.assertIsNone(result) + invoke.assert_called_once() + args, kwargs = invoke.call_args + self.assertEqual(args[0][:3], ['sh', '-c', 'vim -c "set spell" "$1"']) + self.assertTrue(kwargs['check']) + self.assertNotIn('stdout', kwargs) + self.assertNotIn('stderr', kwargs) + self.assertNotIn('stdin', kwargs) + + def test_editor_failure_prevents_send(self): + manager = release.ReleaseManager(edit=True) + with patch.object(manager, 'editor_command', return_value='vim'): + with patch.object(release.subprocess, 'run', side_effect=subprocess.CalledProcessError(1, 'vim')): + with self.assertRaisesRegex(release.ReleaseError, 'Editor exited unsuccessfully'): + manager.edit_message('Subject', 'Body') + + +class PlatformReleaseTests(GitFixture): + def test_platform_tag_name_and_date(self): + self.assertEqual(release.platform_tag('corstone1000', '2026.10.01'), + 'CORSTONE1000-2026.10.01') + with self.assertRaises(release.ReleaseError): + release.platform_tag('corstone1000', '2026.02.30') + + def test_platform_dry_run_does_not_create_tag(self): + manager = release.ReleaseManager() + tag = manager.platform('corstone1000', 'HEAD', '2026.10.01') + self.assertEqual(tag, 'CORSTONE1000-2026.10.01') + self.assertIsNone(manager.tag_object(tag)) + + def test_platform_existing_tag_is_idempotent(self): + self.git('tag', '-a', 'CORSTONE1000-2026.10.01', '-m', 'release') + manager = release.ReleaseManager(execute=True) + self.assertEqual(manager.platform('corstone1000', 'HEAD', '2026.10.01'), + 'CORSTONE1000-2026.10.01') + + def test_push_explicit_platform_tag(self): + self.git('tag', '-a', 'CORSTONE1000-2026.10.01', '-m', 'release') + manager = release.ReleaseManager(execute=True) + manager.push(('CORSTONE1000-2026.10.01',)) + remote = self.git('ls-remote', '--tags', '--refs', 'yoctoproject', + 'refs/tags/CORSTONE1000-2026.10.01') + self.assertIn('refs/tags/CORSTONE1000-2026.10.01', remote) + + def test_push_auto_discovers_platform_tag(self): + self.git('tag', '-a', 'CORSTONE1000-2026.10.01', '-m', 'release') + manager = release.ReleaseManager(execute=True) + manager.push() + remote = self.git('ls-remote', '--tags', '--refs', 'yoctoproject', + 'refs/tags/CORSTONE1000-2026.10.01') + self.assertIn('refs/tags/CORSTONE1000-2026.10.01', remote) From patchwork Tue Nov 10 19:14:05 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jon Mason X-Patchwork-Id: 103603 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.393.1794338048516941587 for ; Tue, 10 Nov 2026 11:14:09 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@arm.com header.s=foss header.b=gRL+LASv; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: jon.mason@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ED8BF14BF for ; Tue, 10 Nov 2026 11:14:04 -0800 (PST) Received: from H24V3P4C17.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id CAFD13F673 for ; Tue, 10 Nov 2026 11:14:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1794338048; bh=FXVSxT8ZrSbdxokcxFHmfe375JE+H35rpPmNkVa/BYc=; h=From:To:Subject:Date:In-Reply-To:References:From; b=gRL+LASvi5DTDk+ZnZYQYvOwCoRXvn9ssAjXrwr7t/cXoSw/0OfSeK0W1/SHlBqcP WXBc2I5g+evVHBqFAplK8BHpYYMFM3DJHHDslKKRIz8ucTgqqEod9sFR45iG6ZdBJF hUu/tyaqDkTlzWP06cowkgLg4NTA4O8Nhy8rucKg= From: Jon Mason To: meta-arm@lists.yoctoproject.org Subject: [PATCH 2/3] scripts: add utility to identify unused BSP recipes and patches Date: Tue, 10 Nov 2026 14:14:05 -0500 Message-ID: <20261110191406.48308-2-jon.mason@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261110191406.48308-1-jon.mason@arm.com> References: <20261110191406.48308-1-jon.mason@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Tue, 10 Nov 2026 19:14:12 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/7206 Add a Python utility to identify potentially unused files in meta-arm. The script performs static analysis of BitBake metadata to find patches that are no longer referenced, unused include files in meta-arm-bsp, and BSP recipes that may have become obsolete as platforms are updated. Recipe detection examines preferred versions, providers, image contents, and dependencies to identify recipes that are no longer referenced. Results are grouped by confidence to help distinguish likely unused recipes from those requiring further investigation. The utility operates independently of BitBake and requires only Python’s standard library. By default, it reports findings without modifying files. Dangling patches can optionally be removed after confirmation, while recipes and include files remain report-only to avoid unintended removal. This utility consolidates several Bash scripts originally written by hand. ChatGPT was used to translate them to Python, combine their functionality, and extend the reference detection. AI-generated: ChatGPT (OpenAI) Signed-off-by: Jon Mason --- scripts/find-unused.py | 279 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 279 insertions(+) create mode 100644 scripts/find-unused.py diff --git a/scripts/find-unused.py b/scripts/find-unused.py new file mode 100644 index 000000000000..782dacf4ff79 --- /dev/null +++ b/scripts/find-unused.py @@ -0,0 +1,279 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Copyright (c) 2026 Arm Limited +"""Report dangling patches, unreferenced BSP includes, and candidate obsolete BSP recipes. + +Static analysis only; no BitBake environment required. Recipe/include results are +review-only. Patch deletion is opt-in and still requires care with dynamic SRC_URI. +""" +import argparse +import fnmatch +import re +import subprocess +import sys +from collections import defaultdict +from pathlib import Path + +META_EXT = {'.bb', '.bbappend', '.inc', '.conf', '.bbclass'} +ASSIGN = re.compile(r'^\s*([A-Za-z0-9_${}/.+%-]+(?::[A-Za-z0-9_${}/.+%-]+)*)\s*(?:\?\?=|\?=|:=|\+=|=\+|\.=|=\.|=)\s*(.*)$') +RECIPE = re.compile(r'^(.+)_([^_]+)\.(bb|bbappend)$') +TOKEN = re.compile(r'[A-Za-z0-9][A-Za-z0-9_.+%-]*') + + +def git(cmd, root): + return subprocess.run(['git', '-C', str(root), *cmd], capture_output=True) + + +def load(root): + r = git(['rev-parse', '--show-toplevel'], root) + if r.returncode: + raise RuntimeError('Not in a Git checkout; specify --root') + root = Path(r.stdout.decode().strip()).resolve() + r = git(['ls-files', '-z'], root) + if r.returncode: + raise RuntimeError('git ls-files failed') + paths = [Path(p.decode('utf-8', 'surrogateescape')) for p in r.stdout.split(b'\0') if p] + sources = {} + for path in paths: + if path.suffix in META_EXT: + try: + sources[path] = (root / path).read_text(encoding='utf-8', errors='replace') + except OSError: + pass + return root, paths, sources + + +def in_bsp(path): + return path.parts and path.parts[0] == 'meta-arm-bsp' + + +def recipe_tree(path): + return any(part.startswith('recipes-') for part in path.parts) + + +def refs(name, sources, skip=None): + return [p for p, text in sources.items() if p != skip and name in text] + + +def patches(paths, sources): + for path in paths: + if path.suffix != '.patch' or not recipe_tree(path): + continue + found = refs(path.name, sources) + if not found: + yield ('patches', path, 'No literal filename reference in tracked metadata', True) + + +def includes(paths, sources): + for path in paths: + if not in_bsp(path) or path.suffix != '.inc': + continue + found = refs(path.name, sources, path) + if found: + continue + # A basename may be constructed from variable expansions, e.g. + # require ${BPN}.inc. Treat these as ambiguous rather than safe. + parent = path.parent + dynamic = [p for p, content in sources.items() + if p != path and (p.parent == parent or parent in p.parents) + and re.search(r'(?m)^\s*(?:require|include)\s+[^\n#]*\$\{', content)] + note = 'No literal include reference found' + if dynamic: + note += '; dynamic include(s) in ' + ', '.join(str(p) for p in dynamic[:3]) + yield ('includes', path, note + ' (review only)', False) + + +def logical_lines(text): + return re.sub(r'\\\s*\n\s*', ' ', text).splitlines() + + +def metadata(sources): + selections = defaultdict(list) + mentions = defaultdict(set) + for path, content in sources.items(): + for line in logical_lines(content): + line = line.split('#', 1)[0] + m = ASSIGN.match(line) + if not m: + continue + key, value = m.groups() + if key.startswith('PREFERRED_VERSION_'): + name = key[len('PREFERRED_VERSION_'):].split(':', 1)[0] + versions = re.findall(r'["\']([^"\']+)["\']', value) + if versions: + selections[name].extend((v.strip(), path) for v in versions) + continue + # Only dependency/provider/image variables imply usage. Merely + # mentioning a recipe in an arbitrary string is not sufficient. + base = key.split(':', 1)[0] + if (base in {'IMAGE_INSTALL', 'CORE_IMAGE_EXTRA_INSTALL', 'PACKAGE_INSTALL', + 'MACHINE_ESSENTIAL_EXTRA_RDEPENDS', 'MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS', + 'MACHINE_EXTRA_RDEPENDS', 'MACHINE_EXTRA_RRECOMMENDS', + 'DEPENDS', 'RDEPENDS', 'RRECOMMENDS', 'EXTRA_IMAGEDEPENDS', + 'PREFERRED_PROVIDER_virtual/kernel', 'PREFERRED_PROVIDER_virtual/bootloader'} + or base.startswith(('RDEPENDS_', 'RRECOMMENDS_', 'PREFERRED_PROVIDER_'))): + for token in TOKEN.findall(value): + mentions[token].add(path) + return selections, mentions + + +def version_matches(version, pattern): + if '${' in pattern or '$(' in pattern: + return None + return fnmatch.fnmatchcase(version, pattern.replace('%', '*')) + + +def recipes(paths, sources, verbose=False): + selections, mentions = metadata(sources) + groups = defaultdict(list) + for path in paths: + if not in_bsp(path) or not recipe_tree(path): + continue + m = RECIPE.match(path.name) + if m: + name, version, kind = m.groups() + groups[name].append((path, version, kind)) + + for name, entries in sorted(groups.items()): + picks = selections.get(name, []) + usage = mentions.get(name, set()) + # Version-specific recipes are candidates only when there is evidence + # of a version transition, or a vanished explicit pin. No pin and no + # references is suspicious, but not proof: default selection is valid. + for path, version, kind in entries: + if version == '%' or '${' in version: + continue + matched = [(p, src) for p, src in picks if version_matches(version, p)] + if matched: + if verbose: + yield ('referenced', path, 'Version selected by ' + ', '.join(str(src) for _, src in matched[:4]), False) + continue + unknown = [(p, src) for p, src in picks if version_matches(version, p) is None] + if unknown: + if verbose: + yield ('referenced', path, 'Dynamic PREFERRED_VERSION: cannot resolve statically', False) + continue + # Check versions of the same type, not unrelated bb vs bbappend. + peers = [(other, v) for other, v, k in entries if k == kind and other != path] + newer_or_selected = [v for _, v in peers if v != '%' and (not picks or any(version_matches(v, p) for p, _ in picks))] + if picks: + reason = 'No matching PREFERRED_VERSION; selected: ' + ', '.join(sorted({p for p, _ in picks})) + if not newer_or_selected: + reason += '; no alternative matching local version' + else: + reason = 'No PREFERRED_VERSION found' + if peers: + reason += '; other local versions: ' + ', '.join(sorted({v for _, v in peers})) + if usage: + reason += '; recipe/provider referenced in ' + ', '.join(str(p) for p in sorted(usage)[:3]) + category = 'potentially obsolete' + elif picks or peers: + category = 'potentially obsolete' + reason += '; no direct image/dependency/provider reference found' + else: + category = 'likely unused' + reason += '; no direct image/dependency/provider reference found (default BitBake selection still possible)' + yield ('recipes', path, f'{category}: {reason}', False) + + +def main(): + ap = argparse.ArgumentParser( + description='Find cleanup candidates in a meta-arm Git checkout (no BitBake required).', + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog="""Checks and scope: + patches All layers: tracked .patch files with no literal filename reference + in tracked BitBake metadata. Dynamic SRC_URI can cause false positives. + includes meta-arm-bsp only: .inc files without literal references; dynamic + include/require statements are noted where detected. + recipes meta-arm-bsp only: versioned .bb/.bbappend files compared with + PREFERRED_VERSION settings and image/dependency/provider mentions. + Results are grouped by confidence: likely unused, then potentially + obsolete. A recipe may be selected implicitly without a version pin. + all Run all checks (default). + +Examples: + scripts/find-unused.py # Report everything + scripts/find-unused.py recipes --verbose # Explain BSP recipe findings + scripts/find-unused.py patches # Find dangling patches + scripts/find-unused.py includes # Review BSP include files + scripts/find-unused.py patches --delete # Confirm each patch removal + scripts/find-unused.py patches --delete --force + scripts/find-unused.py --root /path/to/meta-arm all + +Safety: + Reporting is the default. Only dangling patches can be deleted; recipe + and include findings are ALWAYS review-only, even with --delete --force. + Searches use tracked files from Git but read the current working tree, so + uncommitted edits to machine configurations affect the results. No BitBake + parsing, overrides, providers, or dynamic variable expansions are resolved. +""") + ap.add_argument('check', nargs='?', choices=['all', 'patches', 'includes', 'recipes'], default='all', help='Check to run (default: all)') + ap.add_argument('--root', type=Path, default=Path.cwd(), help='Path inside the meta-arm Git checkout (default: current directory)') + ap.add_argument('--verbose', action='store_true', help='Show reasons and selected-version matches, including referenced recipes') + ap.add_argument('--delete', action='store_true', help='Prompt before deleting dangling patches only') + ap.add_argument('--force', action='store_true', help='With --delete, skip prompts') + ap.add_argument('--dry-run', action='store_true', help='Explicit report-only mode (the default)') + args = ap.parse_args() + if args.force and not args.delete: + ap.error('--force requires --delete') + if args.delete and args.dry_run: + ap.error('--delete conflicts with --dry-run') + try: + root, paths, sources = load(args.root) + except RuntimeError as exc: + ap.error(str(exc)) + if args.verbose: + print(f'Repository: {root} | tracked files: {len(paths)} | metadata: {len(sources)}') + checks = {'patches': patches, 'includes': includes, 'recipes': recipes} + total = 0 + for check, func in checks.items(): + if args.check not in ('all', check): + continue + found = list(func(paths, sources, args.verbose) if check == 'recipes' else func(paths, sources)) + visible = [f for f in found if f[0] != 'referenced'] + total += len(visible) + print(f'\n{check.capitalize()} ({len(visible)} candidates):') + if check == 'recipes': + def recipe_rank(item): + category, path, reason, _ = item + if category == 'referenced': + return (2, str(path)) + if reason.startswith('likely unused:'): + return (0, str(path)) + return (1, str(path)) + found.sort(key=recipe_rank) + last_level = None + for category, path, reason, safe in found: + if category == 'referenced' and not args.verbose: + continue + if check == 'recipes': + level = ('Referenced' if category == 'referenced' else + 'Likely unused' if reason.startswith('likely unused:') else + 'Potentially obsolete') + if level != last_level: + print(f' {level}:') + last_level = level + print(f' {path}') + else: + print(f' {path}') + if args.verbose or check == 'recipes': + print(f' {reason}' if check == 'recipes' else f' {reason}') + if args.delete and safe: + try: + yes = args.force or input(' Delete patch? [y/N] ').strip().lower() == 'y' + except EOFError: + yes = False + if yes: + try: + (root / path).unlink() + print(' Deleted') + except OSError as exc: + print(f' ERROR: {exc}', file=sys.stderr) + elif args.delete and not safe and category != 'referenced': + print(' Review only; not deleted') + print(f'\nTotal: {total} candidates. Static analysis; verify before removing files.') + return 0 + + +if __name__ == '__main__': + sys.exit(main()) From patchwork Tue Nov 10 19:14:06 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jon Mason X-Patchwork-Id: 103605 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.409.1794338048816200705 for ; Tue, 10 Nov 2026 11:14:09 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="dkim: body hash did not verify" header.i=@arm.com header.s=foss header.b=fyWb6GZP; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: jon.mason@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4BF3A1BC0 for ; Tue, 10 Nov 2026 11:14:05 -0800 (PST) Received: from H24V3P4C17.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 27DE93F673 for ; Tue, 10 Nov 2026 11:14:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1794338048; bh=Ty8etVQlqbS/t9cNoPV/BTwtLSAqylZGKaFKFcY9cM0=; h=From:To:Subject:Date:In-Reply-To:References:From; b=fyWb6GZPSdT6bi2X0hn+tebJkl4mc8bDJZfC6vpmfb1RoVjr0dw4iZ1c5aK38VSWK M1bF02EB6Dszzgh2EqKJ3UKdQLhBaoDJrB8fUjVSQZr/S0m7cwXfXnJamhM25Awcwv 1krrYFzSzoP82GxcxN4EEPZVzvabsmv57fYHZoag= From: Jon Mason To: meta-arm@lists.yoctoproject.org Subject: [PATCH 3/3] scripts: add utility to track patch age and upstream status Date: Tue, 10 Nov 2026 14:14:06 -0500 Message-ID: <20261110191406.48308-3-jon.mason@arm.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261110191406.48308-1-jon.mason@arm.com> References: <20261110191406.48308-1-jon.mason@arm.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Tue, 10 Nov 2026 19:14:12 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/meta-arm/message/7208 Add a Python utility to analyze the age, modification history, and upstream status of patches maintained in meta-arm. The script uses Git history to determine when patches were first added, last modified, and how many times they have changed. It also extracts Upstream-Status information to help identify patches that have remained downstream for extended periods. By default, the utility displays summary statistics, including average patch age, patches older than one or two years, and upstream status distribution. Verbose output provides per-patch information, with options to sort by age or upstream status. Results can also be exported to CSV for further analysis and reporting. The utility operates independently of BitBake and requires only Python’s standard library. This utility is based on an original Bash script written by hand to collect patch maintenance statistics for presentations. ChatGPT was used to translate it to Python and extend its functionality. AI-generated: ChatGPT (OpenAI) Signed-off-by: Jon Mason --- scripts/patch-history.py | 181 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100644 scripts/patch-history.py diff --git a/scripts/patch-history.py b/scripts/patch-history.py new file mode 100644 index 000000000000..3bd20e849ad9 --- /dev/null +++ b/scripts/patch-history.py @@ -0,0 +1,181 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Copyright (c) 2026 Arm Limited +"""Report the age, change history, and upstream status of tracked patches.""" + +import argparse +import csv +import datetime as dt +import re +import subprocess +import sys +from collections import Counter, defaultdict +from pathlib import Path + + +def git(root, *args): + result = subprocess.run(['git', '-C', str(root), *args], capture_output=True) + if result.returncode: + raise RuntimeError(result.stderr.decode('utf-8', 'replace').strip()) + return result.stdout + + +def metadata(root, path): + raw = git(root, 'log', '--follow', '--format=%as', '--', path) + dates = [x for x in raw.decode().splitlines() if x] + if not dates: + return '', '', 0 + return dates[-1], dates[0], len(dates) + + +def upstream_status(path): + try: + content = path.read_text(encoding='utf-8', errors='replace') + except OSError as exc: + print(f'warning: cannot read {path}: {exc}', file=sys.stderr) + return 'Missing', '' + match = re.search(r'^\s*Upstream-Status:\s*(.*?)\s*$', content, re.I | re.M) + if not match: + return 'Missing', '' + full = match.group(1) + token = re.match(r'([^\s\[:]+)', full) + return (token.group(1) if token else 'Unknown'), full + + +def layer_of(path): + return next((p for p in path.parts if p.startswith('meta-')), '(root)') + + +def component_of(path): + # For standard recipes-*//files/foo.patch, choose . + parts = path.parts + for i, part in enumerate(parts[:-1]): + if part.startswith('recipes-') and i + 1 < len(parts) - 1: + return parts[i + 1] + return path.parent.name + + +def main(): + parser = argparse.ArgumentParser( + description='Report Git patch age and upstreaming status from a meta-arm checkout.', + epilog='''Examples: + scripts/patch-history.py + scripts/patch-history.py --verbose --sort age + scripts/patch-history.py --verbose --sort status + scripts/patch-history.py --layer meta-arm-bsp + scripts/patch-history.py --output bsp-patches.csv + +Only Git-tracked .patch and .diff files are scanned. Git rename history is +followed where possible. Ages are calculated from the oldest and newest +commits touching each file, not from patch header dates. By default, a +summary is printed to the terminal. --verbose also lists each patch and +its age. --output writes CSV for spreadsheet use; otherwise no file is +created. No files in the tree are changed.''', + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument('--layer', help='Restrict scanning to this layer/directory relative to repository root') + parser.add_argument('--output', '-o', help='Write CSV to this path (no CSV is written by default)') + parser.add_argument('--sort', choices=['path', 'age', 'updated', 'component', 'status'], default='path', + help='Patch listing and CSV row order (status: grouped alphabetically, oldest first within each status; age and updated: oldest first)') + parser.add_argument('--summary', action='store_true', help='Print aggregate statistics (already the default; retained for compatibility)') + parser.add_argument('--verbose', '-v', action='store_true', help='List every patch with its age and upstream status in the terminal') + args = parser.parse_args() + + try: + root = Path(subprocess.check_output(['git', 'rev-parse', '--show-toplevel'], stderr=subprocess.PIPE, + text=True).strip()).resolve() + except subprocess.CalledProcessError: + parser.error('run this from inside a Git repository') + if args.layer: + target = (root / args.layer).resolve() + if not target.is_relative_to(root) or not target.is_dir(): + parser.error('--layer must name an existing directory inside the repository') + scope = target.relative_to(root).as_posix() + else: + scope = None + + try: + files = git(root, 'ls-files', '-z', '--', *( [scope] if scope else [] )).split(b'\0') + except RuntimeError as exc: + parser.error(str(exc)) + today = dt.date.today() + rows = [] + for encoded in files: + if not encoded: + continue + relative = encoded.decode('utf-8', 'surrogateescape') + if Path(relative).suffix.lower() not in ('.patch', '.diff'): + continue + absolute = root / relative + if not absolute.is_file(): + continue + try: + added, updated, changes = metadata(root, relative) + except RuntimeError as exc: + print(f'warning: history unavailable for {relative}: {exc}', file=sys.stderr) + added, updated, changes = '', '', 0 + status, detail = upstream_status(absolute) + rows.append({ + 'Patch Added': added, + 'Last Updated': updated, + 'Times Changed': changes, + 'Component': component_of(Path(relative)), + 'Upstream Status': status, + 'Patch Age (Days)': (today - dt.date.fromisoformat(added)).days if added else '', + 'Days Since Update': (today - dt.date.fromisoformat(updated)).days if updated else '', + 'Layer': layer_of(Path(relative)), + 'Patch File': relative, + 'Upstream Status Detail': detail, + }) + keys = { + 'path': lambda r: r['Patch File'], + 'age': lambda r: (-(r['Patch Age (Days)'] if isinstance(r['Patch Age (Days)'], int) else -1), r['Patch File']), + 'updated': lambda r: (-(r['Days Since Update'] if isinstance(r['Days Since Update'], int) else -1), r['Patch File']), + 'component': lambda r: (r['Component'], r['Patch File']), + 'status': lambda r: (r['Upstream Status'].lower(), -(r['Patch Age (Days)'] if isinstance(r['Patch Age (Days)'], int) else -1), r['Patch File']), + } + rows.sort(key=keys[args.sort]) + if args.output: + output = Path(args.output) + fields = ['Patch Added', 'Last Updated', 'Times Changed', 'Component', 'Upstream Status', + 'Patch Age (Days)', 'Days Since Update', 'Layer', 'Patch File', 'Upstream Status Detail'] + try: + with output.open('w', newline='', encoding='utf-8') as stream: + writer = csv.DictWriter(stream, fieldnames=fields) + writer.writeheader() + writer.writerows(rows) + except OSError as exc: + parser.exit(1, f'error: cannot write {output}: {exc}\n') + print(f'Wrote {len(rows)} patches to {output}') + + if args.verbose: + print('\nPatch Details (age in days)') + print('===========================') + for row in rows: + age = row['Patch Age (Days)'] + print(f"{str(age) if age != '' else 'n/a':>7} {row['Upstream Status']:<16} {row['Patch File']}") + + if True: # Summary is the default terminal output. + ages = [r['Patch Age (Days)'] for r in rows if isinstance(r['Patch Age (Days)'], int)] + print('\nPatch Maintenance Summary') + print('=========================') + print(f'Total patches: {len(rows)}') + if ages: + print(f'Average patch age: {sum(ages) / len(ages):.0f} days') + print(f'Oldest patch: {max(ages)} days') + print(f'Patches older than 1 year: {sum(a > 365 for a in ages)}') + print(f'Patches older than 2 years: {sum(a > 730 for a in ages)}') + counts = Counter(r['Upstream Status'] for r in rows) + groups = defaultdict(list) + for r in rows: + if isinstance(r['Patch Age (Days)'], int): + groups[r['Upstream Status']].append(r['Patch Age (Days)']) + print('\nUpstream Status (count, average age in days):') + for status in sorted(counts, key=str.lower): + ages_for_status = groups[status] + average = f'{sum(ages_for_status) / len(ages_for_status):.0f}' if ages_for_status else 'n/a' + print(f' {status:<20} {counts[status]:>5} {average:>6}') + + +if __name__ == '__main__': + main()