| Message ID | 20241217-capture-output-compat-v1-1-d8f147a12eea@bootlin.com |
|---|---|
| State | Under Review |
| Headers | show |
| Series | [yocto-docs] set_versions.py: use backward-compatible python argument in run | expand |
Hi Antonin, On 12/17/24 11:35 AM, Antonin Godard via lists.yoctoproject.org wrote: > Some workers on the autobuilder reported the following error: > > File "./set_versions.py", line 102, in <module> > subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], capture_output=True, check=True) > File "/usr/lib64/python3.6/subprocess.py", line 423, in run > with Popen(*popenargs, **kwargs) as process: > TypeError: __init__() got an unexpected keyword argument 'capture_output' > > See https://valkyrie.yoctoproject.org/#/builders/34/builds/86. > > This is because capture_output was introduced in Python 3.7, and some of > the support distributions are still on Python 3.6. Since capture_output > is essentially just setting stdout and stderr to PIPE > (https://github.com/python/cpython/blob/3.13/Lib/subprocess.py#L547), do > it manually here to be compatible with older python versions. > > This is also the case for the "text" parameter, introduced in 3.7 to > alias the universal_newlines parameter. Use "universal_newlines" to be > backward-compatible. > > [ YOCTO #15687 ] > Do we really want to open that can of worms? https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-git-tar-python-make-and-gcc-versions clearly states the minimal supported Python version is 3.8.0 so maybe we should just stop building the docs on those distros that are still on old Python? I'm actually surprised building YP on AlmaLinux 8 works then, is it because of uninative? Looks ok to me otherwise. Cheers, Quentin
Hi Quentin, On Tue Dec 17, 2024 at 12:05 PM CET, Quentin Schulz wrote: > Hi Antonin, > > On 12/17/24 11:35 AM, Antonin Godard via lists.yoctoproject.org wrote: >> Some workers on the autobuilder reported the following error: >> >> File "./set_versions.py", line 102, in <module> >> subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], capture_output=True, check=True) >> File "/usr/lib64/python3.6/subprocess.py", line 423, in run >> with Popen(*popenargs, **kwargs) as process: >> TypeError: __init__() got an unexpected keyword argument 'capture_output' >> >> See https://valkyrie.yoctoproject.org/#/builders/34/builds/86. >> >> This is because capture_output was introduced in Python 3.7, and some of >> the support distributions are still on Python 3.6. Since capture_output >> is essentially just setting stdout and stderr to PIPE >> (https://github.com/python/cpython/blob/3.13/Lib/subprocess.py#L547), do >> it manually here to be compatible with older python versions. >> >> This is also the case for the "text" parameter, introduced in 3.7 to >> alias the universal_newlines parameter. Use "universal_newlines" to be >> backward-compatible. >> >> [ YOCTO #15687 ] >> > > Do we really want to open that can of worms? > > https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-git-tar-python-make-and-gcc-versions > clearly states the minimal supported Python version is 3.8.0 so maybe we > should just stop building the docs on those distros that are still on > old Python? I'm actually surprised building YP on AlmaLinux 8 works > then, is it because of uninative? We still support 3.6 on Kirkstone. The docs build uses a buildtools tarball to get a recent Sphinx version, rsvg-convert, etc. I wouldn't know how this is done for other builds, though. > Looks ok to me otherwise. Thank you, Antonin
Hi Antonin, On 12/18/24 9:19 AM, Antonin Godard wrote: > Hi Quentin, > > On Tue Dec 17, 2024 at 12:05 PM CET, Quentin Schulz wrote: >> Hi Antonin, >> >> On 12/17/24 11:35 AM, Antonin Godard via lists.yoctoproject.org wrote: >>> Some workers on the autobuilder reported the following error: >>> >>> File "./set_versions.py", line 102, in <module> >>> subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], capture_output=True, check=True) >>> File "/usr/lib64/python3.6/subprocess.py", line 423, in run >>> with Popen(*popenargs, **kwargs) as process: >>> TypeError: __init__() got an unexpected keyword argument 'capture_output' >>> >>> See https://valkyrie.yoctoproject.org/#/builders/34/builds/86. >>> >>> This is because capture_output was introduced in Python 3.7, and some of >>> the support distributions are still on Python 3.6. Since capture_output >>> is essentially just setting stdout and stderr to PIPE >>> (https://github.com/python/cpython/blob/3.13/Lib/subprocess.py#L547), do >>> it manually here to be compatible with older python versions. >>> >>> This is also the case for the "text" parameter, introduced in 3.7 to >>> alias the universal_newlines parameter. Use "universal_newlines" to be >>> backward-compatible. >>> >>> [ YOCTO #15687 ] >>> >> >> Do we really want to open that can of worms? >> >> https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-git-tar-python-make-and-gcc-versions >> clearly states the minimal supported Python version is 3.8.0 so maybe we >> should just stop building the docs on those distros that are still on >> old Python? I'm actually surprised building YP on AlmaLinux 8 works >> then, is it because of uninative? > > We still support 3.6 on Kirkstone. The docs build uses a buildtools tarball to I see, so the patch should be merged at least for the branches which do still support 3.6. For the others, not necessarily. The script isn't that big and is not that complex (in terms of Python code, the logic.... is another story :) ) so maybe it's fine. Your decision as maintainer. The diff looks sound to me (and I used it for building on openSUSE within containers), so I guess: Reviewed-by: Quentin Schulz <quentin.schulz@cherry.de> Tested-by: Quentin Schulz <quentin.schulz@cherry.de> # openSUSE Leap 15.4-15.6 Thanks! Quentin
Hi Quentin, On Wed Dec 18, 2024 at 2:39 PM CET, Quentin Schulz wrote: > Hi Antonin, > > On 12/18/24 9:19 AM, Antonin Godard wrote: >> Hi Quentin, >> >> On Tue Dec 17, 2024 at 12:05 PM CET, Quentin Schulz wrote: >>> Hi Antonin, >>> >>> On 12/17/24 11:35 AM, Antonin Godard via lists.yoctoproject.org wrote: >>>> Some workers on the autobuilder reported the following error: >>>> >>>> File "./set_versions.py", line 102, in <module> >>>> subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], capture_output=True, check=True) >>>> File "/usr/lib64/python3.6/subprocess.py", line 423, in run >>>> with Popen(*popenargs, **kwargs) as process: >>>> TypeError: __init__() got an unexpected keyword argument 'capture_output' >>>> >>>> See https://valkyrie.yoctoproject.org/#/builders/34/builds/86. >>>> >>>> This is because capture_output was introduced in Python 3.7, and some of >>>> the support distributions are still on Python 3.6. Since capture_output >>>> is essentially just setting stdout and stderr to PIPE >>>> (https://github.com/python/cpython/blob/3.13/Lib/subprocess.py#L547), do >>>> it manually here to be compatible with older python versions. >>>> >>>> This is also the case for the "text" parameter, introduced in 3.7 to >>>> alias the universal_newlines parameter. Use "universal_newlines" to be >>>> backward-compatible. >>>> >>>> [ YOCTO #15687 ] >>>> >>> >>> Do we really want to open that can of worms? >>> >>> https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-git-tar-python-make-and-gcc-versions >>> clearly states the minimal supported Python version is 3.8.0 so maybe we >>> should just stop building the docs on those distros that are still on >>> old Python? I'm actually surprised building YP on AlmaLinux 8 works >>> then, is it because of uninative? >> >> We still support 3.6 on Kirkstone. The docs build uses a buildtools tarball to > > I see, so the patch should be merged at least for the branches which do > still support 3.6. For the others, not necessarily. The script isn't > that big and is not that complex (in terms of Python code, the logic.... > is another story :) ) so maybe it's fine. Your decision as maintainer. > The diff looks sound to me (and I used it for building on openSUSE > within containers), so I guess: Unfortunately I think it's not that simple - the autobuilder checks out the set_versions.py script from master for every branch it builds (see the run-docs-build script on the yocto-autobuilder-helper repo). I think merging this for every branch is fine though. > Reviewed-by: Quentin Schulz <quentin.schulz@cherry.de> > Tested-by: Quentin Schulz <quentin.schulz@cherry.de> # openSUSE Leap > 15.4-15.6 Thank you! Antonin
diff --git a/documentation/set_versions.py b/documentation/set_versions.py index 376337e7b544829e7f65cb5dcacda635980ee28d..5c55f470d782bb613e619dfb01ca28dea5b54d59 100755 --- a/documentation/set_versions.py +++ b/documentation/set_versions.py @@ -99,12 +99,12 @@ docconfver = None # Test tags exist and inform the user to fetch if not try: - subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], capture_output=True, check=True) + subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True) except subprocess.CalledProcessError: sys.exit("Please run 'git fetch --tags' before building the documentation") # Try and figure out what we are -tags = subprocess.run(["git", "tag", "--points-at", "HEAD"], capture_output=True, text=True).stdout +tags = subprocess.run(["git", "tag", "--points-at", "HEAD"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True).stdout for t in tags.split(): if t.startswith("yocto-"): ourversion = t[6:] @@ -122,7 +122,7 @@ if ourversion: bitbakeversion = bitbake_mapping[i] else: # We're floating on a branch - branch = subprocess.run(["git", "branch", "--show-current"], capture_output=True, text=True).stdout.strip() + branch = subprocess.run(["git", "branch", "--show-current"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True).stdout.strip() ourbranch = branch if branch != "master" and branch not in release_series: # We're not on a known release branch so we have to guess. Compare the numbers of commits @@ -130,7 +130,7 @@ else: possible_branch = None branch_count = 0 for b in itertools.chain(release_series.keys(), ["master"]): - result = subprocess.run(["git", "log", "--format=oneline", "HEAD..origin/" + b], capture_output=True, text=True) + result = subprocess.run(["git", "log", "--format=oneline", "HEAD..origin/" + b], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) if result.returncode == 0: count = result.stdout.count('\n') if not possible_branch or count < branch_count: @@ -153,9 +153,9 @@ else: else: sys.exit("Unknown series for branch %s" % branch) - previoustags = subprocess.run(["git", "tag", "--merged", "HEAD"], capture_output=True, text=True).stdout + previoustags = subprocess.run(["git", "tag", "--merged", "HEAD"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True).stdout previoustags = [t[6:] for t in previoustags.split() if t.startswith("yocto-" + release_series[ourseries])] - futuretags = subprocess.run(["git", "tag", "--merged", ourbranch], capture_output=True, text=True).stdout + futuretags = subprocess.run(["git", "tag", "--merged", ourbranch], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True).stdout futuretags = [t[6:] for t in futuretags.split() if t.startswith("yocto-" + release_series[ourseries])] # Append .999 against the last known version @@ -228,7 +228,7 @@ with open("sphinx-static/switchers.js.in", "r") as r, open("sphinx-static/switch for branch in activereleases + ([ourseries] if ourseries not in activereleases else []): if branch == devbranch: continue - branch_versions = subprocess.run('git tag --list yocto-%s*' % (release_series[branch]), shell=True, capture_output=True, text=True).stdout.split() + branch_versions = subprocess.run('git tag --list yocto-%s*' % (release_series[branch]), shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True).stdout.split() branch_versions = sorted([v.replace("yocto-" + release_series[branch] + ".", "").replace("yocto-" + release_series[branch], "0") for v in branch_versions], key=int) if not branch_versions: continue @@ -273,7 +273,7 @@ def tag_to_semver_like(v): v_maj, v_min, v_patch = v_semver.groups('0') return int("{:0>2}{:0>2}{:0>2}".format(v_maj, v_min, v_patch), 10) -yocto_tags = subprocess.run(["git", "tag", "--list", "--sort=version:refname", "yocto-*"], capture_output=True, text=True).stdout +yocto_tags = subprocess.run(["git", "tag", "--list", "--sort=version:refname", "yocto-*"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True).stdout yocto_tags = sorted(yocto_tags.split() + missing_tags, key=tag_to_semver_like) tags = [tag[6:] for tag in yocto_tags]
Some workers on the autobuilder reported the following error: File "./set_versions.py", line 102, in <module> subprocess.run(["git", "show", "yocto-%s" % release_series[activereleases[0]]], capture_output=True, check=True) File "/usr/lib64/python3.6/subprocess.py", line 423, in run with Popen(*popenargs, **kwargs) as process: TypeError: __init__() got an unexpected keyword argument 'capture_output' See https://valkyrie.yoctoproject.org/#/builders/34/builds/86. This is because capture_output was introduced in Python 3.7, and some of the support distributions are still on Python 3.6. Since capture_output is essentially just setting stdout and stderr to PIPE (https://github.com/python/cpython/blob/3.13/Lib/subprocess.py#L547), do it manually here to be compatible with older python versions. This is also the case for the "text" parameter, introduced in 3.7 to alias the universal_newlines parameter. Use "universal_newlines" to be backward-compatible. [ YOCTO #15687 ] Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> --- Test run here: https://valkyrie.yoctoproject.org/#/builders/34/builds/94. --- documentation/set_versions.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) --- base-commit: ee16c96202e5027d1a8d7e89e11c25f127c78326 change-id: 20241216-capture-output-compat-639e50f53be9 Best regards,