| Message ID | 20260624-add-pypi-v8-v4-0-ff499f1fd5a5@windriver.com |
|---|---|
| Headers | show |
| Series | bitbake-setup PyPI Packaging | expand |
On Wed, 2026-06-24 at 10:20 -0700, Rob Woolley via lists.openembedded.org wrote: > This is an update to the previous series to rebase on master. It is also > available at: https://github.com/robwoolley/bitbake/tree/b4/add-pypi-v8 > > This version of the series integrates the vendoring module to update > beautifulsoup4, ply, progressbar, simplediff, and typing_extensions. > > I also included Richard's latest commit to add version variables to the > bitbake scripts and dropped the mechanism that I proposed. Hi Rob, Patches 1-5 and 11 look good to me. For patches 6-10 which improve vendoring of upstream packages, I think we need to document things clearly. I've also made a couple of review comments on some of these patches. Looking at the Python vendoring module [1]: Should I use it? This tool has no stability promises -- it has only one intended user: pip. There may be unannounced changes to this codebase at any time, as long as the intended user (i.e. the pip project) is prepared for those changes. As a general rule of thumb, if the project is going to be a PyPI package, it should not use this tool. Many downstream redistributors have policies against this kind of bundling of dependencies, which means that they'll patch your software to debundle it. This can cause various kinds of issues, due to violations of assumptions being made about where the dependencies are available/which versions are being used. These issues result in difficult-to-debug errors, which are fairly difficult to communicate with end users. pip is a very special case with a thorough rationale [2] for vendoring/bundling dependencies with itself. We need to write down why we're using vendoring despite this warning, with a rationale (similar to how pip documents this). We also need to document who's going to run `vendoring sync` and how often. Do we automate this or rely on running it manually? [1]: https://pypi.org/project/vendoring/ [2]: https://pip.pypa.io/en/latest/development/vendoring-policy/#rationale Best regards,
Hi Paul, Thanks for the feedback. I agree with your recommendations to add more explanation about what is going on. I don't think we have had a formal discussion on which tool to use for managing vendored modules. I went with vendoring for 3 reasons: (1) that was the one that Ross had mentioned he used before (2) vendoring was last updated in May 2026 whereas python-vendorize hasn't had an update on PyPI since October 2022 despite changes on GitHub last made in April 2024. (3) it was promoted as having more advanced features and better suited for projects that are more than just a simple CLI My take on the stability promise is that they want to make clear that vendoring is intended for pip itself and they are wisely marking other uses as unsupported. The result of this is that vendoring will continue to be maintained as long as pip uses it. I interpreted the general rule of thumb as recommending against vendoring modules if you want to be in PyPI. However, there is a notable exception for packages that do bootstrapping (albeit they mean Python bootstrapping) and those that necessitate changes to the vendored modules. I feel that bitbake and bitbake-setup could argue a case for both. I asked Claude about what would be involved with switching to python-vendorize. One important difference is that it does not support automatically merging patches against the pristine source. If we went to use it, we would need to add an additional tool to automatically apply the patches. To the question you asked in another e-mail: these patches came directly from the git commit history for bitbake. They were custom changes to the vendored modules that I reapplied after the update. The vendoring module also takes care of license collection and generation of the .pyi stubs. We certainly need the former and the latter may be useful for improving good hygiene going forward. The import rewriting style of python-vendorize is also simply relative (ie. from . import x) whereas vendoring uses absolute imports (eg. import bb._vendor.x). I think the absolute imports would be preferred as they are more readable and produce fewer errors. The versions of each upstream package is currently being stored in the vendor.txt file. The versions represent the published releases in PyPI. Hope this helps clarify things. Happy to continue the discussion and make any changes that are required. Cheers, Rob
This is an update to the previous series to rebase on master. It is also available at: https://github.com/robwoolley/bitbake/tree/b4/add-pypi-v8 This version of the series integrates the vendoring module to update beautifulsoup4, ply, progressbar, simplediff, and typing_extensions. I also included Richard's latest commit to add version variables to the bitbake scripts and dropped the mechanism that I proposed. Here are the steps of the test procedure I used before sending this series: ``` python3 -m venv venv source venv/bin/activate pip install build BB_SKIP_PYPI_TESTS=no bin/bitbake-selftest -v bb.tests.setup.PyPIPackagingTest contrib/pypi/package-bitbake-setup.py -v cd packaging_workspace/ pip install -e . which bitbake-setup bitbake-setup --version bitbake-setup list pip uninstall bitbake-setup python3 -m build pip install dist/bitbake_setup-2.19.0-py3-none-any.whl which bitbake-setup bitbake-setup --version cd $(mktemp -d) bitbake-setup init . /tmp/tmp.cHLz5edo1w/bitbake-builds/oe-nodistro-master/build/init-build-env which bitbake cd /tmp/tmp.cHLz5edo1w/bitbake-builds/oe-nodistro-master/layers/bitbake git remote add bitbake-setup-pypi /ala-lpggp31/rwoolley/bitbake-setup/bitbake git remote update bitbake-setup-pypi git checkout --track bitbake-setup-pypi/b4/add-pypi-v8 cd $BUILDDIR bitbake quilt-native bitbake core-image-minimal bitbake-selftest ``` Signed-off-by: Rob Woolley <rob.woolley@windriver.com> --- Richard Purdie (1): bin/*: Add/improve __version__ processing Rob Woolley (10): bitbake-setup: Add version option pypi: Add PyPI packaging for bitbake-setup pypi: Add packaging documentation for developers gitignore: Ignore temporary staging directory Add pyproject.toml and vendor.txt for vendoring Add vendor patches Update vendorized modules vendor.txt: Add typing_extensions for bs4 Update typing_extensions with vendoring bitbake-setup: Add exception for E402 for bb.__version__ .gitignore | 1 + bin/bitbake | 2 +- bin/bitbake-diffsigs | 7 + bin/bitbake-getvar | 6 + bin/bitbake-hashclient | 8 + bin/bitbake-hashserv | 7 +- bin/bitbake-layers | 6 + bin/bitbake-prserv | 8 +- bin/bitbake-selftest | 7 + bin/bitbake-server | 6 + bin/bitbake-setup | 7 + bin/bitbake-worker | 6 + bin/toaster | 2 + bin/toaster-eventreplay | 7 + contrib/pypi/BUILD.md | 50 + contrib/pypi/LICENSE | 23 + contrib/pypi/README.md | 42 + contrib/pypi/package-bitbake-setup.py | 79 + contrib/pypi/pyproject.toml | 92 + lib/bb/_vendor/__init__.py | 18 - .../{bs4/LICENSE => beautifulsoup4.LICENSE} | 0 lib/bb/_vendor/bs4/AUTHORS | 49 - lib/bb/_vendor/bs4/CHANGELOG | 1839 --------- lib/bb/_vendor/bs4/__init__.py | 856 ++-- lib/bb/_vendor/bs4/_deprecation.py | 80 + lib/bb/_vendor/bs4/_typing.py | 205 + lib/bb/_vendor/bs4/_warnings.py | 98 + lib/bb/_vendor/bs4/builder/__init__.py | 828 ++-- lib/bb/_vendor/bs4/builder/_html5lib.py | 592 +-- lib/bb/_vendor/bs4/builder/_htmlparser.py | 426 +- lib/bb/_vendor/bs4/builder/_lxml.py | 387 +- lib/bb/_vendor/bs4/css.py | 268 +- lib/bb/_vendor/bs4/dammit.py | 1622 +++++--- lib/bb/_vendor/bs4/diagnose.py | 169 +- lib/bb/_vendor/bs4/element.py | 3494 ++++++++++------ lib/bb/_vendor/bs4/exceptions.py | 28 + lib/bb/_vendor/bs4/filter.py | 764 ++++ lib/bb/_vendor/bs4/formatter.py | 211 +- lib/bb/_vendor/bs4/py.typed | 0 lib/bb/_vendor/ply.pyi | 1 + lib/bb/_vendor/ply/__init__.py | 1 + lib/bb/_vendor/ply/cpp.py | 918 +++++ lib/bb/_vendor/ply/ctokens.py | 133 + lib/bb/_vendor/ply/lex.py | 845 ++-- lib/bb/_vendor/ply/yacc.py | 2231 +++++----- lib/bb/_vendor/ply/ygen.py | 74 + lib/bb/_vendor/progressbar.pyi | 1 + lib/bb/_vendor/progressbar/__init__.py | 4 +- lib/bb/_vendor/progressbar/progressbar.py | 21 +- lib/bb/_vendor/progressbar/widgets.py | 14 +- lib/bb/_vendor/simplediff.pyi | 1 + lib/bb/_vendor/simplediff/__init__.py | 2 +- lib/bb/_vendor/typing_extensions.LICENSE | 279 ++ lib/bb/_vendor/typing_extensions.py | 4317 ++++++++++++++++++++ lib/bb/_vendor/typing_extensions.pyi | 1 + lib/bb/main.py | 2 +- lib/bb/tests/setup.py | 289 +- lib/hashserv/__init__.py | 2 + lib/prserv/__init__.py | 2 +- pyproject.toml | 19 + vendor.txt | 5 + vendor/licenses/ply/LICENSE | 32 + .../bs4-0010-lib-bs4-Avoid-soupsieve-warning.patch | 40 + vendor/patches/bs4-remove-double-imports.patch | 23 + ...lib-implement-basic-task-progress-support.patch | 126 + ...-initial-pass-of-SPDX-license-headers-to-.patch | 80 + ...gressbar-accept-value-over-initial-maxval.patch | 55 + ...-Add-self._fd_console-to-use-for-self._ha.patch | 54 + ...ar-Make-bars-show-correctly-with-maxval-0.patch | 32 + ...-knotty-Allow-mixing-log-messages-and-pro.patch | 32 + 70 files changed, 15318 insertions(+), 6618 deletions(-) --- base-commit: 733161c76bd2ac777555d3eac113ff54b4c63ae2 change-id: 20260619-add-pypi-v8-89df7be8c5c2 Best regards, -- Rob Woolley <rob.woolley@windriver.com>