| Message ID | 20250611153319.54778-1-anibal@limonsoftware.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/3] wic: pluginbase workaround on invalid module filenames | expand |
On Wed Jun 11, 2025 at 5:33 PM CEST, Anibal Limon wrote: > Apply a workaround on invalid identifiers replace '-' to '_' to load > the module with valid identifier. > > Let the user know (warning) about plugin doesn't has a valid module > identifier to encourage fix, it gives time before rename oe-core wic > plugins and drop workaround. > > Signed-off-by: Anibal Limon <anibal@limonsoftware.com> > --- Hi Anibal, Thanks for the new series. I believe we have oe oe-selftest errors on x86, while creating image: 2025-06-12 09:33:17,228 - oe-selftest - INFO - overlayfs.OverlayFSEtcRunTimeTests.test_lower_layer_access (subunit.RemotedTestCase) 2025-06-12 09:33:17,228 - oe-selftest - INFO - ... FAIL ... | File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/scripts/lib/wic/plugins/source/bootimg_biosplusefi.py", line 159, in do_configure_partition | cls.__instanciateSubClasses() | ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ | File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/scripts/lib/wic/plugins/source/bootimg_biosplusefi.py", line 108, in __instanciateSubClasses | loader.exec_module(mod) | ~~~~~~~~~~~~~~~~~~^^^^^ | File "<frozen importlib._bootstrap_external>", line 1022, in exec_module | File "<frozen importlib._bootstrap_external>", line 1159, in get_code | File "<frozen importlib._bootstrap_external>", line 1217, in get_data | FileNotFoundError: [Errno 2] No such file or directory: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/scripts/lib/wic/plugins/source/bootimg-pcbios.py' | WARNING: exit code 1 from a shell command. https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/1750 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/1689 Can you have a look at this issue?
Hi Mathieu, On Thu, Jun 12, 2025 at 5:47 AM Mathieu Dubois-Briand < mathieu.dubois-briand@bootlin.com> wrote: > On Wed Jun 11, 2025 at 5:33 PM CEST, Anibal Limon wrote: > > Apply a workaround on invalid identifiers replace '-' to '_' to load > > the module with valid identifier. > > > > Let the user know (warning) about plugin doesn't has a valid module > > identifier to encourage fix, it gives time before rename oe-core wic > > plugins and drop workaround. > > > > Signed-off-by: Anibal Limon <anibal@limonsoftware.com> > > --- > > Hi Anibal, > > Thanks for the new series. > > I believe we have oe oe-selftest errors on x86, while creating image: > > 2025-06-12 09:33:17,228 - oe-selftest - INFO - > overlayfs.OverlayFSEtcRunTimeTests.test_lower_layer_access > (subunit.RemotedTestCase) > 2025-06-12 09:33:17,228 - oe-selftest - INFO - ... FAIL > ... > | File > "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/scripts/lib/wic/plugins/source/bootimg_biosplusefi.py", > line 159, in do_configure_partition > | cls.__instanciateSubClasses() > | ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ > | File > "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/scripts/lib/wic/plugins/source/bootimg_biosplusefi.py", > line 108, in __instanciateSubClasses > | loader.exec_module(mod) > | ~~~~~~~~~~~~~~~~~~^^^^^ > | File "<frozen importlib._bootstrap_external>", line 1022, in > exec_module > | File "<frozen importlib._bootstrap_external>", line 1159, in get_code > | File "<frozen importlib._bootstrap_external>", line 1217, in get_data > | FileNotFoundError: [Errno 2] No such file or directory: > '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/scripts/lib/wic/plugins/source/bootimg-pcbios.py' > | WARNING: exit code 1 from a shell command. > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/1750 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/1689 > > Can you have a look at this issue? > I saw RP is fixing the renaming issues, applying patches upon, to see if it is possible to land the renames. If not, only the initial two patches of the series will be applied. Thanks!, Anibal > > -- > Mathieu Dubois-Briand, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > >
On Wed 2025-06-11 @ 03:33:17 PM, Anibal Limon via lists.openembedded.org wrote: > Apply a workaround on invalid identifiers replace '-' to '_' to load > the module with valid identifier. > > Let the user know (warning) about plugin doesn't has a valid module > identifier to encourage fix, it gives time before rename oe-core wic > plugins and drop workaround. As far as I'm aware, everything in wic currently works, despite these bad names. I'm curious to find out what your use-case is that prompted this patch. Are you hoping to use these plugins with another tool? > Signed-off-by: Anibal Limon <anibal@limonsoftware.com> > --- > scripts/lib/wic/pluginbase.py | 25 +++++++++++++++++++++---- > 1 file changed, 21 insertions(+), 4 deletions(-) > > diff --git a/scripts/lib/wic/pluginbase.py b/scripts/lib/wic/pluginbase.py > index b64568339b..ec515229e7 100644 > --- a/scripts/lib/wic/pluginbase.py > +++ b/scripts/lib/wic/pluginbase.py > @@ -8,6 +8,7 @@ > __all__ = ['ImagerPlugin', 'SourcePlugin'] > > import os > +import sys > import logging > import types > > @@ -20,7 +21,8 @@ from wic.misc import get_bitbake_var > > PLUGIN_TYPES = ["imager", "source"] > > -SCRIPTS_PLUGIN_DIR = ["scripts/lib/wic/plugins", "lib/wic/plugins"] > +SCRIPTS_PLUGIN_WIC_DIR = "wic/plugins" > +SCRIPTS_PLUGIN_BASE_DIR = ["scripts/lib", "lib"] > > logger = logging.getLogger('wic') > > @@ -40,7 +42,9 @@ class PluginMgr: > cls._plugin_dirs = [os.path.join(os.path.dirname(__file__), 'plugins')] > layers = get_bitbake_var("BBLAYERS") or '' > for layer_path in layers.split(): > - for script_plugin_dir in SCRIPTS_PLUGIN_DIR: > + for script_plugin_base_dir in SCRIPTS_PLUGIN_BASE_DIR: > + script_plugin_dir = os.path.join(script_plugin_base_dir, > + SCRIPTS_PLUGIN_WIC_DIR); > path = os.path.join(layer_path, script_plugin_dir) > path = os.path.abspath(os.path.expanduser(path)) > if path not in cls._plugin_dirs and os.path.isdir(path): > @@ -53,13 +57,26 @@ class PluginMgr: > if os.path.isdir(ppath): > for fname in os.listdir(ppath): > if fname.endswith('.py'): > - mname = fname[:-3] > mpath = os.path.join(ppath, fname) > + # If module doesn't have a valid identifier let the > + # user know and apply a workaround replacing '-' > + # with '_' to load module with valid identifier. > + mname = fname[:-3] > + if not mname.isidentifier(): > + logger.warn("invalid identifier at plugin module"\ > + " %s, please fix it.", mpath) > + mname = mname.replace("-", "_") There are several ways that a module name could be incorrect in addition to using dashes (upper vs lower case, periods, etc), and I believe runs of dashes/dots/underscores are to be replaced with a single underscore: https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization So if we do move forward with this, maybe it should be expanded to include all the cases. > + > logger.debug("loading plugin module %s", mpath) > - spec = importlib.util.spec_from_file_location(mname, mpath) > + spec = importlib.util.spec_from_file_location( > + mname, mpath) > module = importlib.util.module_from_spec(spec) > spec.loader.exec_module(module) > > + cname = os.path.join(SCRIPTS_PLUGIN_WIC_DIR, > + ptype, mname).replace('/', '.') > + sys.modules[cname] = module > + > return PLUGINS.get(ptype) > > class PluginMeta(type): > -- > 2.39.5 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#218458): https://lists.openembedded.org/g/openembedded-core/message/218458 > Mute This Topic: https://lists.openembedded.org/mt/113590717/900817 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [twoerner@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Thu, Jun 12, 2025 at 3:02 PM Trevor Woerner <twoerner@gmail.com> wrote: > On Wed 2025-06-11 @ 03:33:17 PM, Anibal Limon via lists.openembedded.org > wrote: > > Apply a workaround on invalid identifiers replace '-' to '_' to load > > the module with valid identifier. > > > > Let the user know (warning) about plugin doesn't has a valid module > > identifier to encourage fix, it gives time before rename oe-core wic > > plugins and drop workaround. > > As far as I'm aware, everything in wic currently works, despite these bad > names. I'm curious to find out what your use-case is that prompted this > patch. > Are you hoping to use these plugins with another tool? > Is to enable extending wic plugins in other layers, see for example meta-virtualization code to enable import, removed after this fix. https://lists.yoctoproject.org/g/meta-virtualization/message/9240?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Acreated%2C%2Cwic%2C20%2C2%2C0%2C112618356 Regards!, Anibal > > > Signed-off-by: Anibal Limon <anibal@limonsoftware.com> > > --- > > scripts/lib/wic/pluginbase.py | 25 +++++++++++++++++++++---- > > 1 file changed, 21 insertions(+), 4 deletions(-) > > > > diff --git a/scripts/lib/wic/pluginbase.py > b/scripts/lib/wic/pluginbase.py > > index b64568339b..ec515229e7 100644 > > --- a/scripts/lib/wic/pluginbase.py > > +++ b/scripts/lib/wic/pluginbase.py > > @@ -8,6 +8,7 @@ > > __all__ = ['ImagerPlugin', 'SourcePlugin'] > > > > import os > > +import sys > > import logging > > import types > > > > @@ -20,7 +21,8 @@ from wic.misc import get_bitbake_var > > > > PLUGIN_TYPES = ["imager", "source"] > > > > -SCRIPTS_PLUGIN_DIR = ["scripts/lib/wic/plugins", "lib/wic/plugins"] > > +SCRIPTS_PLUGIN_WIC_DIR = "wic/plugins" > > +SCRIPTS_PLUGIN_BASE_DIR = ["scripts/lib", "lib"] > > > > logger = logging.getLogger('wic') > > > > @@ -40,7 +42,9 @@ class PluginMgr: > > cls._plugin_dirs = [os.path.join(os.path.dirname(__file__), > 'plugins')] > > layers = get_bitbake_var("BBLAYERS") or '' > > for layer_path in layers.split(): > > - for script_plugin_dir in SCRIPTS_PLUGIN_DIR: > > + for script_plugin_base_dir in SCRIPTS_PLUGIN_BASE_DIR: > > + script_plugin_dir = > os.path.join(script_plugin_base_dir, > > + > SCRIPTS_PLUGIN_WIC_DIR); > > path = os.path.join(layer_path, script_plugin_dir) > > path = os.path.abspath(os.path.expanduser(path)) > > if path not in cls._plugin_dirs and > os.path.isdir(path): > > @@ -53,13 +57,26 @@ class PluginMgr: > > if os.path.isdir(ppath): > > for fname in os.listdir(ppath): > > if fname.endswith('.py'): > > - mname = fname[:-3] > > mpath = os.path.join(ppath, fname) > > + # If module doesn't have a valid identifier > let the > > + # user know and apply a workaround > replacing '-' > > + # with '_' to load module with valid > identifier. > > + mname = fname[:-3] > > + if not mname.isidentifier(): > > + logger.warn("invalid identifier at > plugin module"\ > > + " %s, please fix it.", > mpath) > > + mname = mname.replace("-", "_") > > There are several ways that a module name could be incorrect in addition to > using dashes (upper vs lower case, periods, etc), and I believe runs of > dashes/dots/underscores are to be replaced with a single underscore: > > https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization > > So if we do move forward with this, maybe it should be expanded to include > all > the cases. > > > + > > logger.debug("loading plugin module %s", > mpath) > > - spec = > importlib.util.spec_from_file_location(mname, mpath) > > + spec = > importlib.util.spec_from_file_location( > > + mname, mpath) > > module = > importlib.util.module_from_spec(spec) > > spec.loader.exec_module(module) > > > > + cname = os.path.join(SCRIPTS_PLUGIN_WIC_DIR, > > + ptype, > mname).replace('/', '.') > > + sys.modules[cname] = module > > + > > return PLUGINS.get(ptype) > > > > class PluginMeta(type): > > -- > > 2.39.5 > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#218458): > https://lists.openembedded.org/g/openembedded-core/message/218458 > > Mute This Topic: https://lists.openembedded.org/mt/113590717/900817 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > twoerner@gmail.com] > > -=-=-=-=-=-=-=-=-=-=-=- > > > >
diff --git a/scripts/lib/wic/pluginbase.py b/scripts/lib/wic/pluginbase.py index b64568339b..ec515229e7 100644 --- a/scripts/lib/wic/pluginbase.py +++ b/scripts/lib/wic/pluginbase.py @@ -8,6 +8,7 @@ __all__ = ['ImagerPlugin', 'SourcePlugin'] import os +import sys import logging import types @@ -20,7 +21,8 @@ from wic.misc import get_bitbake_var PLUGIN_TYPES = ["imager", "source"] -SCRIPTS_PLUGIN_DIR = ["scripts/lib/wic/plugins", "lib/wic/plugins"] +SCRIPTS_PLUGIN_WIC_DIR = "wic/plugins" +SCRIPTS_PLUGIN_BASE_DIR = ["scripts/lib", "lib"] logger = logging.getLogger('wic') @@ -40,7 +42,9 @@ class PluginMgr: cls._plugin_dirs = [os.path.join(os.path.dirname(__file__), 'plugins')] layers = get_bitbake_var("BBLAYERS") or '' for layer_path in layers.split(): - for script_plugin_dir in SCRIPTS_PLUGIN_DIR: + for script_plugin_base_dir in SCRIPTS_PLUGIN_BASE_DIR: + script_plugin_dir = os.path.join(script_plugin_base_dir, + SCRIPTS_PLUGIN_WIC_DIR); path = os.path.join(layer_path, script_plugin_dir) path = os.path.abspath(os.path.expanduser(path)) if path not in cls._plugin_dirs and os.path.isdir(path): @@ -53,13 +57,26 @@ class PluginMgr: if os.path.isdir(ppath): for fname in os.listdir(ppath): if fname.endswith('.py'): - mname = fname[:-3] mpath = os.path.join(ppath, fname) + # If module doesn't have a valid identifier let the + # user know and apply a workaround replacing '-' + # with '_' to load module with valid identifier. + mname = fname[:-3] + if not mname.isidentifier(): + logger.warn("invalid identifier at plugin module"\ + " %s, please fix it.", mpath) + mname = mname.replace("-", "_") + logger.debug("loading plugin module %s", mpath) - spec = importlib.util.spec_from_file_location(mname, mpath) + spec = importlib.util.spec_from_file_location( + mname, mpath) module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) + cname = os.path.join(SCRIPTS_PLUGIN_WIC_DIR, + ptype, mname).replace('/', '.') + sys.modules[cname] = module + return PLUGINS.get(ptype) class PluginMeta(type):
Apply a workaround on invalid identifiers replace '-' to '_' to load the module with valid identifier. Let the user know (warning) about plugin doesn't has a valid module identifier to encourage fix, it gives time before rename oe-core wic plugins and drop workaround. Signed-off-by: Anibal Limon <anibal@limonsoftware.com> --- scripts/lib/wic/pluginbase.py | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)