| Message ID | 20230208071251.5412-3-peter.marko@siemens.com |
|---|---|
| State | New |
| Headers | show |
| Series | systemd: split timesync and networkd to packages | expand |
> -----Original Message----- > From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Peter Marko > Sent: den 8 februari 2023 08:13 > To: openembedded-core@lists.openembedded.org > Cc: Peter Marko <peter.marko@siemens.com> > Subject: [OE-core][PATCH 2/3] systemd.bbclass: add non-recursive service packaging > > When service is split to separate package, it will take > all services it depends on. It does not matter is the dependency Change "is" to "if". > is strong or week or if there is rdepends/rrecommends which would Change "week" to "weak". > be the proper way to pull it. > > New variable SYSTEMD_PACKAGES_DONT_RECURSE allows to > skip this recursion for packages which are extracted to a package. > It is mostly useful for catch-all main package and splitting Change "catch-all main package" to either "a catch-all main package" or "catch-all main packages". > additional packages with PACKAGE_BEFORE_PN. > > Signed-off-by: Peter Marko <peter.marko@siemens.com> > --- > documentation/ref-manual/variables.rst | 10 ++++++++++ > meta/classes-recipe/systemd.bbclass | 15 ++++++++------- > 2 files changed, 18 insertions(+), 7 deletions(-) > > diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst > index 725f5c54cc..910b99aed2 100644 > --- a/documentation/ref-manual/variables.rst > +++ b/documentation/ref-manual/variables.rst > @@ -8271,6 +8271,16 @@ system and gives an overview of their function and contents. > :term:`SYSTEMD_PACKAGES`. Overrides not included in :term:`SYSTEMD_PACKAGES` > will be silently ignored. > > + :term:`SYSTEMD_PACKAGES_DONT_RECURSE` Negative variable names should be avoided when possible. I suggest changing this to "SYSTEMD_PACKAGES_RECURSE". See below for how the implementation should be changed. > + By default service files declared in :term:`SYSTEMD_SERVICE` are scanned > + and all related service files are added to parsed package recursively. Change "parsed" to "the parsed". > + > + It allows more readable and future-proof recipes, however it does not work well > + when services are split to separate packages. This variable prevents this behavior. Change "prevents" to "can be used to prevent". > + Here is an example from systemd recipe:: > + > + SYSTEMD_PACKAGES_DONT_RECURSE:${PN}-networkd = "1" > + > :term:`SYSVINIT_ENABLED_GETTYS` > When using > :ref:`SysVinit <dev-manual/new-recipe:enabling system services>`, > diff --git a/meta/classes-recipe/systemd.bbclass b/meta/classes-recipe/systemd.bbclass > index f9c92e6c2a..c8cee482fe 100644 > --- a/meta/classes-recipe/systemd.bbclass > +++ b/meta/classes-recipe/systemd.bbclass > @@ -124,19 +124,19 @@ python systemd_populate_packages() { > return appended > > # Add systemd files to FILES:*-systemd, parse for Also= and follow recursive > - def systemd_add_files_and_parse(pkg_systemd, path, service, keys): > + def systemd_add_files_and_parse(pkg_systemd, path, service, keys, recurse): > # avoid infinite recursion > - if systemd_append_file(pkg_systemd, oe.path.join(path, service)): > + if systemd_append_file(pkg_systemd, oe.path.join(path, service)) and recurse: > fullpath = oe.path.join(d.getVar("D"), path, service) > if service.find('.service') != -1: > # for *.service add *@.service > service_base = service.replace('.service', '') > - systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys) > + systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys, recurse) > if service.find('.socket') != -1: > # for *.socket add *.service and *@.service > service_base = service.replace('.socket', '') > - systemd_add_files_and_parse(pkg_systemd, path, service_base + '.service', keys) > - systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys) > + systemd_add_files_and_parse(pkg_systemd, path, service_base + '.service', keys, recurse) > + systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys, recurse) > for key in keys.split(): > # recurse all dependencies found in keys ('Also';'Conflicts';..) and add to files > cmd = "grep %s %s | sed 's,%s=,,g' | tr ',' '\\n'" % (key, shlex.quote(fullpath), key) > @@ -144,7 +144,7 @@ python systemd_populate_packages() { > line = pipe.readline() > while line: > line = line.replace('\n', '') > - systemd_add_files_and_parse(pkg_systemd, path, line, keys) > + systemd_add_files_and_parse(pkg_systemd, path, line, keys, recurse) > line = pipe.readline() > pipe.close() > > @@ -157,6 +157,7 @@ python systemd_populate_packages() { > keys = 'Also' > # scan for all in SYSTEMD_SERVICE[] > for pkg_systemd in systemd_packages.split(): > + recurse = False if d.getVar('SYSTEMD_PACKAGES_DONT_RECURSE:' + pkg_systemd) else True Change to: recurse = not bb.utils.to_boolean(d.getVar('SYSTEMD_PACKAGES_DONT_RECURSE:' + pkg_systemd)) Or preferably, to avoid the negative variable name, change to: recurse = bb.utils.to_boolean(d.getVar('SYSTEMD_PACKAGES_RECURSE:' + pkg_systemd), True) You should probably also change "d.getVar('SYSTEMD_PACKAGES_RECURSE:' + pkg_systemd)" to "get_package_var(d, 'SYSTEMD_PACKAGES_RECURSE', pkg_systemd)", which would allow setting SYSTEMD_PACKAGES_RECURSE = "0" in the recipe to disable recursing for all packages. In that case, also adapt the documentation accordingly. > for service in get_package_var(d, 'SYSTEMD_SERVICE', pkg_systemd).split(): > path_found = '' > > @@ -178,7 +179,7 @@ python systemd_populate_packages() { > break > > if path_found != '': > - systemd_add_files_and_parse(pkg_systemd, path_found, service, keys) > + systemd_add_files_and_parse(pkg_systemd, path_found, service, keys, recurse) > else: > bb.fatal("Didn't find service unit '{0}', specified in SYSTEMD_SERVICE:{1}. {2}".format( > service, pkg_systemd, "Also looked for service unit '{0}'.".format(base) if base is not None else "")) > -- > 2.30.2 //Peter
On Wed, 2023-02-08 at 08:12 +0100, Peter Marko wrote: > When service is split to separate package, it will take > all services it depends on. It does not matter is the dependency > is strong or week or if there is rdepends/rrecommends which would > be the proper way to pull it. > > New variable SYSTEMD_PACKAGES_DONT_RECURSE allows to > skip this recursion for packages which are extracted to a package. > It is mostly useful for catch-all main package and splitting > additional packages with PACKAGE_BEFORE_PN. > > Signed-off-by: Peter Marko <peter.marko@siemens.com> > --- > documentation/ref-manual/variables.rst | 10 ++++++++++ > meta/classes-recipe/systemd.bbclass | 15 ++++++++------- > 2 files changed, 18 insertions(+), 7 deletions(-) > > diff --git a/documentation/ref-manual/variables.rst > b/documentation/ref-manual/variables.rst > index 725f5c54cc..910b99aed2 100644 > --- a/documentation/ref-manual/variables.rst > +++ b/documentation/ref-manual/variables.rst > @@ -8271,6 +8271,16 @@ system and gives an overview of their function > and contents. > :term:`SYSTEMD_PACKAGES`. Overrides not included in > :term:`SYSTEMD_PACKAGES` > will be silently ignored. > > + :term:`SYSTEMD_PACKAGES_DONT_RECURSE` > + By default service files declared in :term:`SYSTEMD_SERVICE` > are scanned > + and all related service files are added to parsed package > recursively. > + > + It allows more readable and future-proof recipes, however it > does not work well > + when services are split to separate packages. This variable > prevents this behavior. > + Here is an example from systemd recipe:: > + > + SYSTEMD_PACKAGES_DONT_RECURSE:${PN}-networkd = "1" > + > :term:`SYSVINIT_ENABLED_GETTYS` > When using > :ref:`SysVinit <dev-manual/new-recipe:enabling system > services>`, > diff --git a/meta/classes-recipe/systemd.bbclass b/meta/classes- > recipe/systemd.bbclass > index f9c92e6c2a..c8cee482fe 100644 > --- a/meta/classes-recipe/systemd.bbclass > +++ b/meta/classes-recipe/systemd.bbclass > @@ -124,19 +124,19 @@ python systemd_populate_packages() { > return appended > > # Add systemd files to FILES:*-systemd, parse for Also= and > follow recursive > - def systemd_add_files_and_parse(pkg_systemd, path, service, > keys): > + def systemd_add_files_and_parse(pkg_systemd, path, service, > keys, recurse): > # avoid infinite recursion > - if systemd_append_file(pkg_systemd, oe.path.join(path, > service)): > + if systemd_append_file(pkg_systemd, oe.path.join(path, > service)) and recurse: > fullpath = oe.path.join(d.getVar("D"), path, service) > if service.find('.service') != -1: > # for *.service add *@.service > service_base = service.replace('.service', '') > - systemd_add_files_and_parse(pkg_systemd, path, > service_base + '@.service', keys) > + systemd_add_files_and_parse(pkg_systemd, path, > service_base + '@.service', keys, recurse) > if service.find('.socket') != -1: > # for *.socket add *.service and *@.service > service_base = service.replace('.socket', '') > - systemd_add_files_and_parse(pkg_systemd, path, > service_base + '.service', keys) > - systemd_add_files_and_parse(pkg_systemd, path, > service_base + '@.service', keys) > + systemd_add_files_and_parse(pkg_systemd, path, > service_base + '.service', keys, recurse) > + systemd_add_files_and_parse(pkg_systemd, path, > service_base + '@.service', keys, recurse) > for key in keys.split(): > # recurse all dependencies found in keys > ('Also';'Conflicts';..) and add to files > cmd = "grep %s %s | sed 's,%s=,,g' | tr ',' '\\n'" % > (key, shlex.quote(fullpath), key) > @@ -144,7 +144,7 @@ python systemd_populate_packages() { > line = pipe.readline() > while line: > line = line.replace('\n', '') > - systemd_add_files_and_parse(pkg_systemd, path, > line, keys) > + systemd_add_files_and_parse(pkg_systemd, path, > line, keys, recurse) > line = pipe.readline() > pipe.close() > > @@ -157,6 +157,7 @@ python systemd_populate_packages() { > keys = 'Also' There would probably be another solution without introducing a new SYSTEMD_PACKAGES_DONT_RECURSE variable. But that would require a fix for the systemd.bbclass, which is a bit hard to test. To me, it looks like the issue is that systemd.bbclass looks for service files with "Also" in them. Depending on that, it changes the FILES variable of packages. However, according to https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Also= "Also" means that a package with such a service file depends on another package. This means that systemd.bbclass should change the RDEPENDS variable but not the FILES variable. This bug potentially causes service files to end up in the wrong package and that should be fixed. My first thought was to fix the class as it was probably originally intended. Instead of altering the FILES variable it should automatically add packages to the RDEPENDS variable which are referred by service files in this package with "Also". However, since the "Also" could potentially also refer from a service file provided by one recipe to a service file provided by another recipe this saems not to be a complete solution. Maybe the best would be: 1. Fix the invalid packaging by deleting the code from systemd.bbclass which handles the "Also" service files: for key in keys.split(): # recurse all dependencies found in keys ('Also';'Conflicts';..) and add to files cmd = "grep %s %s | sed 's,%s=,,g' | tr ',' '\\n'" % (key, shlex.quote(fullpath), key) pipe = os.popen(cmd, 'r') line = pipe.readline() while line: line = line.replace('\n', '') systemd_add_files_and_parse(pkg_systemd, path, line, keys) bb.warn("After: %s : %s" % (pkg_systemd, path)) line = pipe.readline() pipe.close() 2. This might results in build failures because of not packaged service files. That's not only bad because the automatic packaging was potentially wrong anyway. The build failures need to be fixed manually by adding the service files to the FILES variables of the broken recipes. 3. To prevent the risk of building images with service files refering to packages which are not in the image, a QA check on image level could be added. I'm thinkg about grepping for "After" in all systemd units an verify the referred file is in the image. I'm going to analyze how many recipes in poky would be affected by such a fix. Does that make sense? Regards, Adrian > # scan for all in SYSTEMD_SERVICE[] > for pkg_systemd in systemd_packages.split(): > + recurse = False if > d.getVar('SYSTEMD_PACKAGES_DONT_RECURSE:' + pkg_systemd) else True > for service in get_package_var(d, 'SYSTEMD_SERVICE', > pkg_systemd).split(): > path_found = '' > > @@ -178,7 +179,7 @@ python systemd_populate_packages() { > break > > if path_found != '': > - systemd_add_files_and_parse(pkg_systemd, > path_found, service, keys) > + systemd_add_files_and_parse(pkg_systemd, > path_found, service, keys, recurse) > else: > bb.fatal("Didn't find service unit '{0}', > specified in SYSTEMD_SERVICE:{1}. {2}".format( > service, pkg_systemd, "Also looked for > service unit '{0}'.".format(base) if base is not None else "")) > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#176838): > https://lists.openembedded.org/g/openembedded-core/message/176838 > Mute This Topic: https://lists.openembedded.org/mt/96825675/4454582 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: > https://lists.openembedded.org/g/openembedded-core/unsub [ > adrian.freihofer@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst index 725f5c54cc..910b99aed2 100644 --- a/documentation/ref-manual/variables.rst +++ b/documentation/ref-manual/variables.rst @@ -8271,6 +8271,16 @@ system and gives an overview of their function and contents. :term:`SYSTEMD_PACKAGES`. Overrides not included in :term:`SYSTEMD_PACKAGES` will be silently ignored. + :term:`SYSTEMD_PACKAGES_DONT_RECURSE` + By default service files declared in :term:`SYSTEMD_SERVICE` are scanned + and all related service files are added to parsed package recursively. + + It allows more readable and future-proof recipes, however it does not work well + when services are split to separate packages. This variable prevents this behavior. + Here is an example from systemd recipe:: + + SYSTEMD_PACKAGES_DONT_RECURSE:${PN}-networkd = "1" + :term:`SYSVINIT_ENABLED_GETTYS` When using :ref:`SysVinit <dev-manual/new-recipe:enabling system services>`, diff --git a/meta/classes-recipe/systemd.bbclass b/meta/classes-recipe/systemd.bbclass index f9c92e6c2a..c8cee482fe 100644 --- a/meta/classes-recipe/systemd.bbclass +++ b/meta/classes-recipe/systemd.bbclass @@ -124,19 +124,19 @@ python systemd_populate_packages() { return appended # Add systemd files to FILES:*-systemd, parse for Also= and follow recursive - def systemd_add_files_and_parse(pkg_systemd, path, service, keys): + def systemd_add_files_and_parse(pkg_systemd, path, service, keys, recurse): # avoid infinite recursion - if systemd_append_file(pkg_systemd, oe.path.join(path, service)): + if systemd_append_file(pkg_systemd, oe.path.join(path, service)) and recurse: fullpath = oe.path.join(d.getVar("D"), path, service) if service.find('.service') != -1: # for *.service add *@.service service_base = service.replace('.service', '') - systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys) + systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys, recurse) if service.find('.socket') != -1: # for *.socket add *.service and *@.service service_base = service.replace('.socket', '') - systemd_add_files_and_parse(pkg_systemd, path, service_base + '.service', keys) - systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys) + systemd_add_files_and_parse(pkg_systemd, path, service_base + '.service', keys, recurse) + systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys, recurse) for key in keys.split(): # recurse all dependencies found in keys ('Also';'Conflicts';..) and add to files cmd = "grep %s %s | sed 's,%s=,,g' | tr ',' '\\n'" % (key, shlex.quote(fullpath), key) @@ -144,7 +144,7 @@ python systemd_populate_packages() { line = pipe.readline() while line: line = line.replace('\n', '') - systemd_add_files_and_parse(pkg_systemd, path, line, keys) + systemd_add_files_and_parse(pkg_systemd, path, line, keys, recurse) line = pipe.readline() pipe.close() @@ -157,6 +157,7 @@ python systemd_populate_packages() { keys = 'Also' # scan for all in SYSTEMD_SERVICE[] for pkg_systemd in systemd_packages.split(): + recurse = False if d.getVar('SYSTEMD_PACKAGES_DONT_RECURSE:' + pkg_systemd) else True for service in get_package_var(d, 'SYSTEMD_SERVICE', pkg_systemd).split(): path_found = '' @@ -178,7 +179,7 @@ python systemd_populate_packages() { break if path_found != '': - systemd_add_files_and_parse(pkg_systemd, path_found, service, keys) + systemd_add_files_and_parse(pkg_systemd, path_found, service, keys, recurse) else: bb.fatal("Didn't find service unit '{0}', specified in SYSTEMD_SERVICE:{1}. {2}".format( service, pkg_systemd, "Also looked for service unit '{0}'.".format(base) if base is not None else ""))
When service is split to separate package, it will take all services it depends on. It does not matter is the dependency is strong or week or if there is rdepends/rrecommends which would be the proper way to pull it. New variable SYSTEMD_PACKAGES_DONT_RECURSE allows to skip this recursion for packages which are extracted to a package. It is mostly useful for catch-all main package and splitting additional packages with PACKAGE_BEFORE_PN. Signed-off-by: Peter Marko <peter.marko@siemens.com> --- documentation/ref-manual/variables.rst | 10 ++++++++++ meta/classes-recipe/systemd.bbclass | 15 ++++++++------- 2 files changed, 18 insertions(+), 7 deletions(-)