| Message ID | 20250610153125.4858-1-gavrosc@yahoo.com |
|---|---|
| State | New |
| Headers | show |
| Series | host_dmesg_logger: dmesg diff on BuildStarted and BuildCompleted | expand |
On 2025-06-10 11:31 a.m., Christos Gavros wrote: > This class is used to capture the host dmesg output > when BuildStarted and BuildCompleted. Then computes > a diff and prints the result. I'd like to see a few words about when this is expected to be useful and what the drawbacks are. In the bug Alex said: For debugging purposes, it would be nice to record the host dmesg at the start of bitbake and the end of the build to be able to do a diff. This would probably uncover interesting failures that are currently hidden/ignored by the underlying build system. We know there are gcc/ld segfaults and sometime qemu segfaults. The obvious downside (to most people) is that it will capture unrelated dmesg output should the build be running on a shared machine and that some distro limit dmesg usage to root/sudo. Also please mention usage: To enable dmesg logging add the following to your local.conf: INHERIT += "host_dmesg_logger" Now that I type that, it's really a host_dmesg_diff_logger but maybe we don't care. I didn't actually try it yet ... but LGTM other than the comments above and some whitespace issue mentioned below. Thanks for the patch Christos! ../Randy > Fixes [YOCTO #15557] > > CC: Yoann Congal<yoann.congal@smile.fr> > CC: Randy MacLeod<randy.macleod@windriver.com> > CC: Alexandre Belloni<alexandre.belloni@bootlin.com> > Signed-off-by: Christos Gavros<gavrosc@yahoo.com> > --- > meta/classes-global/host_dmesg_logger.bbclass | 86 +++++++++++++++++++ > 1 file changed, 86 insertions(+) > create mode 100644 meta/classes-global/host_dmesg_logger.bbclass > > diff --git a/meta/classes-global/host_dmesg_logger.bbclass b/meta/classes-global/host_dmesg_logger.bbclass > new file mode 100644 > index 0000000000..0caf5f0612 > --- /dev/null > +++ b/meta/classes-global/host_dmesg_logger.bbclass > @@ -0,0 +1,86 @@ > +# > +# Copyright OpenEmbedded Contributors > +# > +# SPDX-License-Identifier: MIT > +# > + > +# This class captures the host's dmesg output at the start and completion of a build. > +# It stores these outputs in a variable and upon build completion, computes a diff between > +# the two logs to detect any changes that occurred during the build process. It prints the diff > +# using bb.warn(). The user needs to have privileges to run dmesg. > + > +def append_log(msg, d): > + """ > + Appends a log message to variable '_dmesg_log' > + """ > + current_log = d.getVar('_dmesg_log') or '' > + current_log += msg + '\n' > + d.setVar('_dmesg_log', current_log) > + > +def capture_dmesg(): > + """ > + Returns the current output of dmesg command > + """ > + import subprocess > + > + try: > + result = subprocess.run(['dmesg'], capture_output=True, text=True, check=False) > + if result.returncode != 0: > + return f"Error running dmesg: {result.stderr.strip()}" > + return result.stdout > + except Exception as e: > + return f"Exception running dmesg: {str(e)}" > + > + > +addhandler hostdmesglogger_eventhandler > +python hostdmesglogger_eventhandler() { > + import difflib > + > + start_marker = "=== BuildStarted dmesg ===" > + end_marker = "=== BuildCompleted dmesg ===" > + diff_marker = "=== dmesg diff ===" > + > + # execute dmesg when BuildStarted event is fired > + if isinstance(e, bb.event.BuildStarted): > + dmesg_output = capture_dmesg() > + if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"): > + bb.warn(dmesg_output) > + else: > + append_log(start_marker, d) > + append_log(dmesg_output, d) > + > + # execute dmesg when BuildCompleted event is fired > + if isinstance(e, bb.event.BuildCompleted): > + dmesg_output = capture_dmesg() > + if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"): > + bb.warn(dmesg_output) > + else: > + append_log(end_marker, d) > + append_log(dmesg_output, d) > + > + content = d.getVar('_dmesg_log') or '' > + if start_marker in content and end_marker in content: > + start_dmesg = content.split(start_marker)[1].split(end_marker)[0] > + end_dmesg = content.split(end_marker)[1] > + > + start_lines = start_dmesg.strip().splitlines() > + end_lines = end_dmesg.strip().splitlines() > + > + # generating diff between BuildStarted and BuildCompleted dmesg outputs > + diff = list(difflib.unified_diff( > + start_lines, end_lines, > + fromfile='dmesg_start', > + tofile='dmesg_end', > + lineterm='' Add spaces around "=" ? > + )) > + > + append_log(diff_marker, d) > + if diff: > + for line in diff: > + bb.warn(line) > + else: > + bb.warn("No differences in dmesg output.") > + else: > + bb.warn("Could not find both dmesg sections for diff.") > +} > +hostdmesglogger_eventhandler[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted"
On 2025-06-18 21:40, Randy MacLeod wrote: > On 2025-06-10 11:31 a.m., Christos Gavros wrote: >> This class is used to capture the host dmesg output >> when BuildStarted and BuildCompleted. Then computes >> a diff and prints the result. > > I'd like to see a few words about when this is expected to be useful > and what the drawbacks are. > > In the bug Alex said: > > For debugging purposes, it would be nice to record the host dmesg > at the start of bitbake and the end of the build to be able to do a diff. > This would probably uncover interesting failures that are currently > hidden/ignored by the underlying build system. > We know there are gcc/ld segfaults and sometime qemu segfaults. > > The obvious downside (to most people) is that it will capture > unrelated dmesg output should > the build be running on a shared machine and that some distro limit > dmesg usage to root/sudo. > i will extend the description of the class and send v2. > > Also please mention usage: > > To enable dmesg logging add the following to your local.conf: > > INHERIT += "host_dmesg_logger" > > i will extend the description of the class and send v2. > > Now that I type that, it's really a host_dmesg_diff_logger but maybe > we don't care. > I kept the name that Richard suggested in the mail. > > I didn't actually try it yet ... but LGTM other than the comments > above and some whitespace issue > mentioned below. > > Thanks for the patch Christos! > I will fix them in v2 > > ../Randy > > >> Fixes [YOCTO #15557] >> >> CC: Yoann Congal<yoann.congal@smile.fr> >> CC: Randy MacLeod<randy.macleod@windriver.com> >> CC: Alexandre Belloni<alexandre.belloni@bootlin.com> >> Signed-off-by: Christos Gavros<gavrosc@yahoo.com> >> --- >> meta/classes-global/host_dmesg_logger.bbclass | 86 +++++++++++++++++++ >> 1 file changed, 86 insertions(+) >> create mode 100644 meta/classes-global/host_dmesg_logger.bbclass >> >> diff --git a/meta/classes-global/host_dmesg_logger.bbclass b/meta/classes-global/host_dmesg_logger.bbclass >> new file mode 100644 >> index 0000000000..0caf5f0612 >> --- /dev/null >> +++ b/meta/classes-global/host_dmesg_logger.bbclass >> @@ -0,0 +1,86 @@ >> +# >> +# Copyright OpenEmbedded Contributors >> +# >> +# SPDX-License-Identifier: MIT >> +# >> + >> +# This class captures the host's dmesg output at the start and completion of a build. >> +# It stores these outputs in a variable and upon build completion, computes a diff between >> +# the two logs to detect any changes that occurred during the build process. It prints the diff >> +# using bb.warn(). The user needs to have privileges to run dmesg. >> + >> +def append_log(msg, d): >> + """ >> + Appends a log message to variable '_dmesg_log' >> + """ >> + current_log = d.getVar('_dmesg_log') or '' >> + current_log += msg + '\n' >> + d.setVar('_dmesg_log', current_log) >> + >> +def capture_dmesg(): >> + """ >> + Returns the current output of dmesg command >> + """ >> + import subprocess >> + >> + try: >> + result = subprocess.run(['dmesg'], capture_output=True, text=True, check=False) >> + if result.returncode != 0: >> + return f"Error running dmesg: {result.stderr.strip()}" >> + return result.stdout >> + except Exception as e: >> + return f"Exception running dmesg: {str(e)}" >> + >> + >> +addhandler hostdmesglogger_eventhandler >> +python hostdmesglogger_eventhandler() { >> + import difflib >> + >> + start_marker = "=== BuildStarted dmesg ===" >> + end_marker = "=== BuildCompleted dmesg ===" >> + diff_marker = "=== dmesg diff ===" >> + >> + # execute dmesg when BuildStarted event is fired >> + if isinstance(e, bb.event.BuildStarted): >> + dmesg_output = capture_dmesg() >> + if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"): >> + bb.warn(dmesg_output) >> + else: >> + append_log(start_marker, d) >> + append_log(dmesg_output, d) >> + >> + # execute dmesg when BuildCompleted event is fired >> + if isinstance(e, bb.event.BuildCompleted): >> + dmesg_output = capture_dmesg() >> + if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"): >> + bb.warn(dmesg_output) >> + else: >> + append_log(end_marker, d) >> + append_log(dmesg_output, d) >> + >> + content = d.getVar('_dmesg_log') or '' >> + if start_marker in content and end_marker in content: >> + start_dmesg = content.split(start_marker)[1].split(end_marker)[0] >> + end_dmesg = content.split(end_marker)[1] >> + >> + start_lines = start_dmesg.strip().splitlines() >> + end_lines = end_dmesg.strip().splitlines() >> + >> + # generating diff between BuildStarted and BuildCompleted dmesg outputs >> + diff = list(difflib.unified_diff( >> + start_lines, end_lines, >> + fromfile='dmesg_start', >> + tofile='dmesg_end', >> + lineterm='' > Add spaces around "=" ? >> + )) >> + >> + append_log(diff_marker, d) >> + if diff: >> + for line in diff: >> + bb.warn(line) >> + else: >> + bb.warn("No differences in dmesg output.") >> + else: >> + bb.warn("Could not find both dmesg sections for diff.") >> +} >> +hostdmesglogger_eventhandler[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted" > > > -- > # Randy MacLeod > # Wind River Linux
diff --git a/meta/classes-global/host_dmesg_logger.bbclass b/meta/classes-global/host_dmesg_logger.bbclass new file mode 100644 index 0000000000..0caf5f0612 --- /dev/null +++ b/meta/classes-global/host_dmesg_logger.bbclass @@ -0,0 +1,86 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +# This class captures the host's dmesg output at the start and completion of a build. +# It stores these outputs in a variable and upon build completion, computes a diff between +# the two logs to detect any changes that occurred during the build process. It prints the diff +# using bb.warn(). The user needs to have privileges to run dmesg. + +def append_log(msg, d): + """ + Appends a log message to variable '_dmesg_log' + """ + current_log = d.getVar('_dmesg_log') or '' + current_log += msg + '\n' + d.setVar('_dmesg_log', current_log) + +def capture_dmesg(): + """ + Returns the current output of dmesg command + """ + import subprocess + + try: + result = subprocess.run(['dmesg'], capture_output=True, text=True, check=False) + if result.returncode != 0: + return f"Error running dmesg: {result.stderr.strip()}" + return result.stdout + except Exception as e: + return f"Exception running dmesg: {str(e)}" + + +addhandler hostdmesglogger_eventhandler +python hostdmesglogger_eventhandler() { + import difflib + + start_marker = "=== BuildStarted dmesg ===" + end_marker = "=== BuildCompleted dmesg ===" + diff_marker = "=== dmesg diff ===" + + # execute dmesg when BuildStarted event is fired + if isinstance(e, bb.event.BuildStarted): + dmesg_output = capture_dmesg() + if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"): + bb.warn(dmesg_output) + else: + append_log(start_marker, d) + append_log(dmesg_output, d) + + # execute dmesg when BuildCompleted event is fired + if isinstance(e, bb.event.BuildCompleted): + dmesg_output = capture_dmesg() + if dmesg_output.startswith("Error running dmesg:") or dmesg_output.startswith("Exception running dmesg:"): + bb.warn(dmesg_output) + else: + append_log(end_marker, d) + append_log(dmesg_output, d) + + content = d.getVar('_dmesg_log') or '' + if start_marker in content and end_marker in content: + start_dmesg = content.split(start_marker)[1].split(end_marker)[0] + end_dmesg = content.split(end_marker)[1] + + start_lines = start_dmesg.strip().splitlines() + end_lines = end_dmesg.strip().splitlines() + + # generating diff between BuildStarted and BuildCompleted dmesg outputs + diff = list(difflib.unified_diff( + start_lines, end_lines, + fromfile='dmesg_start', + tofile='dmesg_end', + lineterm='' + )) + + append_log(diff_marker, d) + if diff: + for line in diff: + bb.warn(line) + else: + bb.warn("No differences in dmesg output.") + else: + bb.warn("Could not find both dmesg sections for diff.") +} +hostdmesglogger_eventhandler[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted"
This class is used to capture the host dmesg output when BuildStarted and BuildCompleted. Then computes a diff and prints the result. Fixes [YOCTO #15557] CC: Yoann Congal <yoann.congal@smile.fr> CC: Randy MacLeod <randy.macleod@windriver.com> CC: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Christos Gavros <gavrosc@yahoo.com> --- meta/classes-global/host_dmesg_logger.bbclass | 86 +++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 meta/classes-global/host_dmesg_logger.bbclass