From patchwork Sun Nov 15 11:33:54 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Dubois-Briand X-Patchwork-Id: 104235 Received: from smtpout-02.galae.net (smtpout-02.galae.net [185.246.84.56]) by mx.groups.io with SMTP id smtpd.msgproc01-g2.5891.1794742442279273059 for ; Sun, 15 Nov 2026 03:34:02 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=dkim header.b=qOnFb1Ac; spf=pass (domain: bootlin.com, ip: 185.246.84.56, mailfrom: mathieu.dubois-briand@bootlin.com) Received: from smtpout-01.galae.net (smtpout-01.galae.net [212.83.139.233]) by smtpout-02.galae.net (Postfix) with ESMTPS id 9E30C1A166F for ; Sun, 15 Nov 2026 11:34:00 +0000 (UTC) Received: from mail.galae.net (mail.galae.net [212.83.136.155]) by smtpout-01.galae.net (Postfix) with ESMTPS id 6BE1860238 for ; Sun, 15 Nov 2026 11:34:00 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id A51E711BC1170; Sun, 15 Nov 2026 12:33:59 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=dkim; t=1794742439; h=from:subject:date:message-id:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=6OAfOiPh94rIzQD7Ken3qafTn1znEBkfdc3rd7fD5o8=; b=qOnFb1AcQl3Mul4t07HU81gMVSRVlREp6vXIaRt8E98TkMtiq2DD5qywzU/2JWBSUt6zUM cGzgq/rx72Df3kxDeat17eggZUDhKKM7TmunsxJ9W26jpQ0TkMXKsBTtc2gwGaLyaPMKh3 ov20m+2eCwWrw2Rka7k8k9X+b7IXjcGZ9IDC2MFiSvDCP3Zmit2P1MCwF3Lg+SlJp85n8S vzMTJtAVxvleQlwebIqHzv6w6ejquu2nEsCXL2fq7+wl+9HFF2bsI/ZNjU0AFjaLw8ihFr YGoqVmdj6FthmGJLNvdM0NvZ8M27YecvBr7eQXaDfs2/tJUVklHKojU739lJxA== From: Mathieu Dubois-Briand Date: Sun, 15 Nov 2026 12:33:54 +0100 Subject: [PATCH RFC v2 1/3] sstate: Replay warnings and errors when restoring from sstate MIME-Version: 1.0 Message-Id: <20261115-mathieu-debug-replay-v4-v2-1-8fbfc07058b1@bootlin.com> References: <20261115-mathieu-debug-replay-v4-v2-0-8fbfc07058b1@bootlin.com> In-Reply-To: <20261115-mathieu-debug-replay-v4-v2-0-8fbfc07058b1@bootlin.com> To: openembedded-core@lists.openembedded.org Cc: Thomas Petazzoni , Mathieu Dubois-Briand X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1794742439; l=5871; i=mathieu.dubois-briand@bootlin.com; s=20241219; h=from:subject:message-id; bh=WZxP2FUMNaSK9vidP6wtE7aLtSvn7miRA7NQKbG/yKg=; b=muThwQIHX3Sz3J6b+KSOUYBMgY/zGQw/w4uJqB4yQtwJxALkTfVAHBt2v8k1m1MOtSQSUOnpo vy5XIZ2eeWvAwSSbYfWMjIoaHAUlShRmuIZVss+aR44/KGUnh3Odxh8 X-Developer-Key: i=mathieu.dubois-briand@bootlin.com; a=ed25519; pk=1PVTmzPXfKvDwcPUzG0aqdGoKZJA3b9s+3DqRlm0Lww= X-Last-TLS-Session-Version: TLSv1.3 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Sun, 15 Nov 2026 11:34:12 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/250274 Save the messages logged by an sstate task into its sstate package (sstate-log.json, with build-specific paths replaced by @VAR@ placeholders) and replay them when the package is restored from the sstate cache, substituting the paths of the build doing the replay. Errors are saved too, since bb.error() can be called without failing the task, so warnings which previously only appeared on the first build (e.g. QA messages) are now shown again on sstate reuse. AI-Generated: Uses opencode (DeepSeek V4 Flash) Signed-off-by: Mathieu Dubois-Briand --- meta/classes-global/sstate.bbclass | 91 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 88 insertions(+), 3 deletions(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index c6d945918232..ed88dae17202 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -101,6 +101,12 @@ SSTATEPREINSTFUNCS = "" SSTATEPOSTUNPACKFUNCS = "sstate_hardcode_path_unpack" EXTRA_STAGING_FIXMES ?= "HOSTTOOLS_DIR" +# Build-specific paths substituted in replayed task log messages. Keep the +# order longest-path-first: a shorter value is a prefix of the longer ones +# (e.g. TOPDIR is a prefix of TMPDIR, HOME of both) and must be substituted +# after them, or it would match inside their values. +SSTATE_LOG_VARS ?= "TMPDIR DL_DIR SSTATE_DIR TOPDIR COREBASE HOME" + # Check whether sstate exists for tasks that support sstate and are in the # locked signatures file. SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK ?= 'error' @@ -200,6 +206,73 @@ def sstate_state_fromvars(d, task = None): ss['fixmedir'] = fixmedir return ss +def sstate_log_strip_paths(d, message): + import re + + # Replace paths in log messages by placeholders, so we are not leaking + # build paths and can show consistent paths on replay. + # Escape literal '@' first. + message = message.replace('@', r'\@') + for var in d.getVar('SSTATE_LOG_VARS').split(): + value = (d.getVar(var) or '').rstrip('/') + if value: + message = re.sub(re.escape(value) + r'(?!\w)', '@%s@' % var, + message) + + return message + +def sstate_log_expand_paths(d, message): + # Add back paths into the replayed log messages + for var in d.getVar('SSTATE_LOG_VARS').split(): + value = (d.getVar(var) or '').rstrip('/') + if value: + message = message.replace('@%s@' % var, value) + + message = message.replace(r'\@', '@') + + return message + +def sstate_log_save(d, record): + # Raw logstore entry (a [levelno, message] pair) -> archived record + # (dict), with build paths replaced by placeholders. Inverse of + # sstate_log_expand. + levelno, message = record + return {'levelno': levelno, 'message': sstate_log_strip_paths(d, message)} + +def sstate_log_emit(d, logentry): + levelno = logentry.get('levelno', 0) + if levelno < bb.msg.BBLogFormatter.WARNONCE or "message" not in logentry: + return + message = sstate_log_expand_paths(d, logentry['message']) + if levelno >= bb.msg.BBLogFormatter.ERROR: + bb.error(message) + elif levelno == bb.msg.BBLogFormatter.ERRORONCE: + bb.erroronce(message) + elif levelno == bb.msg.BBLogFormatter.WARNONCE: + bb.warnonce(message) + else: + bb.warn(message) + +python sstate_save_task_log () { + import json + + # Save the messages logged by this task, so they can be replayed when the + # task is restored from sstate. The records are written by bitbake right + # after the task body, before the postfuncs. + mytask = d.getVar('BB_RUNTASK') + recordsfn = os.path.join(d.getVar('T'), 'logrecords.%s.json' % mytask) + if not os.path.exists(recordsfn): + return + with open(recordsfn) as f: + records = json.load(f) + + tasks = {mytask: [sstate_log_save(d, record) for record in records]} + + logfn = os.path.join(d.getVar('SSTATE_BUILDDIR'), "sstate-log.json") + with open(logfn, 'w') as f: + json.dump({'tasks': tasks}, f) +} + def sstate_add(ss, source, dest, d): if not source.endswith("/"): source = source + "/" @@ -387,11 +460,12 @@ def sstate_installpkg(ss, d): # All hooks should run in the SSTATE_INSTDIR bb.build.exec_func(f, d, (sstateinst,)) - return sstate_installpkgdir(ss, d) + return sstate_installpkgdir(ss, d, replay=True) -def sstate_installpkgdir(ss, d): +def sstate_installpkgdir(ss, d, replay=False): import oe.path import subprocess + import json sstateinst = d.getVar("SSTATE_INSTDIR") d.setVar('SSTATE_FIXMEDIR', ss['fixmedir']) @@ -402,6 +476,17 @@ def sstate_installpkgdir(ss, d): sstate_install(ss, d) + if replay: + logfp = os.path.join(sstateinst, "sstate-log.json") + if os.path.exists(logfp): + with open(logfp, 'r') as logfile: + logdata = json.load(logfile) + records = logdata.get('tasks', {}).get(d.getVar('BB_RUNTASK').removesuffix('_setscene'), []) + for logentry in records: + sstate_log_emit(d, logentry) + + bb.utils.remove(logfp) + return True python sstate_hardcode_path_unpack () { @@ -686,7 +771,7 @@ def sstate_package(ss, d): if d.getVar('SSTATE_SKIP_CREATION') == '1': return - sstate_create_package = ['sstate_report_unihash', 'sstate_create_and_sign_package'] + sstate_create_package = ['sstate_report_unihash', 'sstate_save_task_log', 'sstate_create_and_sign_package'] for f in (d.getVar('SSTATECREATEFUNCS') or '').split() + \ sstate_create_package + \ From patchwork Sun Nov 15 11:33:55 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Dubois-Briand X-Patchwork-Id: 104236 Received: from smtpout-02.galae.net (smtpout-02.galae.net [185.246.84.56]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.5875.1794742442920271422 for ; Sun, 15 Nov 2026 03:34:03 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=dkim header.b=SHxhwZhz; spf=pass (domain: bootlin.com, ip: 185.246.84.56, mailfrom: mathieu.dubois-briand@bootlin.com) Received: from smtpout-01.galae.net (smtpout-01.galae.net [212.83.139.233]) by smtpout-02.galae.net (Postfix) with ESMTPS id 4CA8E1A1670 for ; Sun, 15 Nov 2026 11:34:01 +0000 (UTC) Received: from mail.galae.net (mail.galae.net [212.83.136.155]) by smtpout-01.galae.net (Postfix) with ESMTPS id 1AB0660238 for ; Sun, 15 Nov 2026 11:34:01 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id 42B6811BC1D63; Sun, 15 Nov 2026 12:34:00 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=dkim; t=1794742440; h=from:subject:date:message-id:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=rx+hOStkKyYo63TIvdofFhjS2Ad/e8BqrgoroiQdx2U=; b=SHxhwZhzlZshgretdBcovdVuJ0inaJup7COQMLcBlssfAygRH+vyQ74K7Gntvyk6uKwMHz 3J2eUB95ktK5WBCkxqVOP3CqXHEhiv+VeiOpQADg5ct5tVa6nzc9uEQzi358ER2C6fQ/Dp LM+cfCcsI/XZLsq5tC4QRpr1+j+SnhD/rqmo2I0/UrC1Ng96DbMmC4JQTOskXiveAKJAm3 QZuGIRriCWOyTcBMGzJtkruRdbsWKTSGODR0qDD6eQjoWs2tB5OPsYwVdgLin0r2hkNqxm 20IuDS7KPrNkIbnbgFTtl0rLwV2zG8Yy8jMqLiyM9Lght0eLFz9lISAvlUmNqA== From: Mathieu Dubois-Briand Date: Sun, 15 Nov 2026 12:33:55 +0100 Subject: [PATCH RFC v2 2/3] sstate: Replay the records of tasks covered by setscene MIME-Version: 1.0 Message-Id: <20261115-mathieu-debug-replay-v4-v2-2-8fbfc07058b1@bootlin.com> References: <20261115-mathieu-debug-replay-v4-v2-0-8fbfc07058b1@bootlin.com> In-Reply-To: <20261115-mathieu-debug-replay-v4-v2-0-8fbfc07058b1@bootlin.com> To: openembedded-core@lists.openembedded.org Cc: Thomas Petazzoni , Mathieu Dubois-Briand X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1794742439; l=6756; i=mathieu.dubois-briand@bootlin.com; s=20241219; h=from:subject:message-id; bh=N9Z1ouAio57Rx1IqLfLN2ZQBpc64T02YBVrX9Mrz7IM=; b=iV1rjFkqFfOsVo8W+ywjpa/SrMP1NZZ8U4+SqJezKo+eBKhDeR50stx68zSv/zwzw50fcfoAf WpueBfz9TOUD/fROxEt5+25muGJ1OA4pjSMlZCytghDuPsIVtuYWin/ X-Developer-Key: i=mathieu.dubois-briand@bootlin.com; a=ed25519; pk=1PVTmzPXfKvDwcPUzG0aqdGoKZJA3b9s+3DqRlm0Lww= X-Last-TLS-Session-Version: TLSv1.3 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Sun, 15 Nov 2026 11:34:12 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/250275 Tasks which are not sstate tasks (do_configure, do_compile, do_install) are skipped when covered by a setscene task, so their messages were never shown again on sstate reuse. When creating an sstate package, also bundle the records of the non-sstate tasks this task depends on, from this recipe or from others, read from the location returned by logrecords_path(). On install, replay the records of the covered tasks bitbake assigned to this setscene task through BB_SETSCENE_CLAIMED_TASKS, so each covered task is replayed exactly once even when several setscene tasks cover it. AI-Generated: Uses opencode (DeepSeek V4 Flash) Signed-off-by: Mathieu Dubois-Briand --- meta/classes-global/sstate.bbclass | 103 ++++++++++++++++++++++++++++++------- 1 file changed, 84 insertions(+), 19 deletions(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index ed88dae17202..c7a21310003c 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -206,6 +206,10 @@ def sstate_state_fromvars(d, task = None): ss['fixmedir'] = fixmedir return ss +def sstate_logrecords_path(d, tid): + recordfn, taskname = tid.rsplit(':', 1) + return bb.utils.logrecords_path(d, recordfn, taskname) + def sstate_log_strip_paths(d, message): import re @@ -239,6 +243,11 @@ def sstate_log_save(d, record): levelno, message = record return {'levelno': levelno, 'message': sstate_log_strip_paths(d, message)} +def sstate_log_expand(d, logentry): + # Archived record (dict) -> raw logstore entry, with path placeholders + # expanded back. Inverse of sstate_log_save. + return [logentry['levelno'], sstate_log_expand_paths(d, logentry['message'])] + def sstate_log_emit(d, logentry): levelno = logentry.get('levelno', 0) if levelno < bb.msg.BBLogFormatter.WARNONCE or "message" not in logentry: @@ -253,24 +262,84 @@ def sstate_log_emit(d, logentry): else: bb.warn(message) -python sstate_save_task_log () { +def sstate_replay_logs(d, logdata): + """Replay the log records of the tasks this setscene task was handed via + BB_SETSCENE_CLAIMED_TASKS, plus its own task. + + Fold each task's records back into the shared logstore (always, even + empty, to clear a stale entry from an earlier native run), so an sstate + archive created later in this build still bundles them although the + tasks were restored rather than run natively. Then emit them for the + user. + """ import json - # Save the messages logged by this task, so they can be replayed when the - # task is restored from sstate. The records are written by bitbake right - # after the task body, before the postfuncs. - mytask = d.getVar('BB_RUNTASK') - recordsfn = os.path.join(d.getVar('T'), 'logrecords.%s.json' % mytask) - if not os.path.exists(recordsfn): - return - with open(recordsfn) as f: - records = json.load(f) + claimed = (d.getVar('BB_SETSCENE_CLAIMED_TASKS') or '').split() + filename = d.getVar('BB_FILENAME', False) + claimed.append((filename + ":" if filename else "") + + d.getVar('BB_RUNTASK').removesuffix('_setscene')) + + records = [] + for task in claimed: + taskrecords = logdata.get('tasks', {}).get(task, []) + logrecordsfn = sstate_logrecords_path(d, task) + bb.utils.mkdirhier(os.path.dirname(logrecordsfn)) + with open(logrecordsfn, 'w') as f: + json.dump([sstate_log_expand(d, e) + for e in taskrecords + if 'levelno' in e and 'message' in e], f) + records += taskrecords + for logentry in records: + sstate_log_emit(d, logentry) + +sstate_replay_logs[vardepsexclude] = "BB_SETSCENE_CLAIMED_TASKS BB_FILENAME BB_RUNTASK" - tasks = {mytask: [sstate_log_save(d, record) for record in records]} +python sstate_save_task_log () { + import json - logfn = os.path.join(d.getVar('SSTATE_BUILDDIR'), "sstate-log.json") - with open(logfn, 'w') as f: - json.dump({'tasks': tasks}, f) + # Also bundle the messages of the non-sstate tasks we depend on, from this + # recipe or from others: they are skipped when restored from sstate, and + # bitbake picks which setscene task claims them (BB_SETSCENE_CLAIMED_TASKS). + taskdepdata = d.getVar('BB_TASKDEPDATA', False) or {} + sstatetasks = d.getVar('SSTATETASKS').split() + mytask = d.getVar('BB_RUNTASK') + # The running task is always an entry in taskdepdata, under its own tid. + # Reconstruct that tid from BB_FILENAME and BB_RUNTASK. + filename = d.getVar('BB_FILENAME', False) + start = filename + ":" + mytask if filename else None + if start not in taskdepdata: + bb.fatal("sstate: cannot identify task %s in taskdepdata" % start) + + loggedtasks = {start} + seen = {start} + todo = [start] + while todo: + for dep in taskdepdata[todo.pop()].deps: + t = taskdepdata[dep] + if dep in seen or t.taskname in sstatetasks: + continue + seen.add(dep) + todo.append(dep) + loggedtasks.add(dep) + + # Every task writes its records right after its body, before its postfuncs, to + # the location returned by logrecords_path(), so the files of all dependencies, + # including from other recipes, exist here. + tasklogs = {} + for tid in loggedtasks: + recordsfn = sstate_logrecords_path(d, tid) + if os.path.exists(recordsfn): + with open(recordsfn) as f: + tasklogs[tid] = json.load(f) + + tasks = {} + for tid, records in tasklogs.items(): + tasks[tid] = [sstate_log_save(d, record) for record in records] + + if tasks: + logfn = os.path.join(d.getVar('SSTATE_BUILDDIR'), "sstate-log.json") + with open(logfn, 'w') as f: + json.dump({'tasks': tasks}, f) } def sstate_add(ss, source, dest, d): @@ -480,11 +549,7 @@ def sstate_installpkgdir(ss, d, replay=False): logfp = os.path.join(sstateinst, "sstate-log.json") if os.path.exists(logfp): with open(logfp, 'r') as logfile: - logdata = json.load(logfile) - records = logdata.get('tasks', {}).get(d.getVar('BB_RUNTASK').removesuffix('_setscene'), []) - for logentry in records: - sstate_log_emit(d, logentry) - + sstate_replay_logs(d, json.load(logfile)) bb.utils.remove(logfp) return True From patchwork Sun Nov 15 11:33:56 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Dubois-Briand X-Patchwork-Id: 104237 Received: from smtpout-02.galae.net (smtpout-02.galae.net [185.246.84.56]) by mx.groups.io with SMTP id smtpd.msgproc02-g2.5876.1794742443976204844 for ; Sun, 15 Nov 2026 03:34:04 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@bootlin.com header.s=dkim header.b=SHXHF5xd; spf=pass (domain: bootlin.com, ip: 185.246.84.56, mailfrom: mathieu.dubois-briand@bootlin.com) Received: from smtpout-01.galae.net (smtpout-01.galae.net [212.83.139.233]) by smtpout-02.galae.net (Postfix) with ESMTPS id 544D91A166E for ; Sun, 15 Nov 2026 11:34:02 +0000 (UTC) Received: from mail.galae.net (mail.galae.net [212.83.136.155]) by smtpout-01.galae.net (Postfix) with ESMTPS id 2A64260238 for ; Sun, 15 Nov 2026 11:34:02 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id 1FF7D11BC16DA; Sun, 15 Nov 2026 12:34:01 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bootlin.com; s=dkim; t=1794742441; h=from:subject:date:message-id:to:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=HJM4Rq5Og6WOqPiB87HR6e6PGmKB4O6uxF0pZRE2iZ8=; b=SHXHF5xdoOWE3APhvCwYpaMdscPrvg1yE492znppz9fw8YlqUK1Lzuak0wsJWoPMaGA5nZ 3z4+guOoFCh3pQi93Fitr8HFdk39FGZQPR/zbQ/qhE9rAqqdDTcV6DMatbRXVq87JxWzm3 sdeFcyUHJoJSTuveiViSM5y9cJFyHaxtWYEfv2unekdJU6dma4zTazfXz6kLkQxElGz14E Z2lVsEUX1rFwM2X80KurvigohULso+8jaI4I1J9gKFk47aUZ6Xd3tZ/XG3EvFD/JXiPdok 7lfxS3ulg344mLie45sf+9LjbNANSlBzaUzcP34OjRcTVcga8j0lN3KVJh1dGw== From: Mathieu Dubois-Briand Date: Sun, 15 Nov 2026 12:33:56 +0100 Subject: [PATCH RFC v2 3/3] oeqa/sstatetests: Add tests for sstate log replay MIME-Version: 1.0 Message-Id: <20261115-mathieu-debug-replay-v4-v2-3-8fbfc07058b1@bootlin.com> References: <20261115-mathieu-debug-replay-v4-v2-0-8fbfc07058b1@bootlin.com> In-Reply-To: <20261115-mathieu-debug-replay-v4-v2-0-8fbfc07058b1@bootlin.com> To: openembedded-core@lists.openembedded.org Cc: Thomas Petazzoni , Mathieu Dubois-Briand X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1794742439; l=15869; i=mathieu.dubois-briand@bootlin.com; s=20241219; h=from:subject:message-id; bh=3gO7qUa5VW5oaylE7ii9h2YKelZwWtNOqqDJCwIN45c=; b=fIZ0CI9RaWnTuyrDz+US68rXxTqwL+GNshixFJIcmoNuPfzXFlugGCgPeXFFmJbqa11fbbm+P G/MUClCm2//AzJKo5MKpWNrWgawclGhhzEAm/UyYp5fZpq7gIESpQux X-Developer-Key: i=mathieu.dubois-briand@bootlin.com; a=ed25519; pk=1PVTmzPXfKvDwcPUzG0aqdGoKZJA3b9s+3DqRlm0Lww= X-Last-TLS-Session-Version: TLSv1.3 List-Id: X-Webhook-Received: from 45-33-107-173.ip.linodeusercontent.com [45.33.107.173] by lin-tor-korg-lkml-alma10-1 with HTTPS for ; Sun, 15 Nov 2026 11:34:12 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/250276 Add a selftest suite covering the sstate log replay feature: - warnings saved by a task are replayed once, with their severity and odd content (newlines, '@', non-ASCII) intact; - warnings of non-sstate tasks (do_compile) are replayed exactly once even when several setscene tasks cover them; - warnings of a task from another recipe, covered only by a given sstate task, are replayed; - error and erroronce messages are replayed at ERROR level and fail the build; - paths in replayed messages refer to the build doing the replay, not the one that populated the sstate cache. Four selftest recipes emit the messages, including a shell task (do_emit_shell_log) to cover bbwarn output. AI-Generated: Uses opencode (DeepSeek V4 Flash) Signed-off-by: Mathieu Dubois-Briand --- .../recipes-test/log-replay/log-replay-dep.bb | 18 +++ .../log-replay/log-replay-error-test.bb | 16 ++ .../recipes-test/log-replay/log-replay-test.bb | 34 ++++ .../recipes-test/log-replay/log-replay-user.bb | 32 ++++ meta/lib/oeqa/selftest/cases/sstatetests.py | 171 +++++++++++++++++++++ 5 files changed, 271 insertions(+) diff --git a/meta-selftest/recipes-test/log-replay/log-replay-dep.bb b/meta-selftest/recipes-test/log-replay/log-replay-dep.bb new file mode 100644 index 000000000000..53f473f78109 --- /dev/null +++ b/meta-selftest/recipes-test/log-replay/log-replay-dep.bb @@ -0,0 +1,18 @@ +SUMMARY = "Recipe emitting a warning from a task covered by another recipe's setscene, used to test sstate log replay" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + +inherit native nospdx + +INHIBIT_DEFAULT_DEPS = "1" +EXCLUDE_FROM_WORLD = "1" + +do_patch[noexec] = "1" +do_patch[depends] = "" + +# This recipe has no sstate tasks of its own: when log-replay-user is +# rebuilt from sstate, this task can only be restored (covered) by the user +# recipe's setscene, which must replay its warning. +python do_compile() { + bb.warn("log-replay-dep-warn") +} diff --git a/meta-selftest/recipes-test/log-replay/log-replay-error-test.bb b/meta-selftest/recipes-test/log-replay/log-replay-error-test.bb new file mode 100644 index 000000000000..d95d8d1c6f42 --- /dev/null +++ b/meta-selftest/recipes-test/log-replay/log-replay-error-test.bb @@ -0,0 +1,16 @@ +SUMMARY = "Recipe emitting error-level logs, used to test sstate log replay severity" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + +inherit native nospdx + +INHIBIT_DEFAULT_DEPS = "1" +EXCLUDE_FROM_WORLD = "1" + +do_patch[noexec] = "1" +do_patch[depends] = "" + +python do_populate_sysroot:append() { + bb.error("log-replay-error-plain") + bb.erroronce("log-replay-erroronce") +} diff --git a/meta-selftest/recipes-test/log-replay/log-replay-test.bb b/meta-selftest/recipes-test/log-replay/log-replay-test.bb new file mode 100644 index 000000000000..56e43021de13 --- /dev/null +++ b/meta-selftest/recipes-test/log-replay/log-replay-test.bb @@ -0,0 +1,34 @@ +SUMMARY = "Recipe emitting log messages, used to test sstate log replay" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + +inherit native nospdx + +INHIBIT_DEFAULT_DEPS = "1" +EXCLUDE_FROM_WORLD = "1" + +do_patch[noexec] = "1" +do_patch[depends] = "" + +python do_compile() { + bb.warn("log-replay-compile-warn") +} + +python do_populate_sysroot:append() { + bb.warn("log-replay-python-warn tmpdir=%s dl_dir=%s" % (d.getVar("TMPDIR"), d.getVar("DL_DIR"))) + bb.warnonce("log-replay-python-warnonce") + bb.warn("log-replay-odd: line1\nline2 with @ sign and caf\u00e9") +} + +do_emit_shell_log() { + bbwarn "log-replay-shell-warn" +} +do_emit_shell_log[sstate-inputdirs] = "" +do_emit_shell_log[sstate-outputdirs] = "" +addtask emit_shell_log after do_install before do_build + +SSTATETASKS += "do_emit_shell_log" +python do_emit_shell_log_setscene() { + sstate_setscene(d) +} +addtask do_emit_shell_log_setscene diff --git a/meta-selftest/recipes-test/log-replay/log-replay-user.bb b/meta-selftest/recipes-test/log-replay/log-replay-user.bb new file mode 100644 index 000000000000..b83940eabb29 --- /dev/null +++ b/meta-selftest/recipes-test/log-replay/log-replay-user.bb @@ -0,0 +1,32 @@ +SUMMARY = "Recipe depending on log-replay-dep, used to test cross-recipe sstate log replay" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + +inherit native nospdx + +INHIBIT_DEFAULT_DEPS = "1" +EXCLUDE_FROM_WORLD = "1" + +do_patch[noexec] = "1" +do_patch[depends] = "" + +DEPENDS = "log-replay-dep" + +# Pull log-replay-dep's do_compile into this recipe's dependency closure, so +# it is covered by this recipe's setscene and replayed when restored. +do_compile[depends] += "log-replay-dep:do_compile" + +# An sstate task of its own, so a rebuild of this recipe is served from the +# sstate cache by its setscene. +do_sstate_data() { + : +} +do_sstate_data[sstate-inputdirs] = "" +do_sstate_data[sstate-outputdirs] = "" +addtask sstate_data after do_compile before do_build + +SSTATETASKS += "do_sstate_data" +python do_sstate_data_setscene() { + sstate_setscene(d) +} +addtask do_sstate_data_setscene diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py index 5da8caf1c399..03e28221b8c4 100644 --- a/meta/lib/oeqa/selftest/cases/sstatetests.py +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py @@ -244,6 +244,177 @@ class SStateTests(SStateBase): result = runCmd('git add bar.txt; git commit -asm "add bar"', cwd=srcdir) bitbake("dbus-wait-test -c unpack") +class SStateLogReplayTests(SStateBase): + def assert_restored_from_setscene(self, output, task): + """Assert task was restored from sstate and not run natively. + + Bitbake prints a lifecycle line per task carrying its name: the + setscene variant ends in _setscene, the plain task doesn't. So a + replayed log shows 'task _setscene: Started' and the plain + ': Started' line never appears. + """ + self.assertIn("task %s_setscene: Started" % task, output, msg=output) + self.assertNotIn("task %s: Started" % task, output, msg=output) + + def test_sstate_log_replay(self): + """ + Summary: Messages logged while building a task must be replayed + when the task is restored from the sstate cache. + Expected: Warnings emitted by the initial build are shown again on + the rebuild, which is served from sstate. + Product: OE-Core + """ + # Use a dedicated sstate dir so this test does not depend on the state + # of the shared cache. + sstate_dir = os.path.join(self.topdir, "sstate-log-replay-basic") + self.track_for_cleanup(sstate_dir) + self.write_config('SSTATE_DIR = "%s"\n' % sstate_dir) + # Clean any previous build so the first build populates this test's + # sstate directory, then clean the local work/stamps and rebuild so + # the tasks are restored from the sstate cache. + bitbake("log-replay-test -cclean") + bitbake("log-replay-test") + bitbake("log-replay-test -cclean") + result = bitbake("log-replay-test") + + self.assertRegex(result.output, r"log-replay-test.*do_.*_setscene", + msg="Task was not restored from sstate:\n%s" % result.output) + # Messages saved at warning level (warn and warnonce) replay with the + # WARNING severity. + for marker in ("log-replay-python-warn tmpdir=", + "log-replay-python-warnonce", "log-replay-shell-warn"): + lines = [l for l in result.output.splitlines() if marker in l] + self.assertEqual(len(lines), 1, msg=result.output) + self.assertIn("WARNING:", lines[0]) + # Messages with odd content (newline, '@', non-ASCII) must survive the + # save to and replay from sstate-log.json unmodified. + self.assertIn("log-replay-odd: line1\nline2 with @ sign and caf\u00e9", result.output) + # The messages must be replayed from sstate, not re-emitted by a + # native run of a covered task: do_populate_sysroot and + # do_emit_shell_log were restored by their setscene variants and never + # ran as plain tasks. + for task in ("do_populate_sysroot", "do_emit_shell_log"): + self.assert_restored_from_setscene(result.output, task) + + def test_sstate_log_replay_compile(self): + """ + Summary: Messages logged by non-sstate tasks (do_compile) must be + replayed when the recipe is restored from the sstate cache. + Expected: The do_compile warning is replayed exactly once, even if + several setscene tasks cover do_compile. + Product: OE-Core + """ + sstate_dir = os.path.join(self.topdir, "sstate-log-replay-compile") + self.track_for_cleanup(sstate_dir) + self.write_config('SSTATE_DIR = "%s"\n' % sstate_dir) + bitbake("log-replay-test -cclean") + bitbake("log-replay-test") + bitbake("log-replay-test -cclean") + result = bitbake("log-replay-test") + + self.assertRegex(result.output, r"log-replay-test.*do_.*_setscene", + msg="Task was not restored from sstate:\n%s" % result.output) + lines = [l for l in result.output.splitlines() if "log-replay-compile-warn" in l] + self.assertEqual(len(lines), 1, msg=result.output) + # do_compile is not an sstate task: one of the recipe's setscene tasks + # covers it. It must not have run natively (no lifecycle line for the + # plain task), so its warning can only have been replayed. + self.assertNotIn("task do_compile: Started", result.output, msg=result.output) + + def test_sstate_log_replay_error_severity(self): + """ + Summary: Messages logged at error level must be replayed as errors. + Expected: error and erroronce messages are replayed with the ERROR + severity and reported by the build. + Product: OE-Core + """ + sstate_dir = os.path.join(self.topdir, "sstate-log-replay-error") + self.track_for_cleanup(sstate_dir) + self.write_config('SSTATE_DIR = "%s"\n' % sstate_dir) + bitbake("log-replay-error-test -cclean") + bitbake("log-replay-error-test", ignore_status=True) + bitbake("log-replay-error-test -cclean") + result = bitbake("log-replay-error-test", ignore_status=True) + + self.assertRegex(result.output, r"log-replay-error-test.*do_.*_setscene", + msg="Task was not restored from sstate:\n%s" % result.output) + for marker in ("log-replay-error-plain", "log-replay-erroronce"): + lines = [l for l in result.output.splitlines() if marker in l] + self.assertEqual(len(lines), 1, msg=result.output) + self.assertIn("ERROR:", lines[0]) + self.assertNotEqual(result.status, 0) + # Errors were replayed from sstate by the setscene task, not printed + # by a native run of do_populate_sysroot. + self.assert_restored_from_setscene(result.output, "do_populate_sysroot") + + def test_sstate_log_replay_rewrites_paths(self): + """ + Summary: Paths in replayed messages must refer to the build that is + replaying them, not the one that populated the sstate cache. + Expected: A message saved under TMPDIR A is replayed showing TMPDIR B + when the task is restored under a different TMPDIR. + Product: OE-Core + """ + target = "log-replay-test" + tmpdir_a = os.path.join(self.topdir, "tmp-log-replay-a") + tmpdir_b = os.path.join(self.topdir, "tmp-log-replay-b") + dl_a = os.path.join(self.topdir, "dl-log-replay-a") + dl_b = os.path.join(self.topdir, "dl-log-replay-b") + sstate_dir = os.path.join(self.topdir, "sstate-log-replay") + for path in (tmpdir_a, tmpdir_b, dl_a, dl_b, sstate_dir): + self.track_for_cleanup(path) + + # TMPDIR and DL_DIR are excluded from the task hashes + # (BB_HASHEXCLUDE_COMMON), so a sstate cache populated under the "a" + # paths can be reused under the "b" paths. Only do_populate_sysroot is + # needed: it emits the messages and its setscene replays them. + config = 'TMPDIR = "%s"\nDL_DIR = "%s"\nSSTATE_DIR = "%s"\n' + self.write_config(config % (tmpdir_a, dl_a, sstate_dir)) + bitbake("%s -cclean" % target) + bitbake("%s -c populate_sysroot" % target) + + self.write_config(config % (tmpdir_b, dl_b, sstate_dir)) + result = bitbake("%s -c populate_sysroot" % target) + + self.assertRegex(result.output, r"log-replay-test.*do_.*_setscene", + msg="Task was not restored from sstate:\n%s" % result.output) + warned = [l for l in result.output.splitlines() if "log-replay-python-warn tmpdir=" in l] + self.assertEqual(len(warned), 1, msg=result.output) + self.assertIn(tmpdir_b, warned[0]) + self.assertIn(dl_b, warned[0]) + self.assertNotIn(tmpdir_a, warned[0]) + self.assertNotIn(dl_a, warned[0]) + # The replayed message must be shown by the setscene task, not by a + # native run: do_populate_sysroot never started as a plain task. + self.assert_restored_from_setscene(result.output, "do_populate_sysroot") + + def test_sstate_log_replay_cross_recipe(self): + """ + Summary: Warnings from a task of another recipe, covered only by a + given sstate task, must be replayed when that sstate task + is restored. + Expected: The warning of the dependency recipe's do_compile is shown + again on the rebuild, which is served from sstate. + Product: OE-Core + """ + # Clean any previous build so the first build populates the dedicated + # sstate directory, then clean both so nothing is up to date, and + # rebuild: the dependency's do_compile is covered by the user recipe's + # setscene and must be replayed, not re-run natively. + bitbake("log-replay-user log-replay-dep -cclean") + bitbake("log-replay-user") + bitbake("log-replay-user log-replay-dep -cclean") + result = bitbake("log-replay-user") + + self.assertRegex(result.output, r"log-replay-user.*do_.*_setscene", + msg="Task was not restored from sstate:\n%s" % result.output) + lines = [l for l in result.output.splitlines() if "log-replay-dep-warn" in l] + self.assertEqual(len(lines), 1, msg=result.output) + self.assertIn("WARNING:", lines[0]) + # The dependency's do_compile never ran natively: it was covered by the + # user recipe's setscene and its warning replayed. + self.assertNotIn("task do_compile: Started", result.output, msg=result.output) + class SStateCreation(SStateBase): def test_sstate_creation_distro_specific(self): self.run_test_sstate_creation(['binutils-cross-'+ self.tune_arch, 'binutils-native'], hostdistro_specific=True)