From patchwork Sat Jan 21 21:23:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 18450 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFBE6C61D9D for ; Sat, 21 Jan 2023 21:23:18 +0000 (UTC) Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by mx.groups.io with SMTP id smtpd.web11.6462.1674336188796228281 for ; Sat, 21 Jan 2023 13:23:09 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=PY9ez9L3; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.44, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f44.google.com with SMTP id e19-20020a05600c439300b003db1cac0c1fso6617617wmn.5 for ; Sat, 21 Jan 2023 13:23:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=Cr4uY+9vkyNULM9C7oC2K3i6+NlZ98/GnSpURGemce4=; b=PY9ez9L34k45zuBY2E/dH3DlWqAcLkbmppzsXb1E7kNV9KaCBwSf/piVWQjP+npjnx 6uQrl8vsPXgK0ifXBYpJ88lYPp28lV1x7CaKJKUmKwn32DtXMc88i1qPdQ6yRRP1igss /lvtc0VcZB2x+EsL2bnhuh/QXI9RRp2AnZ1Jw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Cr4uY+9vkyNULM9C7oC2K3i6+NlZ98/GnSpURGemce4=; b=fvUq23mII5vgdME5xTvKkuY1XJcvX4CqIP9gDvU5nBIptaBmnAEUQWuARkk93ozxjd Ucdiok+6ETO2lSgUjCSRv2HBDixfXeI79vxBUvhM1mebM40+gFW7L/NWRag94qUZ0aKO kU26pimF9KcI/UaRZk7fkXknEg9htpDrGQ+j/WCBqbwFT7ydWQBOS0P0gXaHXf8SV+FH Lwyw2g/Gjm5f+8c6XPjv1AKPFPpVgY0L0//69OkV7JMOjrwj1WgNEoaGgVPw+l7CiWwZ +gGGqvXhII3wxNyAM6ITS3WryzbsODto+FaL1+snT8Hei2IofT7nwLzbBFOMO2nqJe45 rXgQ== X-Gm-Message-State: AFqh2kqgRMQXAk4vypXBN3jkYTmrV6Bl7avQUmBcJonnIJl/Ij1vFoP1 +3FC7VmHNNgAjS8L9mrjJVBm/TnNRlR1B4xu X-Google-Smtp-Source: AMrXdXsAv9cxyVDEipUVGL/fG1xh2GX/iAS+K5G/hjKj5Raqn5vjGcFPC2whGzoH/iXiLVVcAC8Q5A== X-Received: by 2002:a1c:f617:0:b0:3d2:191d:2420 with SMTP id w23-20020a1cf617000000b003d2191d2420mr17918918wmc.7.1674336186562; Sat, 21 Jan 2023 13:23:06 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f1f:fd3b:ca38:d2c5]) by smtp.gmail.com with ESMTPSA id p16-20020a05600c359000b003da105437besm6713089wmq.29.2023.01.21.13.23.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Jan 2023 13:23:06 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 1/6] server/process: Fix lockfile contents check bug Date: Sat, 21 Jan 2023 21:23:00 +0000 Message-Id: <20230121212305.2171310-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 21 Jan 2023 21:23:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14330 We need to check against the first line of the file, fix the typo. Signed-off-by: Richard Purdie --- lib/bb/server/process.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py index 4bd68b1e7f..01ce916a3d 100644 --- a/lib/bb/server/process.py +++ b/lib/bb/server/process.py @@ -376,7 +376,7 @@ class ProcessServer(): lock = bb.utils.lockfile(lockfile, shared=False, retry=False, block=False) if not lock: newlockcontents = get_lock_contents(lockfile) - if not newlockcontents.startswith([os.getpid() + "\n", os.getpid() + " "]): + if not newlockcontents[0].startswith([os.getpid() + "\n", os.getpid() + " "]): # A new server was started, the lockfile contents changed, we can exit serverlog("Lockfile now contains different contents, exiting: " + str(newlockcontents)) return From patchwork Sat Jan 21 21:23:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 18448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7EAEC54E94 for ; Sat, 21 Jan 2023 21:23:18 +0000 (UTC) Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by mx.groups.io with SMTP id smtpd.web10.6415.1674336188748018524 for ; Sat, 21 Jan 2023 13:23:09 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=ZHwQWO4x; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.49, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f49.google.com with SMTP id c10-20020a05600c0a4a00b003db0636ff84so6030182wmq.0 for ; Sat, 21 Jan 2023 13:23:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=2EQe4YsqesQOgoZPdpKCH2JMHdTzZgiau54EwtXscr0=; b=ZHwQWO4xmHIAizJl6VSqhknd/goMayD53ZwnpJ+u/MnQN2XHcnts+bvXUiLj177Nrc MPx8fgcZ0MZlXzSnDvJ6mmPiSZfRGsiXc1tjIjYDj4b8H6BtPZku1w1x35jz9IlhGOeb 9jB9onIOJ4BNS4dP9W8ZXOsITvezvdygzJT5Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2EQe4YsqesQOgoZPdpKCH2JMHdTzZgiau54EwtXscr0=; b=sESd36HpotWzlZvNQo9TBuxuv0mBhlspVDzgTHPIz5Ku7KfmGAPcKOjfEDH6b3xfNf +qnVm+QzjNYeMvjln2rFBqSlciLwM4yQ8iEIUG3j+A2Ebk+mWT1K41ybXuScqo7OSg2t 3Sq2k7oreOYPW5ZUj/b4qSCfYml8RZ0uHeHK7a6E+IPj6P834TZVUNwbebuyC8hh0kHj KOw+3EHkDkrGV9drgnj1NyyjAdVNhsLt4Vm0A+JKws0EeoFVTEWumIYbi1oJ0NgNY3fc cuCbWLXj/Q6xUkdb9xP00aVdP1qWXFNo6nDFlnp4W/uvdyt0hTdtqRWvZ66O8qsEdKYX ek8Q== X-Gm-Message-State: AFqh2kq+/pvcvP9c6ZKiZaGLTyy2mITx62HJUrp5V+Rwb0EjlY4t8ksl XtX/n5m4tprYut4O941yhoS2Vjl5zWar4ebk X-Google-Smtp-Source: AMrXdXsAaPRWqSUH/WFsEz4SaFzdVqnTSwTbtFTU0DHcbMjkZVyA1yg4hfM0w2Gf/2nDCqEpXtk7fQ== X-Received: by 2002:a05:600c:35ce:b0:3db:1caf:1020 with SMTP id r14-20020a05600c35ce00b003db1caf1020mr11832734wmq.35.1674336187121; Sat, 21 Jan 2023 13:23:07 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f1f:fd3b:ca38:d2c5]) by smtp.gmail.com with ESMTPSA id p16-20020a05600c359000b003da105437besm6713089wmq.29.2023.01.21.13.23.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Jan 2023 13:23:06 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 2/6] cooker: Fix siggen recipe cache race issue Date: Sat, 21 Jan 2023 21:23:01 +0000 Message-Id: <20230121212305.2171310-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230121212305.2171310-1-richard.purdie@linuxfoundation.org> References: <20230121212305.2171310-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 21 Jan 2023 21:23:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14329 We need to reset the cache before the start() call which starts the parsing processs, tweak the code to ensure this is the case. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 617d582acc..d15ad2fb2e 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2187,12 +2187,11 @@ class CookerParser(object): self.num_processes = min(int(self.cfgdata.getVar("BB_NUMBER_PARSE_THREADS") or multiprocessing.cpu_count()), self.toparse) + bb.cache.SiggenRecipeInfo.reset() self.start() self.haveshutdown = False self.syncthread = None - bb.cache.SiggenRecipeInfo.reset() - def start(self): self.results = self.load_cached() self.processes = [] From patchwork Sat Jan 21 21:23:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 18449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7707C54EB4 for ; Sat, 21 Jan 2023 21:23:18 +0000 (UTC) Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by mx.groups.io with SMTP id smtpd.web10.6416.1674336189293731324 for ; Sat, 21 Jan 2023 13:23:09 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=CBDvD0gg; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.49, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f49.google.com with SMTP id e19-20020a05600c439300b003db1cac0c1fso6617626wmn.5 for ; Sat, 21 Jan 2023 13:23:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=kiODrOxDLTX7OAao2Zl3XyRXGy62wfH5vN9Ot44z2dg=; b=CBDvD0gg5ssmCEYm+nfN5zAkV4A+uSqOgokU/DC+XDcT2kTHqTvjkPiNnhiPmXvpZ0 WdQ9wkoznrjtUvXfHqufDM8ooBEwrmQW9cpZF5F8n8Cu2GXY6GEXRtv6BuCu1y1xIvHJ vtZf8ZSkaw23GdAcDPomcxtzuzLC5TktrzKyo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kiODrOxDLTX7OAao2Zl3XyRXGy62wfH5vN9Ot44z2dg=; b=ufd+U3iewwOFkUNW18RqhXBupBwro3TysbYm1LVlCb31Ywfm62VCIByKHQR/e3th0y hvUzBT3bX4sjd8Cm/jMSqyaXoTBefU9w6vfpRMp6Z/w3Fcr7jAsSlEr7W1XEu24h7hMz 0deetr/Tdj0OXneEbl7rrOj0hErNit7yrA0yBsesR9gAjIQveLcU6O9scsuDYX9r7KxA w6O2uzIRXIsVUcI/esSFrfzT/+RANKg9C0mcwZcPElkgIX1FPHzRe5BDk9e/rmIgEtQr KKGzk0Sy85mZV/eYd5QPWXC5EchqZeL6+8xi2RtOmWr0iJJu+RsH+BwqQOgiDkZ4x3vS pCZQ== X-Gm-Message-State: AFqh2ko/8ZTjyk/8WFDqjk0N6k+PcE7yFB7EY21HOgFy167D6DIRPQTa DhtSVgLkF0yDw/PeymHQp81vFm4TsiUR48sG X-Google-Smtp-Source: AMrXdXsHdqra0RrGn6JGYA5yVpSojin7h8TM8BH2w1vXp7ApO4+B25jFQ0xhzVzs0CJsNZgzaYKdoA== X-Received: by 2002:a05:600c:4f83:b0:3db:eab:a600 with SMTP id n3-20020a05600c4f8300b003db0eaba600mr15570091wmq.7.1674336187664; Sat, 21 Jan 2023 13:23:07 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f1f:fd3b:ca38:d2c5]) by smtp.gmail.com with ESMTPSA id p16-20020a05600c359000b003da105437besm6713089wmq.29.2023.01.21.13.23.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Jan 2023 13:23:07 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 3/6] cache: Only write files if we have data Date: Sat, 21 Jan 2023 21:23:02 +0000 Message-Id: <20230121212305.2171310-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230121212305.2171310-1-richard.purdie@linuxfoundation.org> References: <20230121212305.2171310-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 21 Jan 2023 21:23:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14331 By writing the cache files only if there is data to write, we can save a bit of time. Signed-off-by: Richard Purdie --- lib/bb/cache.py | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/lib/bb/cache.py b/lib/bb/cache.py index ee924b2d2b..c19fe26f1b 100644 --- a/lib/bb/cache.py +++ b/lib/bb/cache.py @@ -873,6 +873,14 @@ class MultiProcessCache(object): if not self.cachefile: return + have_data = False + for c in self.cachedata_extras: + if c: + have_data = True + break + if not have_data: + return + glf = bb.utils.lockfile(self.cachefile + ".lock", shared=True) i = os.getpid() @@ -907,6 +915,8 @@ class MultiProcessCache(object): data = self.cachedata + have_data = False + for f in [y for y in os.listdir(os.path.dirname(self.cachefile)) if y.startswith(os.path.basename(self.cachefile) + '-')]: f = os.path.join(os.path.dirname(self.cachefile), f) try: @@ -921,12 +931,14 @@ class MultiProcessCache(object): os.unlink(f) continue + have_data = True self.merge_data(extradata, data) os.unlink(f) - with open(self.cachefile, "wb") as f: - p = pickle.Pickler(f, -1) - p.dump([data, self.__class__.CACHE_VERSION]) + if have_data: + with open(self.cachefile, "wb") as f: + p = pickle.Pickler(f, -1) + p.dump([data, self.__class__.CACHE_VERSION]) bb.utils.unlockfile(glf) From patchwork Sat Jan 21 21:23:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 18447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4495C54EED for ; Sat, 21 Jan 2023 21:23:18 +0000 (UTC) Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by mx.groups.io with SMTP id smtpd.web10.6417.1674336189937221351 for ; Sat, 21 Jan 2023 13:23:10 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=RFDTL/Nw; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.44, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f44.google.com with SMTP id e19-20020a05600c439300b003db1cac0c1fso6617637wmn.5 for ; Sat, 21 Jan 2023 13:23:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=e8rthqTUt6TlNN5L+CmC3N/1qS3TLrn4JBva87s4vig=; b=RFDTL/NwFRuRPHakBlZlmyn23OgGfkrhpYKCyw3K04OGrd8nB2+4std731KiruI0RM OX6RZAorBMlj9xPxDm20+fA6G2qaQwR4DRZgDKtu57Q9MgeKqiWo89NQCiCpTAj99Iko HGBi8EwQJnp/EjUsrj9xUupfNJix/W00Iu7L4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e8rthqTUt6TlNN5L+CmC3N/1qS3TLrn4JBva87s4vig=; b=w9PdT3A3ByTJke4ze5SUVsFShzCJYkJBMEjI2mua2YLuLcdl2NaD+5VPOWSVnwE17O svmf/LNqdR6CD3BOpLKk1VsMwukjej1iWMp0xj2ZHAGhFdz+YDVdqaRadbblJ8geyq4A yDPRk44epuecu6AWt3raL0Q2I1qkHxceEr75xknFTmVDNGMDz0ZYWj0Jo1FcH8nIEmWz /mtBpmo++romcvqxyFDy4VRo411arHdkdP0DzKM/djd+ay8xLElGi+YVozzk0cERAkrG lM3+wnysfc516Nc2qZPybmUHY21dwXchQi35mlSSNT8E0lUPiSom3C87qumbC1hURP0E c8sw== X-Gm-Message-State: AFqh2koz0gLHrvUqqx1cr423UB48wH0F/i+B6WLqi4ZEIiGah2x/N7d+ FtC49rUhGfzZ+NxbqzKuRVbOfDMIkcGoDKQj X-Google-Smtp-Source: AMrXdXsxqEO5ei3Gu8Jo3bK4IA4tDizliAQZnq3ChVTRSqHcn39Qo5cIqImEHOjOG5Zt80sYbFgJuA== X-Received: by 2002:a05:600c:3c83:b0:3d9:e5d3:bf with SMTP id bg3-20020a05600c3c8300b003d9e5d300bfmr18644747wmb.32.1674336188224; Sat, 21 Jan 2023 13:23:08 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f1f:fd3b:ca38:d2c5]) by smtp.gmail.com with ESMTPSA id p16-20020a05600c359000b003da105437besm6713089wmq.29.2023.01.21.13.23.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Jan 2023 13:23:07 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 4/6] cooker: Fix parsing race around cache handling Date: Sat, 21 Jan 2023 21:23:03 +0000 Message-Id: <20230121212305.2171310-4-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230121212305.2171310-1-richard.purdie@linuxfoundation.org> References: <20230121212305.2171310-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 21 Jan 2023 21:23:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14332 When draining the result queue from the parsing processes, cache objects can be created even if they are then immediately destroyed. The reset in the sync code needs to happen after any objects have been created. Change the ordering to fix this. This ordering has caused various cache errors, particularly when interrupting parsing with Ctrl+C. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index d15ad2fb2e..527f6a0421 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -2233,6 +2233,14 @@ class CookerParser(object): else: bb.error("Parsing halted due to errors, see error messages above") + # Cleanup the queue before call process.join(), otherwise there might be + # deadlocks. + while True: + try: + self.result_queue.get(timeout=0.25) + except queue.Empty: + break + def sync_caches(): for c in self.bb_caches.values(): bb.cache.SiggenRecipeInfo.reset() @@ -2243,14 +2251,6 @@ class CookerParser(object): self.parser_quit.set() - # Cleanup the queue before call process.join(), otherwise there might be - # deadlocks. - while True: - try: - self.result_queue.get(timeout=0.25) - except queue.Empty: - break - for process in self.processes: process.join(0.5) From patchwork Sat Jan 21 21:23:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 18446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5EEEC38142 for ; Sat, 21 Jan 2023 21:23:18 +0000 (UTC) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mx.groups.io with SMTP id smtpd.web11.6463.1674336190708745600 for ; Sat, 21 Jan 2023 13:23:11 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=hU4ZpRkG; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.48, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f48.google.com with SMTP id d4-20020a05600c3ac400b003db1de2aef0so6009234wms.2 for ; Sat, 21 Jan 2023 13:23:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=SpZTXnUbhXiScHiSe/6y+1FKqxPAXZrlcCcvBfUPrs4=; b=hU4ZpRkGolPRfO4W962pVd6daWG1LWV8Db5r4i4J+GocabS31GiGRXgpKPKXLAB6f8 c7eA4kuGzBRrBcwzV4QMXRTfgM6m6cxfbipB9pDPsLDf6EymqGFLpmTVwwhvXMAJsazo UmOqnOv8rEOzNkzXdTU8PXSh/xsYrFhIMErKw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SpZTXnUbhXiScHiSe/6y+1FKqxPAXZrlcCcvBfUPrs4=; b=tFICr4Gk2mn7vAPGJQXweilmNudingn+yTzMNN9nPryd4YGLkYWGigEcOEBJmh+ITU /ckB3X8dn8/QOsXHxtdnWiFG/k/IapvS5epWwwiCa1Y/lqVSby/EYm76U3mg9C5O4p0n xkIA3500e2nwXg5vRTUYrXR5DmwIB3ARflb6IGnt+VHnkPHnnwNPRtY8+tkbL9YWXOKE SQGtiIdjeRXFaXp3oyLSS1IolqlZIuY7y/lTpoOKbqBQgJGUwK9xz0ejy8S5s6vq0KPn j3Ib7hA0/HTBPJ2tQnTMOj8sKHrYFjqgnxmkOMo7xEy7I1CiJZWK2CrmYuc1BlBy+N5g kz9Q== X-Gm-Message-State: AFqh2koZONMXl23Nkg6Il0PbtFex5mnUOdJSIiF0tLLbdN4mG2IUJnGx bzZWz11Xvs8dyXacMP7226xTBwmxvMXMIBVR X-Google-Smtp-Source: AMrXdXvjwKJq19seKy3cybVS0zA3beKUmf2r2JqUBiN7ARI6F81bCCuFAwXK6xZsZCAs0Hh4zsnvHw== X-Received: by 2002:a05:600c:4928:b0:3d2:2043:9cb7 with SMTP id f40-20020a05600c492800b003d220439cb7mr18608010wmp.5.1674336188858; Sat, 21 Jan 2023 13:23:08 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f1f:fd3b:ca38:d2c5]) by smtp.gmail.com with ESMTPSA id p16-20020a05600c359000b003da105437besm6713089wmq.29.2023.01.21.13.23.08 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Jan 2023 13:23:08 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 5/6] data: Add sup0port for new BB_HASH_CODEPARSER_VALS for cache optimisation Date: Sat, 21 Jan 2023 21:23:04 +0000 Message-Id: <20230121212305.2171310-5-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230121212305.2171310-1-richard.purdie@linuxfoundation.org> References: <20230121212305.2171310-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 21 Jan 2023 21:23:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14333 Currently the codeparser cache ends up being extended for every parse run since there are values in the functions such as the result of os.getpid() from LOGFIFO in OE-Core. Digging into that issue, there are also lots of similar but different functions being parsed where the change might just be a path to WORKDIR, a change in PN or PV or something like DATE/TIME. There is no reason we have to use these changing values when computing the dependenies of the functions. Even with a small tweak like: BB_HASH_CODEPARSER_VALS = "LOGFIFO=/ T=/ WORKDIR=/ DATE=1234 TIME=1234 PV=0.0-1 PN=nopn" the cache is reduced from ~4.6MB, increasing by ~300kb for every parse run to around 1.3MB and remaining static for oe-core and meta-oe. In my local build, admittedly heavily experimented with, the cache had grown to 120MB. The benefits of doing this are: * faster load time for bitbake since the cache is smaller to read from disk and load into memory * being able to skip saving the cache upon shutdown * lower memory footprint for bitbake * faster codeparser data lookups (since there is less data to search) We only use these special values when passing code fragments to the codeparser to parse so the real variable values should otherwise be used in the hash data. The overall effect of this change, combined with others to avoid saving unchanged cache files can be ~2s on a ~16s parse on my local system and results in a more responsive feeling bitbake. It also allows parsing performance to be investigated more consistently. Signed-off-by: Richard Purdie --- lib/bb/data.py | 15 ++++++++++----- lib/bb/tests/codeparser.py | 14 +++++++------- 2 files changed, 17 insertions(+), 12 deletions(-) diff --git a/lib/bb/data.py b/lib/bb/data.py index 841369699e..f3ae062022 100644 --- a/lib/bb/data.py +++ b/lib/bb/data.py @@ -261,7 +261,7 @@ def emit_func_python(func, o=sys.__stdout__, d = init()): newdeps |= set((d.getVarFlag(dep, "vardeps") or "").split()) newdeps -= seen -def build_dependencies(key, keys, mod_funcs, shelldeps, varflagsexcl, ignored_vars, d): +def build_dependencies(key, keys, mod_funcs, shelldeps, varflagsexcl, ignored_vars, d, codeparsedata): def handle_contains(value, contains, exclusions, d): newvalue = [] if value: @@ -312,14 +312,14 @@ def build_dependencies(key, keys, mod_funcs, shelldeps, varflagsexcl, ignored_va value = varflags.get("vardepvalue") elif varflags.get("func"): if varflags.get("python"): - value = d.getVarFlag(key, "_content", False) + value = codeparsedata.getVarFlag(key, "_content", False) parser = bb.codeparser.PythonParser(key, logger) parser.parse_python(value, filename=varflags.get("filename"), lineno=varflags.get("lineno")) deps = deps | parser.references deps = deps | (keys & parser.execs) value = handle_contains(value, parser.contains, exclusions, d) else: - value, parsedvar = d.getVarFlag(key, "_content", False, retparser=True) + value, parsedvar = codeparsedata.getVarFlag(key, "_content", False, retparser=True) parser = bb.codeparser.ShellParser(key, logger) parser.parse_shell(parsedvar.value) deps = deps | shelldeps @@ -378,12 +378,17 @@ def generate_dependencies(d, ignored_vars): shelldeps = set(key for key in d.getVar("__exportlist", False) if d.getVarFlag(key, "export", False) and not d.getVarFlag(key, "unexport", False)) varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS') + codeparserd = d.createCopy() + for forced in (d.getVar('BB_HASH_CODEPARSER_VALS') or "").split(): + key, value = forced.split("=", 1) + codeparserd.setVar(key, value) + deps = {} values = {} tasklist = d.getVar('__BBTASKS', False) or [] for task in tasklist: - deps[task], values[task] = build_dependencies(task, keys, mod_funcs, shelldeps, varflagsexcl, ignored_vars, d) + deps[task], values[task] = build_dependencies(task, keys, mod_funcs, shelldeps, varflagsexcl, ignored_vars, d, codeparserd) newdeps = deps[task] seen = set() while newdeps: @@ -392,7 +397,7 @@ def generate_dependencies(d, ignored_vars): newdeps = set() for dep in nextdeps: if dep not in deps: - deps[dep], values[dep] = build_dependencies(dep, keys, mod_funcs, shelldeps, varflagsexcl, ignored_vars, d) + deps[dep], values[dep] = build_dependencies(dep, keys, mod_funcs, shelldeps, varflagsexcl, ignored_vars, d, codeparserd) newdeps |= deps[dep] newdeps -= seen #print "For %s: %s" % (task, str(deps[task])) diff --git a/lib/bb/tests/codeparser.py b/lib/bb/tests/codeparser.py index a508f23bcb..7f5d59ca74 100644 --- a/lib/bb/tests/codeparser.py +++ b/lib/bb/tests/codeparser.py @@ -318,7 +318,7 @@ d.getVar(a(), False) "filename": "example.bb", }) - deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), set(), set(), self.d) + deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), set(), set(), self.d, self.d) self.assertEqual(deps, set(["somevar", "bar", "something", "inexpand", "test", "test2", "a"])) @@ -365,7 +365,7 @@ esac self.d.setVarFlags("FOO", {"func": True}) self.setEmptyVars(execs) - deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), set(), set(), self.d) + deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), set(), set(), self.d, self.d) self.assertEqual(deps, set(["somevar", "inverted"] + execs)) @@ -375,7 +375,7 @@ esac self.d.setVar("FOO", "foo=oe_libinstall; eval $foo") self.d.setVarFlag("FOO", "vardeps", "oe_libinstall") - deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), set(), set(), self.d) + deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), set(), set(), self.d, self.d) self.assertEqual(deps, set(["oe_libinstall"])) @@ -384,7 +384,7 @@ esac self.d.setVar("FOO", "foo=oe_libinstall; eval $foo") self.d.setVarFlag("FOO", "vardeps", "${@'oe_libinstall'}") - deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), set(), set(), self.d) + deps, values = bb.data.build_dependencies("FOO", set(self.d.keys()), set(), set(), set(), set(), self.d, self.d) self.assertEqual(deps, set(["oe_libinstall"])) @@ -399,7 +399,7 @@ esac # Check dependencies self.d.setVar('ANOTHERVAR', expr) self.d.setVar('TESTVAR', 'anothervalue testval testval2') - deps, values = bb.data.build_dependencies("ANOTHERVAR", set(self.d.keys()), set(), set(), set(), set(), self.d) + deps, values = bb.data.build_dependencies("ANOTHERVAR", set(self.d.keys()), set(), set(), set(), set(), self.d, self.d) self.assertEqual(sorted(values.splitlines()), sorted([expr, 'TESTVAR{anothervalue} = Set', @@ -418,14 +418,14 @@ esac self.d.setVar('ANOTHERVAR', varval) self.d.setVar('TESTVAR', 'anothervalue testval testval2') self.d.setVar('TESTVAR2', 'testval3') - deps, values = bb.data.build_dependencies("ANOTHERVAR", set(self.d.keys()), set(), set(), set(), set(["TESTVAR"]), self.d) + deps, values = bb.data.build_dependencies("ANOTHERVAR", set(self.d.keys()), set(), set(), set(), set(["TESTVAR"]), self.d, self.d) self.assertEqual(sorted(values.splitlines()), sorted([varval])) self.assertEqual(deps, set(["TESTVAR2"])) self.assertEqual(self.d.getVar('ANOTHERVAR').split(), ['testval3', 'anothervalue']) # Check the vardepsexclude flag is handled by contains functionality self.d.setVarFlag('ANOTHERVAR', 'vardepsexclude', 'TESTVAR') - deps, values = bb.data.build_dependencies("ANOTHERVAR", set(self.d.keys()), set(), set(), set(), set(), self.d) + deps, values = bb.data.build_dependencies("ANOTHERVAR", set(self.d.keys()), set(), set(), set(), set(), self.d, self.d) self.assertEqual(sorted(values.splitlines()), sorted([varval])) self.assertEqual(deps, set(["TESTVAR2"])) self.assertEqual(self.d.getVar('ANOTHERVAR').split(), ['testval3', 'anothervalue']) From patchwork Sat Jan 21 21:23:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 18445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3545C54EAA for ; Sat, 21 Jan 2023 21:23:18 +0000 (UTC) Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by mx.groups.io with SMTP id smtpd.web10.6418.1674336191358589214 for ; Sat, 21 Jan 2023 13:23:11 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=L7kq2U6h; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.53, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f53.google.com with SMTP id q8so6427837wmo.5 for ; Sat, 21 Jan 2023 13:23:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=c14Fo4Ir/YSfvMlCoKPz2sOF2fuIrWApZrh1hpKV7Vg=; b=L7kq2U6h01OcXG/BSS7wrvwo284TXEyzK8s6lbGuTd1A6kPSUFcbF1Oq3l3gPXWppZ IDKBAKmZqDCxOww2PFOfLUmSgwIwM0oZ3fkId8V8NrkBGCgseatTZynXmgbRaJhFhpNq mHg6Z5WEsCIi7iv6D5766CdltWjuImDGPWwks= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c14Fo4Ir/YSfvMlCoKPz2sOF2fuIrWApZrh1hpKV7Vg=; b=gKfxXPwFiolrLjpHp/YaOnfn0/zAuqI3agdbpH0AxdEs5ijZYuqoyQFYz1Dt0Qy7K+ HfpQzSKuvAyxKOLf5xJB3hUvElT9pauuZZ4vGDfx/tNEO44fj6FrNYmrk5bemRvXZ9Bs PavvBJmoIfq2cyYlBpFBgFeBi7Cvz4yZ9/eiZ1cWB7oDy18Kt2nb4Yx0Bf72Lc4FsQU7 TkSRxIww3X1KN4lWvSFeaq1aB2v5A5kPDHEcFvPhdtjerdGvWU1N3WfA6TiLfpsfWfWS 3N0iuMC9SxgPt+FP2K85xZRPhjgwehCJMEaBqBDT+1SRu3b7olb4cmllf12CnDyXVNF3 EQNA== X-Gm-Message-State: AFqh2kp1z7ne3zV0uVGW1LROn06qHUNTl+OM9WYGfhWnXmEpB2A+aCEk d7G88AvTITK17vMjemF9DgpfySCG3zPMtjti X-Google-Smtp-Source: AMrXdXtaklg88+7RX7aWPi/3ICBoDQ7ynBliYPZMY9UMkQwq5Ph9mutVfHpfe2PIuySB63KxUhqVkg== X-Received: by 2002:a05:600c:1c8b:b0:3db:fe:f84e with SMTP id k11-20020a05600c1c8b00b003db00fef84emr17856453wms.1.1674336189623; Sat, 21 Jan 2023 13:23:09 -0800 (PST) Received: from max.int.rpsys.net ([2001:8b0:aba:5f3c:3f1f:fd3b:ca38:d2c5]) by smtp.gmail.com with ESMTPSA id p16-20020a05600c359000b003da105437besm6713089wmq.29.2023.01.21.13.23.08 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Jan 2023 13:23:09 -0800 (PST) From: Richard Purdie To: bitbake-devel@lists.openembedded.org Subject: [PATCH 6/6] data_smart: Small optimisation to _findVar() Date: Sat, 21 Jan 2023 21:23:05 +0000 Message-Id: <20230121212305.2171310-6-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230121212305.2171310-1-richard.purdie@linuxfoundation.org> References: <20230121212305.2171310-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Sat, 21 Jan 2023 21:23:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/14334 Some users of _findVar don't need the override data and even getVarFlag doesn't need it in some common cases (parsing=True). Rearrange the code as the current overridedata approach doesn't need to be in the _findVar code anyway. This removes some search overhead from a critical path. Signed-off-by: Richard Purdie --- lib/bb/data_smart.py | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index e2c93597e5..c597dbade8 100644 --- a/lib/bb/data_smart.py +++ b/lib/bb/data_smart.py @@ -515,18 +515,18 @@ class DataSmart(MutableMapping): dest = self.dict while dest: if var in dest: - return dest[var], self.overridedata.get(var, None) + return dest[var] if "_data" not in dest: break dest = dest["_data"] - return None, self.overridedata.get(var, None) + return None def _makeShadowCopy(self, var): if var in self.dict: return - local_var, _ = self._findVar(var) + local_var = self._findVar(var) if local_var: self.dict[var] = copy.copy(local_var) @@ -782,10 +782,12 @@ class DataSmart(MutableMapping): if expand and cachename in self.expand_cache: return self.expand_cache[cachename].value - local_var, overridedata = self._findVar(var) + local_var = self._findVar(var) value = None removes = set() - if flag == "_content" and overridedata is not None and not parsing: + if flag == "_content" and not parsing: + overridedata = self.overridedata.get(var, None) + if flag == "_content" and not parsing and overridedata is not None: match = False active = {} self.need_overrides() @@ -900,7 +902,7 @@ class DataSmart(MutableMapping): def delVarFlag(self, var, flag, **loginfo): self.expand_cache = {} - local_var, _ = self._findVar(var) + local_var = self._findVar(var) if not local_var: return if not var in self.dict: @@ -943,7 +945,7 @@ class DataSmart(MutableMapping): self.dict[var][i] = flags[i] def getVarFlags(self, var, expand = False, internalflags=False): - local_var, _ = self._findVar(var) + local_var = self._findVar(var) flags = {} if local_var: