Message ID | 20220830155939.662178-1-JPEWhacker@gmail.com |
---|---|
State | Accepted, archived |
Commit | 20a9911b73df62a0d0d1884e57085f13ac5016dd |
Headers | show |
Series | classes: cve-check: Get shared database lock | expand |
On Tue, Aug 30, 2022 at 5:59 PM Joshua Watt <JPEWhacker@gmail.com> wrote: > > The CVE check database needs to have a shared lock acquired on it before > it is accessed. This to prevent cve-update-db-native from deleting the > database file out from underneath it. > > [YOCTO #14899] > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > + cve_data = get_cve_info(d, patched + unpatched + ignored) > + cve_write_data(d, patched, unpatched, ignored, cve_data, status) > + else: > + bb.note("No CVE database found, skipping CVE check") > With this commit in kirkstone-nut, we're getting an error with meta-zephyr builds: RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0 do_cve_check: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:do_cve_check(d) 0003: File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass', lineno: 142, function: do_cve_check 0138: Check recipe for patched and unpatched CVEs 0139: """ 0140: from oe.cve_check import get_patched_cves 0141: *** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")], shared=True): 0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")): 0144: try: 0145: patched_cves = get_patched_cves(d) 0146: except FileNotFoundError: File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper 0236: <cleanup> 0237: """ 0238: @wraps(func) 0239: def helper(*args, **kwds): *** 0240: return _GeneratorContextManager(func, args, kwds) 0241: return helper 0242: 0243: 0244:def asynccontextmanager(func): File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__ 0079:class _GeneratorContextManagerBase: 0080: """Shared functionality for @contextmanager and @asynccontextmanager.""" 0081: 0082: def __init__(self, func, args, kwds): *** 0083: self.gen = func(*args, **kwds) 0084: self.func, self.args, self.kwds = func, args, kwds 0085: # Issue 19330: ensure context manager instances have good docstrings 0086: doc = getattr(func, "__doc__", None) 0087: if doc is None: Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared' ERROR: Logfile of failure stored in: /tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603 NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0: task do_cve_check: Failed ERROR: Task (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check) failed with exit code '1' Kind regards, Marta
On Fri, Sep 2, 2022 at 9:09 AM Marta Rybczynska via lists.openembedded.org <rybczynska=gmail.com@lists.openembedded.org> wrote: > > On Tue, Aug 30, 2022 at 5:59 PM Joshua Watt <JPEWhacker@gmail.com> wrote: > > > > The CVE check database needs to have a shared lock acquired on it before > > it is accessed. This to prevent cve-update-db-native from deleting the > > database file out from underneath it. > > > > [YOCTO #14899] > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > + cve_data = get_cve_info(d, patched + unpatched + ignored) > > + cve_write_data(d, patched, unpatched, ignored, cve_data, status) > > + else: > > + bb.note("No CVE database found, skipping CVE check") > > > > With this commit in kirkstone-nut, we're getting an error with > meta-zephyr builds: > > RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0 > do_cve_check: Error executing a python function in exec_func_python() > autogenerated: > The stack trace of python calls that resulted in this exception/failure was: > File: 'exec_func_python() autogenerated', lineno: 2, function: <module> > 0001: > *** 0002:do_cve_check(d) > 0003: > File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass', > lineno: 142, function: do_cve_check > 0138: Check recipe for patched and unpatched CVEs > 0139: """ > 0140: from oe.cve_check import get_patched_cves > 0141: > *** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")], > shared=True): > 0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")): > 0144: try: > 0145: patched_cves = get_patched_cves(d) > 0146: except FileNotFoundError: > File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper > 0236: <cleanup> > 0237: """ > 0238: @wraps(func) > 0239: def helper(*args, **kwds): > *** 0240: return _GeneratorContextManager(func, args, kwds) > 0241: return helper > 0242: > 0243: > 0244:def asynccontextmanager(func): > File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__ > 0079:class _GeneratorContextManagerBase: > 0080: """Shared functionality for @contextmanager and @asynccontextmanager.""" > 0081: > 0082: def __init__(self, func, args, kwds): > *** 0083: self.gen = func(*args, **kwds) > 0084: self.func, self.args, self.kwds = func, args, kwds > 0085: # Issue 19330: ensure context manager instances have good docstrings > 0086: doc = getattr(func, "__doc__", None) > 0087: if doc is None: > Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared' > ERROR: Logfile of failure stored in: > /tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603 > NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0: > task do_cve_check: Failed > ERROR: Task (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check) > failed with exit code '1' > This is a mismatch with bitbake, because one commit hasn't reached 2.0. Steve, should I be using https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut for testing? Kind regards, Marta
On Fri, 2022-09-02 at 10:50 +0200, Marta Rybczynska wrote: > On Fri, Sep 2, 2022 at 9:09 AM Marta Rybczynska via > lists.openembedded.org <rybczynska=gmail.com@lists.openembedded.org> > wrote: > > > > On Tue, Aug 30, 2022 at 5:59 PM Joshua Watt <JPEWhacker@gmail.com> wrote: > > > > > > The CVE check database needs to have a shared lock acquired on it before > > > it is accessed. This to prevent cve-update-db-native from deleting the > > > database file out from underneath it. > > > > > > [YOCTO #14899] > > > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > > + cve_data = get_cve_info(d, patched + unpatched + ignored) > > > + cve_write_data(d, patched, unpatched, ignored, cve_data, status) > > > + else: > > > + bb.note("No CVE database found, skipping CVE check") > > > > > > > With this commit in kirkstone-nut, we're getting an error with > > meta-zephyr builds: > > > > RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0 > > do_cve_check: Error executing a python function in exec_func_python() > > autogenerated: > > The stack trace of python calls that resulted in this exception/failure was: > > File: 'exec_func_python() autogenerated', lineno: 2, function: <module> > > 0001: > > *** 0002:do_cve_check(d) > > 0003: > > File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass', > > lineno: 142, function: do_cve_check > > 0138: Check recipe for patched and unpatched CVEs > > 0139: """ > > 0140: from oe.cve_check import get_patched_cves > > 0141: > > *** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")], > > shared=True): > > 0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")): > > 0144: try: > > 0145: patched_cves = get_patched_cves(d) > > 0146: except FileNotFoundError: > > File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper > > 0236: <cleanup> > > 0237: """ > > 0238: @wraps(func) > > 0239: def helper(*args, **kwds): > > *** 0240: return _GeneratorContextManager(func, args, kwds) > > 0241: return helper > > 0242: > > 0243: > > 0244:def asynccontextmanager(func): > > File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__ > > 0079:class _GeneratorContextManagerBase: > > 0080: """Shared functionality for @contextmanager and @asynccontextmanager.""" > > 0081: > > 0082: def __init__(self, func, args, kwds): > > *** 0083: self.gen = func(*args, **kwds) > > 0084: self.func, self.args, self.kwds = func, args, kwds > > 0085: # Issue 19330: ensure context manager instances have good docstrings > > 0086: doc = getattr(func, "__doc__", None) > > 0087: if doc is None: > > Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared' > > ERROR: Logfile of failure stored in: > > /tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603 > > NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0: > > task do_cve_check: Failed > > ERROR: Task (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check) > > failed with exit code '1' > > > > This is a mismatch with bitbake, because one commit hasn't reached 2.0. > Steve, should I be using > https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut for > testing? I've gone ahead and backported that bitbake change to make things easier for people. Cheers, Richard
On Thu, Sep 1, 2022 at 10:51 PM Marta Rybczynska <rybczynska@gmail.com> wrote: > > On Fri, Sep 2, 2022 at 9:09 AM Marta Rybczynska via > lists.openembedded.org <rybczynska=gmail.com@lists.openembedded.org> > wrote: > > > > On Tue, Aug 30, 2022 at 5:59 PM Joshua Watt <JPEWhacker@gmail.com> wrote: > > > > > > The CVE check database needs to have a shared lock acquired on it before > > > it is accessed. This to prevent cve-update-db-native from deleting the > > > database file out from underneath it. > > > > > > [YOCTO #14899] > > > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > > + cve_data = get_cve_info(d, patched + unpatched + ignored) > > > + cve_write_data(d, patched, unpatched, ignored, cve_data, status) > > > + else: > > > + bb.note("No CVE database found, skipping CVE check") > > > > > > > With this commit in kirkstone-nut, we're getting an error with > > meta-zephyr builds: > > > > RROR: zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0 > > do_cve_check: Error executing a python function in exec_func_python() > > autogenerated: > > The stack trace of python calls that resulted in this exception/failure was: > > File: 'exec_func_python() autogenerated', lineno: 2, function: <module> > > 0001: > > *** 0002:do_cve_check(d) > > 0003: > > File: '/tmp/workspace.4jc1Y12y3W/oe-core/meta/classes/cve-check.bbclass', > > lineno: 142, function: do_cve_check > > 0138: Check recipe for patched and unpatched CVEs > > 0139: """ > > 0140: from oe.cve_check import get_patched_cves > > 0141: > > *** 0142: with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")], > > shared=True): > > 0143: if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")): > > 0144: try: > > 0145: patched_cves = get_patched_cves(d) > > 0146: except FileNotFoundError: > > File: '/usr/lib/python3.8/contextlib.py', lineno: 240, function: helper > > 0236: <cleanup> > > 0237: """ > > 0238: @wraps(func) > > 0239: def helper(*args, **kwds): > > *** 0240: return _GeneratorContextManager(func, args, kwds) > > 0241: return helper > > 0242: > > 0243: > > 0244:def asynccontextmanager(func): > > File: '/usr/lib/python3.8/contextlib.py', lineno: 83, function: __init__ > > 0079:class _GeneratorContextManagerBase: > > 0080: """Shared functionality for @contextmanager and @asynccontextmanager.""" > > 0081: > > 0082: def __init__(self, func, args, kwds): > > *** 0083: self.gen = func(*args, **kwds) > > 0084: self.func, self.args, self.kwds = func, args, kwds > > 0085: # Issue 19330: ensure context manager instances have good docstrings > > 0086: doc = getattr(func, "__doc__", None) > > 0087: if doc is None: > > Exception: TypeError: fileslocked() got an unexpected keyword argument 'shared' > > ERROR: Logfile of failure stored in: > > /tmp/workspace.4jc1Y12y3W/build/tmp-newlib/work/i586-yocto-elf/zephyr-philosophers/3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0/temp/log.do_cve_check.433603 > > NOTE: recipe zephyr-philosophers-3.1.0+gitAUTOINC+2ddd73feaf_5f86244bad-r0: > > task do_cve_check: Failed > > ERROR: Task (/tmp/workspace.4jc1Y12y3W/oe-core/../meta-zephyr/meta-zephyr-core/recipes-kernel/zephyr-kernel/zephyr-philosophers.bb:do_cve_check) > > failed with exit code '1' > > > > This is a mismatch with bitbake, because one commit hasn't reached 2.0. > Steve, should I be using > https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut for > testing? If you are going to try to use oe-core stable/kirkstone-nut then, yes, you should also use bitbake stable/2.0-nut since occasionally changes in core must be paired with a bitbake change. However . . . I highly discourage anyone from using the -nut branches! They are guaranteed to be rebased (and broken!), sometimes several times a day. So be prepared for pain! Patches move to stable/kirkstone-next (and stable2.0-next) a day after I post a patchset for review. At this point they have passed all testing and are unlikely to change unless the community finds an issue during the review period. So it would be safer to use the stable/*-next branch (but I still discourage it since it is possible it will change) Steve > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#170239): https://lists.openembedded.org/g/openembedded-core/message/170239 > Mute This Topic: https://lists.openembedded.org/mt/93352038/3617601 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [sakoman@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- >
diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index d95465775d..5c8b512c11 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -145,17 +145,18 @@ python do_cve_check () { """ from oe.cve_check import get_patched_cves - if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")): - try: - patched_cves = get_patched_cves(d) - except FileNotFoundError: - bb.fatal("Failure in searching patches") - ignored, patched, unpatched, status = check_cves(d, patched_cves) - if patched or unpatched or (d.getVar("CVE_CHECK_COVERAGE") == "1" and status): - cve_data = get_cve_info(d, patched + unpatched + ignored) - cve_write_data(d, patched, unpatched, ignored, cve_data, status) - else: - bb.note("No CVE database found, skipping CVE check") + with bb.utils.fileslocked([d.getVar("CVE_CHECK_DB_FILE_LOCK")], shared=True): + if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")): + try: + patched_cves = get_patched_cves(d) + except FileNotFoundError: + bb.fatal("Failure in searching patches") + ignored, patched, unpatched, status = check_cves(d, patched_cves) + if patched or unpatched or (d.getVar("CVE_CHECK_COVERAGE") == "1" and status): + cve_data = get_cve_info(d, patched + unpatched + ignored) + cve_write_data(d, patched, unpatched, ignored, cve_data, status) + else: + bb.note("No CVE database found, skipping CVE check") }
The CVE check database needs to have a shared lock acquired on it before it is accessed. This to prevent cve-update-db-native from deleting the database file out from underneath it. [YOCTO #14899] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/cve-check.bbclass | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)