| Message ID | 20250407093557.50424-1-peter.marko@siemens.com |
|---|---|
| State | New |
| Headers | show |
| Series | cve-update-nvd2-native: add workaround for json5 style list | expand |
Dear community, It looks like NVD introduces new bug in their API 2.0 responses every week. (e.g. last week https://git.openembedded.org/openembedded-core/commit/?id=8ce06538c9cde0f09909a5a2e61ec10b0d35df49) I know that this is an ugly patch, but I propose it anyway. We probably don't want to invest large effort in redesigning to json5 without official statement from NVD. For master this is a minor issue as it has already switched to FKIE as the default source. But scarthgap/kirkstone this is currently the only source for cve-check feature. Shall we consider backporting the FKIE to LTS branches? And meanwhile backport this patch so that cve-check works again? Peter > -----Original Message----- > From: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com> > Sent: Monday, April 7, 2025 11:36 > To: openembedded-core@lists.openembedded.org > Cc: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com> > Subject: [OE-core][PATCH] cve-update-nvd2-native: add workaround for json5 > style list > > From: Peter Marko <peter.marko@siemens.com> > > NVD responses changed to an invalid json between: > * April 5, 2025 at 3:03:44 AM GMT+2 > * April 5, 2025 at 4:19:48 AM GMT+2 > > The last response is since then in format > { > "resultsPerPage": 625, > "startIndex": 288000, > "totalResults": 288625, > "format": "NVD_CVE", > "version": "2.0", > "timestamp": "2025-04-07T07:17:17.534", > "vulnerabilities": [ > {...}, > ... > {...}, > ] > } > > Json does not allow trailing , in responses, that is json5 format. > So cve-update-nvd2-native do_Fetch task fails with log backtrace ending: > > ... > File: '/builds/ccp/meta-siemens/projects/ccp/../../poky/meta/recipes-core/meta/cve- > update-nvd2-native.bb', lineno: 234, function: update_db_file > 0230: if raw_data is None: > 0231: # We haven't managed to download data > 0232: return False > 0233: > *** 0234: data = json.loads(raw_data) > 0235: > 0236: index = data["startIndex"] > 0237: total = data["totalResults"] > 0238: per_page = data["resultsPerPage"] > ... > File: '/usr/lib/python3.11/json/decoder.py', lineno: 355, function: raw_decode > 0351: """ > 0352: try: > 0353: obj, end = self.scan_once(s, idx) > 0354: except StopIteration as err: > *** 0355: raise JSONDecodeError("Expecting value", s, err.value) from > None > 0356: return obj, end > Exception: json.decoder.JSONDecodeError: Expecting value: line 1 column > 1442633 (char 1442632) > ... > > There was no announcement about json format of API v2.0 by nvd. > Also this happens only if whole database is queried (database update is > fine, even when multiple pages as queried). > And lastly it's only the cve list, all other lists inside are fine. > So this looks like a bug in NVD 2.0 introduced with some update. > > Patch this with simple character deletion for now and let's monitor the > situation and possibly switch to json5 in the future. > Note that there is no native json5 support in python, we'd have to use > one of external libraries for it. > > Signed-off-by: Peter Marko <peter.marko@siemens.com> > --- > meta/recipes-core/meta/cve-update-nvd2-native.bb | 5 +++++ > 1 file changed, 5 insertions(+) > > diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes- > core/meta/cve-update-nvd2-native.bb > index b9c18bf6b6..32a14a932b 100644 > --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb > +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb > @@ -229,6 +229,11 @@ def update_db_file(db_tmp_file, d, database_time): > # We haven't managed to download data > return False > > + # hack for json5 style responses > + if raw_data[-3:] == ',]}': > + bb.note("Removing trailing ',' from nvd response") > + raw_data = raw_data[:-3] + ']}' > + > data = json.loads(raw_data) > > index = data["startIndex"]
diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb index b9c18bf6b6..32a14a932b 100644 --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb @@ -229,6 +229,11 @@ def update_db_file(db_tmp_file, d, database_time): # We haven't managed to download data return False + # hack for json5 style responses + if raw_data[-3:] == ',]}': + bb.note("Removing trailing ',' from nvd response") + raw_data = raw_data[:-3] + ']}' + data = json.loads(raw_data) index = data["startIndex"]