diff mbox series

cve-check: Add versioned CVSS vector strings

Message ID 20241130175038.870014-1-colinmca242@gmail.com
State New
Headers show
Series cve-check: Add versioned CVSS vector strings | expand

Commit Message

Colin McAllister Nov. 30, 2024, 5:50 p.m. UTC 
Currently, cve-check includes a vector string for each CVE included in
the issue list for each package. This vector string is the lowest
CVSS version that's available. For example, if a CVE has both a v2 and
v3.1 vector strint, the v2 vector string is only included.

This patch adds each supported vector string (v2, v3, and v4). For v3,
v3.1 is preferred over v3. If a vector string is not available for a
given verison, the string will default to "UNKNOWN".

Signed-off-by: Colin McAllister <colinmca242@gmail.com>
---
This is an alternative patch to "cve-update-nvd2-native: Update vector
logic" where each versioned vector string is attempted to be included.
This does introduce a breaking API change, where vectorString has been
removed and replaced with the versioned vectorString variables. I tried
to update all references to the current use of vector strings, but I
was a little confused looking at meta/lib/oe/spdx30.py. I'm not sure if
this change will affect that file. I did turn on SPDX generation and
ran another build. I looked at the outputs, but didn't see any vector
string output.

 meta/classes/cve-check.bbclass                | 20 +++++----
 meta/classes/vex.bbclass                      |  4 +-
 .../meta/cve-update-nvd2-native.bb            | 42 ++++++++++++++-----
 scripts/cve-json-to-text.py                   |  8 +++-
 4 files changed, 52 insertions(+), 22 deletions(-)

Comments

Mathieu Dubois-Briand Dec. 2, 2024, 11:33 a.m. UTC | #1 
On Sat, Nov 30, 2024 at 05:50:38PM +0000, Colin McAllister via lists.openembedded.org wrote:
> Currently, cve-check includes a vector string for each CVE included in
> the issue list for each package. This vector string is the lowest
> CVSS version that's available. For example, if a CVE has both a v2 and
> v3.1 vector strint, the v2 vector string is only included.
> 
> This patch adds each supported vector string (v2, v3, and v4). For v3,
> v3.1 is preferred over v3. If a vector string is not available for a
> given verison, the string will default to "UNKNOWN".
> 
> Signed-off-by: Colin McAllister <colinmca242@gmail.com>

Hi Colin,

Thanks for your new patch. As for last week, it seems to be triggering
some issues on the autobuilder:

ERROR: cve-update-nvd2-native-1.0-r0 do_unpack: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_unpack(d)
     0003:
File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-core/meta/cve-update-nvd2-native.bb', lineno: 105, function: do_unpack
     0101:do_fetch[vardeps] = ""
     0102:
     0103:python do_unpack() {
     0104:    import shutil
 *** 0105:    shutil.copyfile(d.getVar("CVE_CHECK_DB_DLDIR_FILE"), d.getVar("CVE_CHECK_DB_FILE"))
     0106:}
     0107:do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK} ${CVE_CHECK_DB_FILE_LOCK}"
     0108:
     0109:def cleanup_db_download(db_file, db_tmp_file):
File: '/usr/lib/python3.9/shutil.py', lineno: 264, function: copyfile
     0260:
     0261:    if not follow_symlinks and _islink(src):
     0262:        os.symlink(os.readlink(src), dst)
     0263:    else:
 *** 0264:        with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
     0265:            # macOS
     0266:            if _HAS_FCOPYFILE:
     0267:                try:
     0268:                    _fastcopy_fcopyfile(fsrc, fdst, posix._COPYFILE_DATA)
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK2/nvdcve_2-3.db'

https://valkyrie.yoctoproject.org/#/builders/76/builds/524/steps/15/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/35/builds/532/steps/14/logs/stdio

Is this something you can fix ?
Marko, Peter Dec. 2, 2024, 12:15 p.m. UTC | #2 
This is due to current NVD DB stability issues.
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database update failed
This can happen only if API calls to NVD DB fail.

So either valkyrie infrastructure needs to increase the retry settings to 20+
(via CVE_DB_UPDATE_ATTEMPTS, I did that "temporarily" two weeks ago on our infra),
or this change needs to wait until NVD DB infra is fixed (which can take a loooong time).
Of course the increase of timeout may mean the update job may take 3-4 hours more...
But once it completes, the DB file will be cached and all should return to normal.

Peter

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Mathieu Dubois-Briand via
> lists.openembedded.org
> Sent: Monday, December 2, 2024 12:34
> To: colinmca242@gmail.com
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH] cve-check: Add versioned CVSS vector strings
> 
> On Sat, Nov 30, 2024 at 05:50:38PM +0000, Colin McAllister via
> lists.openembedded.org wrote:
> > Currently, cve-check includes a vector string for each CVE included in
> > the issue list for each package. This vector string is the lowest
> > CVSS version that's available. For example, if a CVE has both a v2 and
> > v3.1 vector strint, the v2 vector string is only included.
> >
> > This patch adds each supported vector string (v2, v3, and v4). For v3,
> > v3.1 is preferred over v3. If a vector string is not available for a
> > given verison, the string will default to "UNKNOWN".
> >
> > Signed-off-by: Colin McAllister <colinmca242@gmail.com>
> 
> Hi Colin,
> 
> Thanks for your new patch. As for last week, it seems to be triggering
> some issues on the autobuilder:
> 
> ERROR: cve-update-nvd2-native-1.0-r0 do_unpack: Error executing a python
> function in exec_func_python() autogenerated:
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>      0001:
>  *** 0002:do_unpack(d)
>      0003:
> File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-
> core/meta/cve-update-nvd2-native.bb', lineno: 105, function: do_unpack
>      0101:do_fetch[vardeps] = ""
>      0102:
>      0103:python do_unpack() {
>      0104:    import shutil
>  *** 0105:    shutil.copyfile(d.getVar("CVE_CHECK_DB_DLDIR_FILE"),
> d.getVar("CVE_CHECK_DB_FILE"))
>      0106:}
>      0107:do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK}
> ${CVE_CHECK_DB_FILE_LOCK}"
>      0108:
>      0109:def cleanup_db_download(db_file, db_tmp_file):
> File: '/usr/lib/python3.9/shutil.py', lineno: 264, function: copyfile
>      0260:
>      0261:    if not follow_symlinks and _islink(src):
>      0262:        os.symlink(os.readlink(src), dst)
>      0263:    else:
>  *** 0264:        with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
>      0265:            # macOS
>      0266:            if _HAS_FCOPYFILE:
>      0267:                try:
>      0268:                    _fastcopy_fcopyfile(fsrc, fdst, posix._COPYFILE_DATA)
> Exception: FileNotFoundError: [Errno 2] No such file or directory:
> '/srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK2/nvdcve_2-
> 3.db'
> 
> https://valkyrie.yoctoproject.org/#/builders/76/builds/524/steps/15/logs/st
> dio
> https://valkyrie.yoctoproject.org/#/builders/35/builds/532/steps/14/logs/st
> dio
> 
> Is this something you can fix ?
> 
> --
> Mathieu Dubois-Briand, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
diff mbox series

Patch

diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
index 6e10dd915a..eacfa7b041 100644
--- a/meta/classes/cve-check.bbclass
+++ b/meta/classes/cve-check.bbclass
@@ -31,7 +31,7 @@ 
 CVE_PRODUCT ??= "${BPN}"
 CVE_VERSION ??= "${PV}"
 
-CVE_CHECK_DB_FILENAME ?= "nvdcve_2-2.db"
+CVE_CHECK_DB_FILENAME ?= "nvdcve_2-3.db"
 CVE_CHECK_DB_DIR ?= "${STAGING_DIR}/CVE_CHECK"
 CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/${CVE_CHECK_DB_FILENAME}"
 CVE_CHECK_DB_FILE_LOCK ?= "${CVE_CHECK_DB_FILE}.lock"
@@ -449,12 +449,14 @@  def get_cve_info(d, cve_data):
                 continue
             #cve_data[row[0]] = {}
             cve_data[row[0]]["NVD-summary"] = row[1]
-            cve_data[row[0]]["NVD-scorev2"] = row[2]
-            cve_data[row[0]]["NVD-scorev3"] = row[3]
-            cve_data[row[0]]["NVD-scorev4"] = row[4]
-            cve_data[row[0]]["NVD-modified"] = row[5]
-            cve_data[row[0]]["NVD-vector"] = row[6]
-            cve_data[row[0]]["NVD-vectorString"] = row[7]
+            cve_data[row[0]]["NVD-vectorStringV2"] = row[2]
+            cve_data[row[0]]["NVD-scorev2"] = row[3]
+            cve_data[row[0]]["NVD-vectorStringV3"] = row[4]
+            cve_data[row[0]]["NVD-scorev3"] = row[5]
+            cve_data[row[0]]["NVD-vectorStringV4"] = row[6]
+            cve_data[row[0]]["NVD-scorev4"] = row[7]
+            cve_data[row[0]]["NVD-modified"] = row[8]
+            cve_data[row[0]]["NVD-vector"] = row[9]
         cursor.close()
     conn.close()
 
@@ -539,12 +541,14 @@  def cve_write_data_json(d, cve_data, cve_status):
         }
         if 'NVD-summary' in cve_data[cve]:
             cve_item["summary"] = cve_data[cve]["NVD-summary"]
+            cve_item["vectorStringV2"] = cve_data[cve]["NVD-vectorStringV2"]
             cve_item["scorev2"] = cve_data[cve]["NVD-scorev2"]
+            cve_item["vectorStringV3"] = cve_data[cve]["NVD-vectorStringV3"]
             cve_item["scorev3"] = cve_data[cve]["NVD-scorev3"]
+            cve_item["vectorStringV4"] = cve_data[cve]["NVD-vectorStringV4"]
             cve_item["scorev4"] = cve_data[cve]["NVD-scorev4"]
             cve_item["modified"] = cve_data[cve]["NVD-modified"]
             cve_item["vector"] = cve_data[cve]["NVD-vector"]
-            cve_item["vectorString"] = cve_data[cve]["NVD-vectorString"]
         if 'status' in cve_data[cve]:
             cve_item["detail"] = cve_data[cve]["status"]
         if 'justification' in cve_data[cve]:
diff --git a/meta/classes/vex.bbclass b/meta/classes/vex.bbclass
index 01d4e52051..5e0131788d 100644
--- a/meta/classes/vex.bbclass
+++ b/meta/classes/vex.bbclass
@@ -280,11 +280,13 @@  def cve_write_data_json(d, cve_data, cve_status):
         }
         if 'NVD-summary' in cve_data[cve]:
             cve_item["summary"] = cve_data[cve]["NVD-summary"]
+            cve_item["vectorStringV2"] = cve_data[cve]["NVD-vectorStringV2"]
             cve_item["scorev2"] = cve_data[cve]["NVD-scorev2"]
+            cve_item["vectorStringV3"] = cve_data[cve]["NVD-vectorStringV3"]
             cve_item["scorev3"] = cve_data[cve]["NVD-scorev3"]
+            cve_item["vectorStringV4"] = cve_data[cve]["NVD-vectorStringV4"]
             cve_item["scorev4"] = cve_data[cve]["NVD-scorev4"]
             cve_item["vector"] = cve_data[cve]["NVD-vector"]
-            cve_item["vectorString"] = cve_data[cve]["NVD-vectorString"]
         if 'status' in cve_data[cve]:
             cve_item["detail"] = cve_data[cve]["status"]
         if 'justification' in cve_data[cve]:
diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb
index a68a8bb89f..69cd3ebb9a 100644
--- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
+++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
@@ -259,8 +259,11 @@  def initialize_db(conn):
 
         c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)")
 
-        c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \
-            SCOREV2 TEXT, SCOREV3 TEXT, SCOREV4 TEXT, MODIFIED INTEGER, VECTOR TEXT, VECTORSTRING TEXT)")
+        c.execute(
+            "CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \
+            VECTORSTRINGV2 TEXT, SCOREV2 TEXT, VECTORSTRINGV3 TEXT, SCOREV3 TEXT, \
+            VECTORSTRINGV4 TEXT, SCOREV4 TEXT, MODIFIED INTEGER, VECTOR TEXT)"
+        )
 
         c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \
             VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \
@@ -334,7 +337,9 @@  def update_db(conn, elt):
     """
 
     accessVector = None
-    vectorString = None
+    vectorStringV2 = None
+    vectorStringV3 = None
+    vectorStringV4 = None
     cveId = elt['cve']['id']
     if elt['cve']['vulnStatus'] ==  "Rejected":
         c = conn.cursor()
@@ -349,35 +354,50 @@  def update_db(conn, elt):
     date = elt['cve']['lastModified']
     try:
         accessVector = elt['cve']['metrics']['cvssMetricV2'][0]['cvssData']['accessVector']
-        vectorString = elt['cve']['metrics']['cvssMetricV2'][0]['cvssData']['vectorString']
+        vectorStringV2 = elt['cve']['metrics']['cvssMetricV2'][0]['cvssData']['vectorString']
         cvssv2 = elt['cve']['metrics']['cvssMetricV2'][0]['cvssData']['baseScore']
     except KeyError:
         cvssv2 = 0.0
     cvssv3 = None
     try:
         accessVector = accessVector or elt['cve']['metrics']['cvssMetricV30'][0]['cvssData']['attackVector']
-        vectorString = vectorString or elt['cve']['metrics']['cvssMetricV30'][0]['cvssData']['vectorString']
+        vectorStringV3 = elt['cve']['metrics']['cvssMetricV30'][0]['cvssData']['vectorString']
         cvssv3 = elt['cve']['metrics']['cvssMetricV30'][0]['cvssData']['baseScore']
     except KeyError:
         pass
     try:
         accessVector = accessVector or elt['cve']['metrics']['cvssMetricV31'][0]['cvssData']['attackVector']
-        vectorString = vectorString or elt['cve']['metrics']['cvssMetricV31'][0]['cvssData']['vectorString']
+        vectorStringV3 = elt['cve']['metrics']['cvssMetricV31'][0]['cvssData']['vectorString']
         cvssv3 = cvssv3 or elt['cve']['metrics']['cvssMetricV31'][0]['cvssData']['baseScore']
     except KeyError:
         pass
     cvssv3 = cvssv3 or 0.0
     try:
         accessVector = accessVector or elt['cve']['metrics']['cvssMetricV40'][0]['cvssData']['attackVector']
-        vectorString = vectorString or elt['cve']['metrics']['cvssMetricV40'][0]['cvssData']['vectorString']
+        vectorStringV4 = elt['cve']['metrics']['cvssMetricV40'][0]['cvssData']['vectorString']
         cvssv4 = elt['cve']['metrics']['cvssMetricV40'][0]['cvssData']['baseScore']
     except KeyError:
         cvssv4 = 0.0
     accessVector = accessVector or "UNKNOWN"
-    vectorString = vectorString or "UNKNOWN"
-
-    conn.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?, ?, ?)",
-                [cveId, cveDesc, cvssv2, cvssv3, cvssv4, date, accessVector, vectorString]).close()
+    vectorStringV2 = vectorStringV2 or "UNKNOWN"
+    vectorStringV3 = vectorStringV3 or "UNKNOWN"
+    vectorStringV4 = vectorStringV4 or "UNKNOWN"
+
+    conn.execute(
+        "insert or replace into NVD values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
+        [
+            cveId,
+            cveDesc,
+            vectorStringV2,
+            cvssv2,
+            vectorStringV3,
+            cvssv3,
+            vectorStringV4,
+            cvssv4,
+            date,
+            accessVector,
+        ],
+    ).close()
 
     try:
         # Remove any pre-existing CVE configuration. Even for partial database
diff --git a/scripts/cve-json-to-text.py b/scripts/cve-json-to-text.py
index 87a5669987..3bab2426d6 100755
--- a/scripts/cve-json-to-text.py
+++ b/scripts/cve-json-to-text.py
@@ -121,16 +121,20 @@  def process_data(filename, data):
                 lines += "CVE DESCRIPTION: %s\n" % issue["description"]
             if "summary" in issue:
                 lines += "CVE SUMMARY: %s\n" % issue["summary"]
+            if "vectorStringV2" in issue:
+                lines += "VECTORSTRING v2: %s\n" % issue["vectorStringV2"]
             if "scorev2" in issue:
                 lines += "CVSS v2 BASE SCORE: %s\n" % issue["scorev2"]
+            if "vectorStringV3" in issue:
+                lines += "VECTORSTRING v3: %s\n" % issue["vectorStringV3"]
             if "scorev3" in issue:
                 lines += "CVSS v3 BASE SCORE: %s\n" % issue["scorev3"]
+            if "vectorStringV4" in issue:
+                lines += "VECTORSTRING v4: %s\n" % issue["vectorStringV4"]
             if "scorev4" in issue:
                 lines += "CVSS v4 BASE SCORE: %s\n" % issue["scorev4"]
             if "vector" in issue:
                 lines += "VECTOR: %s\n" % issue["vector"]
-            if "vectorString" in issue:
-                lines += "VECTORSTRING: %s\n" % issue["vectorString"]
             lines += "MORE INFORMATION: https://nvd.nist.gov/vuln/detail/%s\n" % issue["id"]
             lines += "\n"