cve-check: Add versioned CVSS vector strings

Currently, cve-check includes a vector string for each CVE included in
the issue list for each package. This vector string is the lowest
CVSS version that's available. For example, if a CVE has both a v2 and
v3.1 vector strint, the v2 vector string is only included.

This patch adds each supported vector string (v2, v3, and v4). For v3,
v3.1 is preferred over v3. If a vector string is not available for a
given verison, the string will default to "UNKNOWN".

Signed-off-by: Colin McAllister <colinmca242@gmail.com>
---
This is an alternative patch to "cve-update-nvd2-native: Update vector
logic" where each versioned vector string is attempted to be included.
This does introduce a breaking API change, where vectorString has been
removed and replaced with the versioned vectorString variables. I tried
to update all references to the current use of vector strings, but I
was a little confused looking at meta/lib/oe/spdx30.py. I'm not sure if
this change will affect that file. I did turn on SPDX generation and
ran another build. I looked at the outputs, but didn't see any vector
string output.

 meta/classes/cve-check.bbclass                | 20 +++++----
 meta/classes/vex.bbclass                      |  4 +-
 .../meta/cve-update-nvd2-native.bb            | 42 ++++++++++++++-----
 scripts/cve-json-to-text.py                   |  8 +++-
 4 files changed, 52 insertions(+), 22 deletions(-)

On Sat, Nov 30, 2024 at 05:50:38PM +0000, Colin McAllister via lists.openembedded.org wrote:
> Currently, cve-check includes a vector string for each CVE included in
> the issue list for each package. This vector string is the lowest
> CVSS version that's available. For example, if a CVE has both a v2 and
> v3.1 vector strint, the v2 vector string is only included.
> 
> This patch adds each supported vector string (v2, v3, and v4). For v3,
> v3.1 is preferred over v3. If a vector string is not available for a
> given verison, the string will default to "UNKNOWN".
> 
> Signed-off-by: Colin McAllister <colinmca242@gmail.com>

Hi Colin,

Thanks for your new patch. As for last week, it seems to be triggering
some issues on the autobuilder:

ERROR: cve-update-nvd2-native-1.0-r0 do_unpack: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_unpack(d)
     0003:
File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-core/meta/cve-update-nvd2-native.bb', lineno: 105, function: do_unpack
     0101:do_fetch[vardeps] = ""
     0102:
     0103:python do_unpack() {
     0104:    import shutil
 *** 0105:    shutil.copyfile(d.getVar("CVE_CHECK_DB_DLDIR_FILE"), d.getVar("CVE_CHECK_DB_FILE"))
     0106:}
     0107:do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK} ${CVE_CHECK_DB_FILE_LOCK}"
     0108:
     0109:def cleanup_db_download(db_file, db_tmp_file):
File: '/usr/lib/python3.9/shutil.py', lineno: 264, function: copyfile
     0260:
     0261:    if not follow_symlinks and _islink(src):
     0262:        os.symlink(os.readlink(src), dst)
     0263:    else:
 *** 0264:        with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
     0265:            # macOS
     0266:            if _HAS_FCOPYFILE:
     0267:                try:
     0268:                    _fastcopy_fcopyfile(fsrc, fdst, posix._COPYFILE_DATA)
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK2/nvdcve_2-3.db'

https://valkyrie.yoctoproject.org/#/builders/76/builds/524/steps/15/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/35/builds/532/steps/14/logs/stdio

Is this something you can fix ?

This is due to current NVD DB stability issues.
WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database update failed
This can happen only if API calls to NVD DB fail.

So either valkyrie infrastructure needs to increase the retry settings to 20+
(via CVE_DB_UPDATE_ATTEMPTS, I did that "temporarily" two weeks ago on our infra),
or this change needs to wait until NVD DB infra is fixed (which can take a loooong time).
Of course the increase of timeout may mean the update job may take 3-4 hours more...
But once it completes, the DB file will be cached and all should return to normal.

Peter

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Mathieu Dubois-Briand via
> lists.openembedded.org
> Sent: Monday, December 2, 2024 12:34
> To: colinmca242@gmail.com
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH] cve-check: Add versioned CVSS vector strings
> 
> On Sat, Nov 30, 2024 at 05:50:38PM +0000, Colin McAllister via
> lists.openembedded.org wrote:
> > Currently, cve-check includes a vector string for each CVE included in
> > the issue list for each package. This vector string is the lowest
> > CVSS version that's available. For example, if a CVE has both a v2 and
> > v3.1 vector strint, the v2 vector string is only included.
> >
> > This patch adds each supported vector string (v2, v3, and v4). For v3,
> > v3.1 is preferred over v3. If a vector string is not available for a
> > given verison, the string will default to "UNKNOWN".
> >
> > Signed-off-by: Colin McAllister <colinmca242@gmail.com>
> 
> Hi Colin,
> 
> Thanks for your new patch. As for last week, it seems to be triggering
> some issues on the autobuilder:
> 
> ERROR: cve-update-nvd2-native-1.0-r0 do_unpack: Error executing a python
> function in exec_func_python() autogenerated:
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>      0001:
>  *** 0002:do_unpack(d)
>      0003:
> File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-
> core/meta/cve-update-nvd2-native.bb', lineno: 105, function: do_unpack
>      0101:do_fetch[vardeps] = ""
>      0102:
>      0103:python do_unpack() {
>      0104:    import shutil
>  *** 0105:    shutil.copyfile(d.getVar("CVE_CHECK_DB_DLDIR_FILE"),
> d.getVar("CVE_CHECK_DB_FILE"))
>      0106:}
>      0107:do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK}
> ${CVE_CHECK_DB_FILE_LOCK}"
>      0108:
>      0109:def cleanup_db_download(db_file, db_tmp_file):
> File: '/usr/lib/python3.9/shutil.py', lineno: 264, function: copyfile
>      0260:
>      0261:    if not follow_symlinks and _islink(src):
>      0262:        os.symlink(os.readlink(src), dst)
>      0263:    else:
>  *** 0264:        with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
>      0265:            # macOS
>      0266:            if _HAS_FCOPYFILE:
>      0267:                try:
>      0268:                    _fastcopy_fcopyfile(fsrc, fdst, posix._COPYFILE_DATA)
> Exception: FileNotFoundError: [Errno 2] No such file or directory:
> '/srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK2/nvdcve_2-
> 3.db'
> 
> https://valkyrie.yoctoproject.org/#/builders/76/builds/524/steps/15/logs/st
> dio
> https://valkyrie.yoctoproject.org/#/builders/35/builds/532/steps/14/logs/st
> dio
> 
> Is this something you can fix ?
> 
> --
> Mathieu Dubois-Briand, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Hello,

Would the tests be able to be run again? I think the API appears to be more
stable now. If the tests are still failing, I can rebase this patch on top
of a
commit that changes the retry attempts to a higher number and send in both
patches.

Besides the issue where the database fails to rebuild due to the NIST API's
stability, is there any other feedback or are there any concerns with this
patch?

Thanks,
Colin

On Mon, Dec 2, 2024 at 6:15 AM Marko, Peter <Peter.Marko@siemens.com> wrote:

> This is due to current NVD DB stability issues.
> WARNING: cve-update-nvd2-native-1.0-r0 do_fetch: CVE database update failed
> This can happen only if API calls to NVD DB fail.
>
> So either valkyrie infrastructure needs to increase the retry settings to
> 20+
> (via CVE_DB_UPDATE_ATTEMPTS, I did that "temporarily" two weeks ago on our
> infra),
> or this change needs to wait until NVD DB infra is fixed (which can take a
> loooong time).
> Of course the increase of timeout may mean the update job may take 3-4
> hours more...
> But once it completes, the DB file will be cached and all should return to
> normal.
>
> Peter
>
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-
> > core@lists.openembedded.org> On Behalf Of Mathieu Dubois-Briand via
> > lists.openembedded.org
> > Sent: Monday, December 2, 2024 12:34
> > To: colinmca242@gmail.com
> > Cc: openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH] cve-check: Add versioned CVSS vector
> strings
> >
> > On Sat, Nov 30, 2024 at 05:50:38PM +0000, Colin McAllister via
> > lists.openembedded.org wrote:
> > > Currently, cve-check includes a vector string for each CVE included in
> > > the issue list for each package. This vector string is the lowest
> > > CVSS version that's available. For example, if a CVE has both a v2 and
> > > v3.1 vector strint, the v2 vector string is only included.
> > >
> > > This patch adds each supported vector string (v2, v3, and v4). For v3,
> > > v3.1 is preferred over v3. If a vector string is not available for a
> > > given verison, the string will default to "UNKNOWN".
> > >
> > > Signed-off-by: Colin McAllister <colinmca242@gmail.com>
> >
> > Hi Colin,
> >
> > Thanks for your new patch. As for last week, it seems to be triggering
> > some issues on the autobuilder:
> >
> > ERROR: cve-update-nvd2-native-1.0-r0 do_unpack: Error executing a python
> > function in exec_func_python() autogenerated:
> > The stack trace of python calls that resulted in this exception/failure
> was:
> > File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
> >      0001:
> >  *** 0002:do_unpack(d)
> >      0003:
> > File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-
> > core/meta/cve-update-nvd2-native.bb', lineno: 105, function: do_unpack
> >      0101:do_fetch[vardeps] = ""
> >      0102:
> >      0103:python do_unpack() {
> >      0104:    import shutil
> >  *** 0105:    shutil.copyfile(d.getVar("CVE_CHECK_DB_DLDIR_FILE"),
> > d.getVar("CVE_CHECK_DB_FILE"))
> >      0106:}
> >      0107:do_unpack[lockfiles] += "${CVE_CHECK_DB_DLDIR_LOCK}
> > ${CVE_CHECK_DB_FILE_LOCK}"
> >      0108:
> >      0109:def cleanup_db_download(db_file, db_tmp_file):
> > File: '/usr/lib/python3.9/shutil.py', lineno: 264, function: copyfile
> >      0260:
> >      0261:    if not follow_symlinks and _islink(src):
> >      0262:        os.symlink(os.readlink(src), dst)
> >      0263:    else:
> >  *** 0264:        with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
> >      0265:            # macOS
> >      0266:            if _HAS_FCOPYFILE:
> >      0267:                try:
> >      0268:                    _fastcopy_fcopyfile(fsrc, fdst,
> posix._COPYFILE_DATA)
> > Exception: FileNotFoundError: [Errno 2] No such file or directory:
> > '/srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK2/nvdcve_2-
> > 3.db'
> >
> >
> https://valkyrie.yoctoproject.org/#/builders/76/builds/524/steps/15/logs/st
> > dio
> >
> https://valkyrie.yoctoproject.org/#/builders/35/builds/532/steps/14/logs/st
> > dio
> >
> > Is this something you can fix ?
> >
> > --
> > Mathieu Dubois-Briand, Bootlin
> > Embedded Linux and Kernel engineering
> > https://bootlin.com
>