[v3,0/4] cve-check: allow feed selection

This series is allowing choice of the NVD feed to use, you can
configure them using the NVD_DB_VERSION variable in local.conf

Available feeds:
- NVD2 (default) - the current NVD API v2 feed
- NVD1 - the old NVD feed (deprecated, but still working)
- FKIE - the NVD feed restoration from FKIE-CAD

Marta Rybczynska (4):
  cve-update-db-native: restore
  cve-update-db-native: update structure
  cve-update-db-native: add the fkie source
  cve-check: allow feed choice

 meta/classes/cve-check.bbclass                |  14 +-
 .../recipes-core/meta/cve-update-db-native.bb | 405 ++++++++++++++++++
 2 files changed, 417 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-core/meta/cve-update-db-native.bb

On 5 Feb 2025, at 14:34, Marta Rybczynska via lists.openembedded.org <rybczynska=gmail.com@lists.openembedded.org> wrote:
> 
> This series is allowing choice of the NVD feed to use, you can
> configure them using the NVD_DB_VERSION variable in local.conf
> 
> Available feeds:
> - NVD2 (default) - the current NVD API v2 feed
> - NVD1 - the old NVD feed (deprecated, but still working)
> - FKIE - the NVD feed restoration from FKIE-CAD

Something broke.  If I pick FKIE:

$ bitbake core-image-sato --runall cve_check
…
WARNING: cve-update-db-native-1.0-r0 do_fetch: /work/ross/build/tmp/sysroots/CVE_CHECK/temp_nvdcve_1.1.db
[ I added that warning to show the path that it is trying to use ]

ERROR: cve-update-db-native-1.0-r0 do_fetch: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_fetch(d)
     0003:
File: '/home/rosbur01/Yocto/poky/meta/recipes-core/meta/cve-update-db-native.bb', lineno: 70, function: do_fetch
     0066:    bb.utils.mkdirhier(db_dir)
     0067:    if os.path.exists(db_file):
     0068:        shutil.copy2(db_file, db_tmp_file)
     0069:
 *** 0070:    if update_db_file(db_tmp_file, d) == True:
     0071:        # Update downloaded correctly, can swap files
     0072:        shutil.move(db_tmp_file, db_file)
     0073:    else:
     0074:        # Update failed, do not modify the database
File: '/home/rosbur01/Yocto/poky/meta/recipes-core/meta/cve-update-db-native.bb', lineno: 152, function: update_db_file
     0148:    is_nvd = d.getVar("NVD_DB_VERSION") == "NVD1"
     0149:
     0150:    # Connect to database
     0151:    bb.warn(db_tmp_file)
 *** 0152:    conn = sqlite3.connect(db_tmp_file)
     0153:    initialize_db(conn)
     0154:
     0155:    with bb.progress.ProgressHandler(d) as ph, open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') as cve_f:
     0156:        total_years = date.today().year + 1 - YEAR_START
Exception: sqlite3.OperationalError: unable to open database file

I’m surprised to see that meta/classes/cve-check.bbclass has CVE_CHECK_DB_DIR ?= "${STAGING_DIR}/CVE_CHECK”, I’d have thought that would go into a different directory.  I’m guessing the problem is that the directory wasn’t created?

Ross

On Mon, Feb 10, 2025 at 9:50 PM Ross Burton <Ross.Burton@arm.com> wrote:

> On 5 Feb 2025, at 14:34, Marta Rybczynska via lists.openembedded.org
> <rybczynska=gmail.com@lists.openembedded.org> wrote:
> >
> > This series is allowing choice of the NVD feed to use, you can
> > configure them using the NVD_DB_VERSION variable in local.conf
> >
> > Available feeds:
> > - NVD2 (default) - the current NVD API v2 feed
> > - NVD1 - the old NVD feed (deprecated, but still working)
> > - FKIE - the NVD feed restoration from FKIE-CAD
>
> Something broke.  If I pick FKIE:
>
> $ bitbake core-image-sato --runall cve_check
> …
> WARNING: cve-update-db-native-1.0-r0 do_fetch:
> /work/ross/build/tmp/sysroots/CVE_CHECK/temp_nvdcve_1.1.db
> [ I added that warning to show the path that it is trying to use ]
>
> ERROR: cve-update-db-native-1.0-r0 do_fetch: Error executing a python
> function in exec_func_python() autogenerated:
>
> The stack trace of python calls that resulted in this exception/failure
> was:
> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>      0001:
>  *** 0002:do_fetch(d)
>      0003:
> File: '/home/rosbur01/Yocto/poky/meta/recipes-core/meta/
> cve-update-db-native.bb', lineno: 70, function: do_fetch
>      0066:    bb.utils.mkdirhier(db_dir)
>      0067:    if os.path.exists(db_file):
>      0068:        shutil.copy2(db_file, db_tmp_file)
>      0069:
>  *** 0070:    if update_db_file(db_tmp_file, d) == True:
>      0071:        # Update downloaded correctly, can swap files
>      0072:        shutil.move(db_tmp_file, db_file)
>      0073:    else:
>      0074:        # Update failed, do not modify the database
> File: '/home/rosbur01/Yocto/poky/meta/recipes-core/meta/
> cve-update-db-native.bb', lineno: 152, function: update_db_file
>      0148:    is_nvd = d.getVar("NVD_DB_VERSION") == "NVD1"
>      0149:
>      0150:    # Connect to database
>      0151:    bb.warn(db_tmp_file)
>  *** 0152:    conn = sqlite3.connect(db_tmp_file)
>      0153:    initialize_db(conn)
>      0154:
>      0155:    with bb.progress.ProgressHandler(d) as ph,
> open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') as cve_f:
>      0156:        total_years = date.today().year + 1 - YEAR_START
> Exception: sqlite3.OperationalError: unable to open database file
>
>
There was a quite hard to hit issue caused by the fact that the temporary
file wasn't in the same directory as the final downloaded file. I've
changed that in v4, as it makes sense to keep them together, and is also
the same as the NVD2 fetcher does.

I think, however, that you would be able to hit this only when moving
between different versions of the patchset. I've tested with various
combinations of removed directories and it works fine now from this end.

Kind regards,
Marta