cve-update: log timestamps and add force update for future time

From: Peter Marko <peter.marko@siemens.com>

CVE update is currently not working properly on autobuilder.
This improves logging for problem analysis.

Future time is something which could be reason for current autobuilder
problems since the DB was not updated for more than 3 months by now.

Signed-off-by: Peter Marko <peter.marko@siemens.com>
---
 meta/recipes-core/meta/cve-update-db-native.bb   |  7 ++++++-
 meta/recipes-core/meta/cve-update-nvd2-native.bb | 11 ++++++++---
 2 files changed, 14 insertions(+), 4 deletions(-)

On 26.8.2025 21.59, Peter Marko via lists.openembedded.org wrote:
> From: Peter Marko <peter.marko@siemens.com>
> 
> CVE update is currently not working properly on autobuilder.
> This improves logging for problem analysis.
> 
> Future time is something which could be reason for current autobuilder
> problems since the DB was not updated for more than 3 months by now.
> 

CVE check mechanism, which relies on a CVE database that can silently fail to update for several months, seems troubling to say the least.

Looking at the do_fetch() method in both recipes they seem to use the os.path.getmtime() method to resolve the last modified time of the database. The value is however based on database file modification time. This seems inherently fragile, and I would be curious if there would be a less problem-prone approach that could be used to resolve the last modified date, such as resorting to the most recent CVE last modified date. Seems it can be extracted from the database file by issuing:

   select MODIFIED from NVD order by MODIFIED desc limit 1;

I'm not sure if all things considered it is completely appropriate to utilize the NVD MODIFIED value in this fashion, but it would feel at least less hazardous than referencing the modification time of the database file itself.

-Niko


> Signed-off-by: Peter Marko <peter.marko@siemens.com>
> ---
>   meta/recipes-core/meta/cve-update-db-native.bb   |  7 ++++++-
>   meta/recipes-core/meta/cve-update-nvd2-native.bb | 11 ++++++++---
>   2 files changed, 14 insertions(+), 4 deletions(-)
> 
> diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb
> index 0c7bc5f4151..713c73e574f 100644
> --- a/meta/recipes-core/meta/cve-update-db-native.bb
> +++ b/meta/recipes-core/meta/cve-update-db-native.bb
> @@ -57,7 +57,12 @@ python do_fetch() {
>               if not os.path.exists(db_file):
>                   bb.error("CVE database %s not present, database fetch/update skipped" % db_file)
>               return
> -        if time.time() - os.path.getmtime(db_file) < update_interval:
> +        curr_time = time.time()
> +        database_time = os.path.getmtime(db_file)
> +        bb.note("Current time: %s; DB time: %s" % (time.ctime(curr_time), time.ctime(database_time)))
> +        if curr_time < database_time:
> +            bb.warn("Database time is in the future, force DB update")
> +        elif curr_time - database_time < update_interval:
>               bb.note("CVE database recently updated, skipping")
>               return
>   
> diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb
> index f7a306c995c..1411d16e20a 100644
> --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
> +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
> @@ -71,10 +71,15 @@ python do_fetch() {
>               if not os.path.exists(db_file):
>                   bb.error("CVE database %s not present, database fetch/update skipped" % db_file)
>               return
> -        if time.time() - os.path.getmtime(db_file) < update_interval:
> -            bb.note("CVE database recently updated, skipping")
> -            return
> +        curr_time = time.time()
>           database_time = os.path.getmtime(db_file)
> +        bb.note("Current time: %s; DB time: %s" % (time.ctime(curr_time), time.ctime(database_time)))
> +        if curr_time < database_time:
> +            bb.warn("Database time is in the future, force DB update")
> +            database_time = 0
> +        elif curr_time - database_time < update_interval:
> +            bb.note("CVE database recently updated, skipping")
> +            return
>   
>       except OSError:
>           pass
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#222475): https://lists.openembedded.org/g/openembedded-core/message/222475
> Mute This Topic: https://lists.openembedded.org/mt/114905419/3618471
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [niko.mauno@vaisala.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

> -----Original Message-----
> From: Niko Mauno <niko.mauno@vaisala.com>
> Sent: Thursday, September 18, 2025 14:48
> To: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>;
> openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core][PATCH] cve-update: log timestamps and add force update
> for future time
> 
> On 26.8.2025 21.59, Peter Marko via lists.openembedded.org wrote:
> > From: Peter Marko <peter.marko@siemens.com>
> >
> > CVE update is currently not working properly on autobuilder.
> > This improves logging for problem analysis.
> >
> > Future time is something which could be reason for current autobuilder
> > problems since the DB was not updated for more than 3 months by now.
> >
> 
> CVE check mechanism, which relies on a CVE database that can silently fail to
> update for several months, seems troubling to say the least.
> 
> Looking at the do_fetch() method in both recipes they seem to use the
> os.path.getmtime() method to resolve the last modified time of the database. The
> value is however based on database file modification time. This seems inherently
> fragile, and I would be curious if there would be a less problem-prone approach
> that could be used to resolve the last modified date, such as resorting to the most
> recent CVE last modified date. Seems it can be extracted from the database file
> by issuing:
> 
>    select MODIFIED from NVD order by MODIFIED desc limit 1;
> 
> I'm not sure if all things considered it is completely appropriate to utilize the NVD
> MODIFIED value in this fashion, but it would feel at least less hazardous than
> referencing the modification time of the database file itself.
> 

I was considering multiple options, also using a text file with timestamp instead of the getmtime.
Your proposal looks nicer.
But I decided to first look at the AB logs before implementing something.

Unfortunately, my patch to expose the logs was not merged yet :-(
https://lists.yoctoproject.org/g/yocto-patches/message/2083

> -Niko
> 
> 
> > Signed-off-by: Peter Marko <peter.marko@siemens.com>
> > ---
> >   meta/recipes-core/meta/cve-update-db-native.bb   |  7 ++++++-
> >   meta/recipes-core/meta/cve-update-nvd2-native.bb | 11 ++++++++---
> >   2 files changed, 14 insertions(+), 4 deletions(-)
> >
> > diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-
> core/meta/cve-update-db-native.bb
> > index 0c7bc5f4151..713c73e574f 100644
> > --- a/meta/recipes-core/meta/cve-update-db-native.bb
> > +++ b/meta/recipes-core/meta/cve-update-db-native.bb
> > @@ -57,7 +57,12 @@ python do_fetch() {
> >               if not os.path.exists(db_file):
> >                   bb.error("CVE database %s not present, database fetch/update
> skipped" % db_file)
> >               return
> > -        if time.time() - os.path.getmtime(db_file) < update_interval:
> > +        curr_time = time.time()
> > +        database_time = os.path.getmtime(db_file)
> > +        bb.note("Current time: %s; DB time: %s" % (time.ctime(curr_time),
> time.ctime(database_time)))
> > +        if curr_time < database_time:
> > +            bb.warn("Database time is in the future, force DB update")
> > +        elif curr_time - database_time < update_interval:
> >               bb.note("CVE database recently updated, skipping")
> >               return
> >
> > diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-
> core/meta/cve-update-nvd2-native.bb
> > index f7a306c995c..1411d16e20a 100644
> > --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
> > +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
> > @@ -71,10 +71,15 @@ python do_fetch() {
> >               if not os.path.exists(db_file):
> >                   bb.error("CVE database %s not present, database fetch/update
> skipped" % db_file)
> >               return
> > -        if time.time() - os.path.getmtime(db_file) < update_interval:
> > -            bb.note("CVE database recently updated, skipping")
> > -            return
> > +        curr_time = time.time()
> >           database_time = os.path.getmtime(db_file)
> > +        bb.note("Current time: %s; DB time: %s" % (time.ctime(curr_time),
> time.ctime(database_time)))
> > +        if curr_time < database_time:
> > +            bb.warn("Database time is in the future, force DB update")
> > +            database_time = 0
> > +        elif curr_time - database_time < update_interval:
> > +            bb.note("CVE database recently updated, skipping")
> > +            return
> >
> >       except OSError:
> >           pass
> >
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#222475): https://lists.openembedded.org/g/openembedded-
> core/message/222475
> > Mute This Topic: https://lists.openembedded.org/mt/114905419/3618471
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
> [niko.mauno@vaisala.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >

On Thu, 2025-09-18 at 12:55 +0000, Marko, Peter wrote:
> 
> 
> > -----Original Message-----
> > From: Niko Mauno <niko.mauno@vaisala.com>
> > Sent: Thursday, September 18, 2025 14:48
> > To: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>;
> > openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core][PATCH] cve-update: log timestamps and add force update
> > for future time
> > 
> > On 26.8.2025 21.59, Peter Marko via lists.openembedded.org wrote:
> > > From: Peter Marko <peter.marko@siemens.com>
> > > 
> > > CVE update is currently not working properly on autobuilder.
> > > This improves logging for problem analysis.
> > > 
> > > Future time is something which could be reason for current autobuilder
> > > problems since the DB was not updated for more than 3 months by now.
> > > 
> > 
> > CVE check mechanism, which relies on a CVE database that can silently fail to
> > update for several months, seems troubling to say the least.
> > 
> > Looking at the do_fetch() method in both recipes they seem to use the
> > os.path.getmtime() method to resolve the last modified time of the database. The
> > value is however based on database file modification time. This seems inherently
> > fragile, and I would be curious if there would be a less problem-prone approach
> > that could be used to resolve the last modified date, such as resorting to the most
> > recent CVE last modified date. Seems it can be extracted from the database file
> > by issuing:
> > 
> >    select MODIFIED from NVD order by MODIFIED desc limit 1;
> > 
> > I'm not sure if all things considered it is completely appropriate to utilize the NVD
> > MODIFIED value in this fashion, but it would feel at least less hazardous than
> > referencing the modification time of the database file itself.
> > 
> 
> I was considering multiple options, also using a text file with timestamp instead of the getmtime.
> Your proposal looks nicer.
> But I decided to first look at the AB logs before implementing something.
> 
> Unfortunately, my patch to expose the logs was not merged yet :-(
> https://lists.yoctoproject.org/g/yocto-patches/message/2083

Basically I'm worried that this is something to debug a one off issue
but we'll be left with files building up on the shared storage. We've
multiple other debugging experiments like this from previous such
situations. People add them but never remove things.

I don't know how to handle it but I'm worried about things like this
being left indefinitely and not scaling.

Cheers,

Richard

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: Friday, September 19, 2025 0:16
> To: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>; Niko Mauno
> <niko.mauno@vaisala.com>; openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core][PATCH] cve-update: log timestamps and add force update
> for future time
> 
> On Thu, 2025-09-18 at 12:55 +0000, Marko, Peter wrote:
> >
> >
> > > -----Original Message-----
> > > From: Niko Mauno <niko.mauno@vaisala.com>
> > > Sent: Thursday, September 18, 2025 14:48
> > > To: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>;
> > > openembedded-core@lists.openembedded.org
> > > Subject: Re: [OE-core][PATCH] cve-update: log timestamps and add force
> update
> > > for future time
> > >
> > > On 26.8.2025 21.59, Peter Marko via lists.openembedded.org wrote:
> > > > From: Peter Marko <peter.marko@siemens.com>
> > > >
> > > > CVE update is currently not working properly on autobuilder.
> > > > This improves logging for problem analysis.
> > > >
> > > > Future time is something which could be reason for current autobuilder
> > > > problems since the DB was not updated for more than 3 months by now.
> > > >
> > >
> > > CVE check mechanism, which relies on a CVE database that can silently fail to
> > > update for several months, seems troubling to say the least.
> > >
> > > Looking at the do_fetch() method in both recipes they seem to use the
> > > os.path.getmtime() method to resolve the last modified time of the database.
> The
> > > value is however based on database file modification time. This seems
> inherently
> > > fragile, and I would be curious if there would be a less problem-prone
> approach
> > > that could be used to resolve the last modified date, such as resorting to the
> most
> > > recent CVE last modified date. Seems it can be extracted from the database
> file
> > > by issuing:
> > >
> > >    select MODIFIED from NVD order by MODIFIED desc limit 1;
> > >
> > > I'm not sure if all things considered it is completely appropriate to utilize the
> NVD
> > > MODIFIED value in this fashion, but it would feel at least less hazardous than
> > > referencing the modification time of the database file itself.
> > >
> >
> > I was considering multiple options, also using a text file with timestamp instead
> of the getmtime.
> > Your proposal looks nicer.
> > But I decided to first look at the AB logs before implementing something.
> >
> > Unfortunately, my patch to expose the logs was not merged yet :-(
> > https://lists.yoctoproject.org/g/yocto-patches/message/2083
> 
> Basically I'm worried that this is something to debug a one off issue
> but we'll be left with files building up on the shared storage. We've
> multiple other debugging experiments like this from previous such
> situations. People add them but never remove things.

I believe that I have written the patches in a way that they are not temporary.
Then focus on improving logging which will be hopefully kept also after the potential rework of timestamp handling.
Please let me know if this can be merged or in which other direction do you think that we could go in investigating the clearly non-working yocto CVE statistics.

Peter

> 
> I don't know how to handle it but I'm worried about things like this
> being left indefinitely and not scaling.
> 
> Cheers,
> 
> Richard

On Tue, 2025-09-23 at 12:30 +0000, Marko, Peter wrote:
> 
> 
> > -----Original Message-----
> > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> > Sent: Friday, September 19, 2025 0:16
> > To: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>; Niko
> > Mauno
> > <niko.mauno@vaisala.com>; openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core][PATCH] cve-update: log timestamps and add
> > force update
> > for future time
> > 
> > On Thu, 2025-09-18 at 12:55 +0000, Marko, Peter wrote:
> > > 
> > > 
> > > > -----Original Message-----
> > > > From: Niko Mauno <niko.mauno@vaisala.com>
> > > > Sent: Thursday, September 18, 2025 14:48
> > > > To: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>;
> > > > openembedded-core@lists.openembedded.org
> > > > Subject: Re: [OE-core][PATCH] cve-update: log timestamps and
> > > > add force
> > update
> > > > for future time
> > > > 
> > > > On 26.8.2025 21.59, Peter Marko via lists.openembedded.org
> > > > wrote:
> > > > > From: Peter Marko <peter.marko@siemens.com>
> > > > > 
> > > > > CVE update is currently not working properly on autobuilder.
> > > > > This improves logging for problem analysis.
> > > > > 
> > > > > Future time is something which could be reason for current
> > > > > autobuilder
> > > > > problems since the DB was not updated for more than 3 months
> > > > > by now.
> > > > > 
> > > > 
> > > > CVE check mechanism, which relies on a CVE database that can
> > > > silently fail to
> > > > update for several months, seems troubling to say the least.
> > > > 
> > > > Looking at the do_fetch() method in both recipes they seem to
> > > > use the
> > > > os.path.getmtime() method to resolve the last modified time of
> > > > the database.
> > The
> > > > value is however based on database file modification time. This
> > > > seems
> > inherently
> > > > fragile, and I would be curious if there would be a less
> > > > problem-prone
> > approach
> > > > that could be used to resolve the last modified date, such as
> > > > resorting to the
> > most
> > > > recent CVE last modified date. Seems it can be extracted from
> > > > the database
> > file
> > > > by issuing:
> > > > 
> > > >    select MODIFIED from NVD order by MODIFIED desc limit 1;
> > > > 
> > > > I'm not sure if all things considered it is completely
> > > > appropriate to utilize the
> > NVD
> > > > MODIFIED value in this fashion, but it would feel at least less
> > > > hazardous than
> > > > referencing the modification time of the database file itself.
> > > > 
> > > 
> > > I was considering multiple options, also using a text file with
> > > timestamp instead
> > of the getmtime.
> > > Your proposal looks nicer.
> > > But I decided to first look at the AB logs before implementing
> > > something.
> > > 
> > > Unfortunately, my patch to expose the logs was not merged yet :-(
> > > https://lists.yoctoproject.org/g/yocto-patches/message/2083
> > 
> > Basically I'm worried that this is something to debug a one off
> > issue
> > but we'll be left with files building up on the shared storage.
> > We've
> > multiple other debugging experiments like this from previous such
> > situations. People add them but never remove things.
> 
> I believe that I have written the patches in a way that they are not
> temporary.
> Then focus on improving logging which will be hopefully kept also
> after the potential rework of timestamp handling.
> Please let me know if this can be merged or in which other direction
> do you think that we could go in investigating the clearly non-
> working yocto CVE statistics.

I get lots of requests to keep all kinds of build data "just in case".
If I said yes to all of them, we'd just end up with mountains of files
in an unmaintainable mess. In fact that already happens, hence the
source of frustration and my reluctance to save more logs, especially
since this isn't apparently temporary for debugging but permanent :(.

I'm frustrated as where I have said "yes" to adding extra files in the
past, they're cluttering up UIs, space on the NAS and it is left to
others to try and cleanup the mess that builds up. These files are
probably only useful to you and you alone since nobody else knows where
they end up, why or how to use them. There is no documentation on any
of this.

Yes, we could develop policies and processes to clean these things up.
Nobody has actually done that though, or has much interest in it, they
just hope someone else does.

I'm trying really hard to sort bitbake-setup and I'm sorry but I simply
don't have the bandwidth to try and work out how to resolve this :(

I guess I just merge the patch and get quietly upset/annoyed/frustrated
since if I'm seen to "block" any form of CVE work, that is a huge
problem too.

Cheers,

Richard

On Tue, Sep 23, 2025 at 2:47 PM Richard Purdie via lists.openembedded.org
<richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:

> On Tue, 2025-09-23 at 12:30 +0000, Marko, Peter wrote:
> >
> >
> > > -----Original Message-----
> > > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > Sent: Friday, September 19, 2025 0:16
> > > To: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>; Niko
> > > Mauno
> > > <niko.mauno@vaisala.com>; openembedded-core@lists.openembedded.org
> > > Subject: Re: [OE-core][PATCH] cve-update: log timestamps and add
> > > force update
> > > for future time
> > >
> > > On Thu, 2025-09-18 at 12:55 +0000, Marko, Peter wrote:
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Niko Mauno <niko.mauno@vaisala.com>
> > > > > Sent: Thursday, September 18, 2025 14:48
> > > > > To: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>;
> > > > > openembedded-core@lists.openembedded.org
> > > > > Subject: Re: [OE-core][PATCH] cve-update: log timestamps and
> > > > > add force
> > > update
> > > > > for future time
> > > > >
> > > > > On 26.8.2025 21.59, Peter Marko via lists.openembedded.org
> > > > > wrote:
> > > > > > From: Peter Marko <peter.marko@siemens.com>
> > > > > >
> > > > > > CVE update is currently not working properly on autobuilder.
> > > > > > This improves logging for problem analysis.
> > > > > >
> > > > > > Future time is something which could be reason for current
> > > > > > autobuilder
> > > > > > problems since the DB was not updated for more than 3 months
> > > > > > by now.
> > > > > >
> > > > >
> > > > > CVE check mechanism, which relies on a CVE database that can
> > > > > silently fail to
> > > > > update for several months, seems troubling to say the least.
> > > > >
> > > > > Looking at the do_fetch() method in both recipes they seem to
> > > > > use the
> > > > > os.path.getmtime() method to resolve the last modified time of
> > > > > the database.
> > > The
> > > > > value is however based on database file modification time. This
> > > > > seems
> > > inherently
> > > > > fragile, and I would be curious if there would be a less
> > > > > problem-prone
> > > approach
> > > > > that could be used to resolve the last modified date, such as
> > > > > resorting to the
> > > most
> > > > > recent CVE last modified date. Seems it can be extracted from
> > > > > the database
> > > file
> > > > > by issuing:
> > > > >
> > > > >    select MODIFIED from NVD order by MODIFIED desc limit 1;
> > > > >
> > > > > I'm not sure if all things considered it is completely
> > > > > appropriate to utilize the
> > > NVD
> > > > > MODIFIED value in this fashion, but it would feel at least less
> > > > > hazardous than
> > > > > referencing the modification time of the database file itself.
> > > > >
> > > >
> > > > I was considering multiple options, also using a text file with
> > > > timestamp instead
> > > of the getmtime.
> > > > Your proposal looks nicer.
> > > > But I decided to first look at the AB logs before implementing
> > > > something.
> > > >
> > > > Unfortunately, my patch to expose the logs was not merged yet :-(
> > > > https://lists.yoctoproject.org/g/yocto-patches/message/2083
> > >
> > > Basically I'm worried that this is something to debug a one off
> > > issue
> > > but we'll be left with files building up on the shared storage.
> > > We've
> > > multiple other debugging experiments like this from previous such
> > > situations. People add them but never remove things.
> >
> > I believe that I have written the patches in a way that they are not
> > temporary.
> > Then focus on improving logging which will be hopefully kept also
> > after the potential rework of timestamp handling.
> > Please let me know if this can be merged or in which other direction
> > do you think that we could go in investigating the clearly non-
> > working yocto CVE statistics.
>
> I get lots of requests to keep all kinds of build data "just in case".
> If I said yes to all of them, we'd just end up with mountains of files
> in an unmaintainable mess. In fact that already happens, hence the
> source of frustration and my reluctance to save more logs, especially
> since this isn't apparently temporary for debugging but permanent :(.
>
> I'm frustrated as where I have said "yes" to adding extra files in the
> past, they're cluttering up UIs, space on the NAS and it is left to
> others to try and cleanup the mess that builds up. These files are
> probably only useful to you and you alone since nobody else knows where
> they end up, why or how to use them. There is no documentation on any
> of this.
>
> Yes, we could develop policies and processes to clean these things up.
> Nobody has actually done that though, or has much interest in it, they
> just hope someone else does.
>
> I'm trying really hard to sort bitbake-setup and I'm sorry but I simply
> don't have the bandwidth to try and work out how to resolve this :(
>
> I guess I just merge the patch and get quietly upset/annoyed/frustrated
> since if I'm seen to "block" any form of CVE work, that is a huge
> problem too.
>
>
The design decision until now was that the database update is not a critical
function and not to cause an error in case it breaks.

I think it might be more appropriate to make the build with cve-check
enabled break instead. Especially when you have FKIE flow that just
should not break.

What do you think?

Kind regards,
Marta

> -----Original Message-----
> From: Marta Rybczynska <rybczynska@gmail.com>
> Sent: Tuesday, September 23, 2025 17:00
> To: richard.purdie@linuxfoundation.org
> Cc: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com>; Niko Mauno
> <niko.mauno@vaisala.com>; openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core][PATCH] cve-update: log timestamps and add force update
> for future time
> 
> 
> 
> On Tue, Sep 23, 2025 at 2:47 PM Richard Purdie via lists.openembedded.org
> <http://lists.openembedded.org>
> <richard.purdie=linuxfoundation.org@lists.openembedded.org
> <mailto:linuxfoundation.org@lists.openembedded.org> > wrote:
> 
> 
> 	On Tue, 2025-09-23 at 12:30 +0000, Marko, Peter wrote:
> 	>
> 	>
> 	> > -----Original Message-----
> 	> > From: Richard Purdie <richard.purdie@linuxfoundation.org
> <mailto:richard.purdie@linuxfoundation.org> >
> 	> > Sent: Friday, September 19, 2025 0:16
> 	> > To: Marko, Peter (FT D EU SK BFS1) <Peter.Marko@siemens.com
> <mailto:Peter.Marko@siemens.com> >; Niko
> 	> > Mauno
> 	> > <niko.mauno@vaisala.com <mailto:niko.mauno@vaisala.com> >;
> openembedded-core@lists.openembedded.org <mailto:openembedded-
> core@lists.openembedded.org>
> 	> > Subject: Re: [OE-core][PATCH] cve-update: log timestamps and add
> 	> > force update
> 	> > for future time
> 	> >
> 	> > On Thu, 2025-09-18 at 12:55 +0000, Marko, Peter wrote:
> 	> > >
> 	> > >
> 	> > > > -----Original Message-----
> 	> > > > From: Niko Mauno <niko.mauno@vaisala.com
> <mailto:niko.mauno@vaisala.com> >
> 	> > > > Sent: Thursday, September 18, 2025 14:48
> 	> > > > To: Marko, Peter (FT D EU SK BFS1)
> <Peter.Marko@siemens.com <mailto:Peter.Marko@siemens.com> >;
> 	> > > > openembedded-core@lists.openembedded.org
> <mailto:openembedded-core@lists.openembedded.org>
> 	> > > > Subject: Re: [OE-core][PATCH] cve-update: log timestamps and
> 	> > > > add force
> 	> > update
> 	> > > > for future time
> 	> > > >
> 	> > > > On 26.8.2025 21.59, Peter Marko via lists.openembedded.org
> <http://lists.openembedded.org>
> 	> > > > wrote:
> 	> > > > > From: Peter Marko <peter.marko@siemens.com
> <mailto:peter.marko@siemens.com> >
> 	> > > > >
> 	> > > > > CVE update is currently not working properly on autobuilder.
> 	> > > > > This improves logging for problem analysis.
> 	> > > > >
> 	> > > > > Future time is something which could be reason for current
> 	> > > > > autobuilder
> 	> > > > > problems since the DB was not updated for more than 3 months
> 	> > > > > by now.
> 	> > > > >
> 	> > > >
> 	> > > > CVE check mechanism, which relies on a CVE database that can
> 	> > > > silently fail to
> 	> > > > update for several months, seems troubling to say the least.
> 	> > > >
> 	> > > > Looking at the do_fetch() method in both recipes they seem to
> 	> > > > use the
> 	> > > > os.path.getmtime() method to resolve the last modified time of
> 	> > > > the database.
> 	> > The
> 	> > > > value is however based on database file modification time. This
> 	> > > > seems
> 	> > inherently
> 	> > > > fragile, and I would be curious if there would be a less
> 	> > > > problem-prone
> 	> > approach
> 	> > > > that could be used to resolve the last modified date, such as
> 	> > > > resorting to the
> 	> > most
> 	> > > > recent CVE last modified date. Seems it can be extracted from
> 	> > > > the database
> 	> > file
> 	> > > > by issuing:
> 	> > > >
> 	> > > >    select MODIFIED from NVD order by MODIFIED desc limit 1;
> 	> > > >
> 	> > > > I'm not sure if all things considered it is completely
> 	> > > > appropriate to utilize the
> 	> > NVD
> 	> > > > MODIFIED value in this fashion, but it would feel at least less
> 	> > > > hazardous than
> 	> > > > referencing the modification time of the database file itself.
> 	> > > >
> 	> > >
> 	> > > I was considering multiple options, also using a text file with
> 	> > > timestamp instead
> 	> > of the getmtime.
> 	> > > Your proposal looks nicer.
> 	> > > But I decided to first look at the AB logs before implementing
> 	> > > something.
> 	> > >
> 	> > > Unfortunately, my patch to expose the logs was not merged yet :-(
> 	> > > https://lists.yoctoproject.org/g/yocto-patches/message/2083
> 	> >
> 	> > Basically I'm worried that this is something to debug a one off
> 	> > issue
> 	> > but we'll be left with files building up on the shared storage.
> 	> > We've
> 	> > multiple other debugging experiments like this from previous such
> 	> > situations. People add them but never remove things.
> 	>
> 	> I believe that I have written the patches in a way that they are not
> 	> temporary.
> 	> Then focus on improving logging which will be hopefully kept also
> 	> after the potential rework of timestamp handling.
> 	> Please let me know if this can be merged or in which other direction
> 	> do you think that we could go in investigating the clearly non-
> 	> working yocto CVE statistics.
> 
> 	I get lots of requests to keep all kinds of build data "just in case".
> 	If I said yes to all of them, we'd just end up with mountains of files
> 	in an unmaintainable mess. In fact that already happens, hence the
> 	source of frustration and my reluctance to save more logs, especially
> 	since this isn't apparently temporary for debugging but permanent :(.
> 
> 	I'm frustrated as where I have said "yes" to adding extra files in the
> 	past, they're cluttering up UIs, space on the NAS and it is left to
> 	others to try and cleanup the mess that builds up. These files are
> 	probably only useful to you and you alone since nobody else knows where
> 	they end up, why or how to use them. There is no documentation on any
> 	of this.
> 
> 	Yes, we could develop policies and processes to clean these things up.
> 	Nobody has actually done that though, or has much interest in it, they
> 	just hope someone else does.
> 
> 	I'm trying really hard to sort bitbake-setup and I'm sorry but I simply
> 	don't have the bandwidth to try and work out how to resolve this :(
> 
> 	I guess I just merge the patch and get quietly upset/annoyed/frustrated
> 	since if I'm seen to "block" any form of CVE work, that is a huge
> 	problem too.
> 
> 
> 
> 
> The design decision until now was that the database update is not a critical
> function and not to cause an error in case it breaks.
> 
> I think it might be more appropriate to make the build with cve-check
> enabled break instead. Especially when you have FKIE flow that just
> should not break.
> 
> What do you think?

I didn't see any evidence that the database update is failing.
Clean logs suggest that the update was not even attempted.
That’s why I added logs around the timestamps (already merget to core).
However those are put into task log, not to bitbake log.
And this commit is supposed to make this log available as AB job artifact.

Regarding failing the job if update fails, I'm OK with that.

Peter

> 
> Kind regards,
> Marta