[poky,master,1/3] cve_check.py: Add new method get_ignored_cves

Message ID 20220511143613.25002-1-akash.hadke@kpit.com
State New
Headers show
Series [poky,master,1/3] cve_check.py: Add new method get_ignored_cves | expand

Commit Message

Akash Hadke May 11, 2022, 2:36 p.m. UTC
Add new method get_ignored_cves in cve_check.py
to get ignored CVEs from recipe by excluding distro-wide
ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc

While calling this method use below code to get argument values
paths = d.getVar('PATH').split(':')
cves = d.getVar('CVE_CHECK_IGNORE').split()

Signed-off-by: Akash Hadke <akash.hadke@kpit.com>
Signed-off-by: Akash Hadke <hadkeakash4@gmail.com>
---
 meta/lib/oe/cve_check.py | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

Comments

Marta Rybczynska May 17, 2022, 9:12 a.m. UTC | #1
On Wed, May 11, 2022 at 4:37 PM akash hadke via lists.openembedded.org
<akash.hadke=kpit.com@lists.openembedded.org> wrote:
>
> Add new method get_ignored_cves in cve_check.py
> to get ignored CVEs from recipe by excluding distro-wide
> ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc
>
> While calling this method use below code to get argument values
> paths = d.getVar('PATH').split(':')
> cves = d.getVar('CVE_CHECK_IGNORE').split()
>

Hello Akash,
While looking into this patch set I'm wondering what is your use case.
It seems to be to get a list
of ignored and patched CVEs. This is already available from the
cve-check output or from the create-spdx
output after some parsing. With the new JSON format for cve-check it
becomes very easy. If you could
elaborate more on the way you plan to use this data, I'm pretty sure
we can come with a simple
post-processing script to do the same.

BTW Why do assume people always include
meta/conf/distro/include/cve-extra-exclusions.inc ?
We don't do that at Oniro and we use our own judgement on outstanding CVEs.

Regards,
Marta
Akash Hadke May 17, 2022, 11:42 a.m. UTC | #2
Hello Marta,

Actually, I wanted to add the ignored and patched CVEs in buildhistory and for that purpose, I am exporting variables CVE_IGNORED and CVE_PATCHED with those values. I don't want to use cve-check.bbclass as it checks for the CVEs from the NVD database, and I only want to get ignored and patched CVEs from the recipe.

Regarding meta/conf/distro/include/cve-extra-exclusions.inc if any project includes it then CVEs that are ignored in cve-extra-exclusions.inc will get shown for each recipe in the CVE_CHECK_IGNORED list even though the CVEs are not related to that component recipe. Hence, I have did the changes to exclude CVEs from cve-extra-exclusions.inc ‚Äč

Best Regards,
Akash
________________________________
From: Marta Rybczynska <rybczynska@gmail.com>
Sent: 17 May 2022 14:42
To: Akash Hadke <Akash.Hadke@kpit.com>
Cc: OE-core <openembedded-core@lists.openembedded.org>; Ranjitsinh Rathod <Ranjitsinh.Rathod@kpit.com>; Akash Hadke <hadkeakash4@gmail.com>
Subject: Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves

Caution: This email originated from outside of the KPIT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

On Wed, May 11, 2022 at 4:37 PM akash hadke via lists.openembedded.org
<akash.hadke=kpit.com@lists.openembedded.org> wrote:
>
> Add new method get_ignored_cves in cve_check.py
> to get ignored CVEs from recipe by excluding distro-wide
> ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc
>
> While calling this method use below code to get argument values
> paths = d.getVar('PATH').split(':')
> cves = d.getVar('CVE_CHECK_IGNORE').split()
>

Hello Akash,
While looking into this patch set I'm wondering what is your use case.
It seems to be to get a list
of ignored and patched CVEs. This is already available from the
cve-check output or from the create-spdx
output after some parsing. With the new JSON format for cve-check it
becomes very easy. If you could
elaborate more on the way you plan to use this data, I'm pretty sure
we can come with a simple
post-processing script to do the same.

BTW Why do assume people always include
meta/conf/distro/include/cve-extra-exclusions.inc ?
We don't do that at Oniro and we use our own judgement on outstanding CVEs.

Regards,
Marta
Marta Rybczynska May 17, 2022, 1:33 p.m. UTC | #3
On Tue, May 17, 2022 at 1:42 PM Akash Hadke <Akash.Hadke@kpit.com> wrote:
>
> Hello Marta,
>
> Actually, I wanted to add the ignored and patched CVEs in buildhistory and for that purpose, I am exporting variables CVE_IGNORED and CVE_PATCHED with those values. I don't want to use cve-check.bbclass as it checks for the CVEs from the NVD database, and I only want to get ignored and patched CVEs from the recipe.

Hello again Akash,
What you'd like to do is to see the difference in ignored and patched
CVEs in buildhistory? Do I get it right?

>
> Regarding meta/conf/distro/include/cve-extra-exclusions.inc if any project includes it then CVEs that are ignored in cve-extra-exclusions.inc will get shown for each recipe in the CVE_CHECK_IGNORED list even though the CVEs are not related to that component recipe. Hence, I have did the changes to exclude CVEs from cve-extra-exclusions.inc

I think I understand the idea. The point I'm making is that if someone
does not include the cve-extra-exclusions.inc in their distro, the
code will still use it and filter out CVEs they still see when doing
cve-check.

Kind regards,
Marta

>
> Best Regards,
> Akash
> ________________________________
> From: Marta Rybczynska <rybczynska@gmail.com>
> Sent: 17 May 2022 14:42
> To: Akash Hadke <Akash.Hadke@kpit.com>
> Cc: OE-core <openembedded-core@lists.openembedded.org>; Ranjitsinh Rathod <Ranjitsinh.Rathod@kpit.com>; Akash Hadke <hadkeakash4@gmail.com>
> Subject: Re: [OE-core] [poky][master][PATCH 1/3] cve_check.py: Add new method get_ignored_cves
>
> Caution: This email originated from outside of the KPIT. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> On Wed, May 11, 2022 at 4:37 PM akash hadke via lists.openembedded.org
> <akash.hadke=kpit.com@lists.openembedded.org> wrote:
> >
> > Add new method get_ignored_cves in cve_check.py
> > to get ignored CVEs from recipe by excluding distro-wide
> > ignored CVEs from meta/conf/distro/include/cve-extra-exclusions.inc
> >
> > While calling this method use below code to get argument values
> > paths = d.getVar('PATH').split(':')
> > cves = d.getVar('CVE_CHECK_IGNORE').split()
> >
>
> Hello Akash,
> While looking into this patch set I'm wondering what is your use case.
> It seems to be to get a list
> of ignored and patched CVEs. This is already available from the
> cve-check output or from the create-spdx
> output after some parsing. With the new JSON format for cve-check it
> becomes very easy. If you could
> elaborate more on the way you plan to use this data, I'm pretty sure
> we can come with a simple
> post-processing script to do the same.
>
> BTW Why do assume people always include
> meta/conf/distro/include/cve-extra-exclusions.inc ?
> We don't do that at Oniro and we use our own judgement on outstanding CVEs.
>
> Regards,
> Marta
Akash Hadke May 17, 2022, 1:51 p.m. UTC | #4
Hi Marta,

If you see the code from the succeeding patch to this https://lists.openembedded.org/g/openembedded-core/message/165502 here I have checked if cve-extra-exclusions.inc is included or not. If it is not included then the code will not get executed.
Richard Purdie May 17, 2022, 2:19 p.m. UTC | #5
On Tue, 2022-05-17 at 11:42 +0000, akash hadke via lists.openembedded.org wrote:
> Actually, I wanted to add the ignored and patched CVEs in
> buildhistory and for that purpose, I am exporting variables
> CVE_IGNORED and CVE_PATCHED with those values. I don't want to use
> cve-check.bbclass as it checks for the CVEs from the NVD database,
> and I only want to get ignored and patched CVEs from the recipe.

I'd really prefer to have one cve handling class where we can configure
it to get the data different people need rather than multiple
difference cve classes which are going to confuse people. Could we have
a way to disable NVD data from the cve-check class?

Cheers,

Richard
Akash Hadke May 18, 2022, 9:46 a.m. UTC | #6
Hi Richard,

I tried modifying the cve-check.bbclass but did not able to get the solution for disabling the NVD data, because when we inherit cve-check it executes the cve_check task that checks CVEs from NVD DB. So I am also not sure how to disable it hence I used the separate bbclass.
Richard Purdie May 18, 2022, 10:33 a.m. UTC | #7
On Wed, 2022-05-18 at 02:46 -0700, akash hadke via
lists.openembedded.org wrote:
> Hi Richard,
> 
> I tried modifying the cve-check.bbclass but did not able to get the
> solution for disabling the NVD data, because when we inherit cve-
> check it executes the cve_check task that checks CVEs from NVD DB. So
> I am also not sure how to disable it hence I used the separate
> bbclass.

I guess the task dependencies are a bit of an issue there. There are
probably ways to make those configurable, it would just take a little
work.

I really do want to encourage us to work together on common cve tooling
rather than having several partial implementations so I can't take this
patch series.

Cheers,

Richard
Marta Rybczynska May 18, 2022, 11:58 a.m. UTC | #8
On Wed, May 18, 2022 at 12:33 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2022-05-18 at 02:46 -0700, akash hadke via
> lists.openembedded.org wrote:
> > Hi Richard,
> >
> > I tried modifying the cve-check.bbclass but did not able to get the
> > solution for disabling the NVD data, because when we inherit cve-
> > check it executes the cve_check task that checks CVEs from NVD DB. So
> > I am also not sure how to disable it hence I used the separate
> > bbclass.
>
> I guess the task dependencies are a bit of an issue there. There are
> probably ways to make those configurable, it would just take a little
> work.
>
> I really do want to encourage us to work together on common cve tooling
> rather than having several partial implementations so I can't take this
> patch series.
>

I plan to cut those dependencies and make different stages optional.
In my case the first use
is to be able to run multiple cve-checks with the same database
(guaranteed without updates)
or run cve-check with some given known database (for testing).
We can add an option to make the actual check with the database optional.

Exporting the data to the buildhistory looks like a useful thing to do, too.

I can draft the split after YPS.

Kind regards,
Marta

Patch

diff --git a/meta/lib/oe/cve_check.py b/meta/lib/oe/cve_check.py
index dc7d2e2826..d96d47b737 100644
--- a/meta/lib/oe/cve_check.py
+++ b/meta/lib/oe/cve_check.py
@@ -163,3 +163,41 @@  def cve_check_merge_jsons(output, data):
             return
 
     output["package"].append(data["package"][0])
+
+def get_ignored_cves(paths, cves):
+    """
+    Get ignored CVEs from the recipe and exlude the CVEs from
+    meta/conf/distro/include/cve-extra-exclusions.inc
+
+    While calling this method use below code to get argument values
+    paths = d.getVar('PATH').split(':')
+    cves = d.getVar('CVE_CHECK_IGNORE').split()
+    """
+    import os
+
+    cve_extra_exclusion_inc_file = "../meta/conf/distro/include/cve-extra-exclusions.inc"
+    for path in paths:
+        check_for_correct_file_path = os.path.join(path, cve_extra_exclusion_inc_file)
+        if os.path.isfile(check_for_correct_file_path):
+            inc_file = check_for_correct_file_path
+        else:
+            continue
+
+    cve_check_ignored = set()
+    ignored_cves = set()
+    with open(inc_file) as f:
+        lines = f.readlines()
+        for line in lines:
+            if line.strip():
+                # Ignore the comments from cve-extra-exclusions.inc
+                if not re.search("^#", line):
+                    cve_match = re.findall(r'CVE\-\d{4}\-\d+', line)
+                    for cve in cve_match:
+                        ignored_cves.add(cve)
+
+    for cve in cves:
+        if cve not in ignored_cves:
+            cve_check_ignored.add(cve)
+    ignored_cves_from_recipe = " ".join(cve_check_ignored)
+
+    return ignored_cves_from_recipe