| Message ID | 20221118160850.3964225-1-mikko.rapeli@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | [v2,1/2] oeqa: add utils/data.py with get_data() function | expand |
On 18 Nov 2022, at 16:08, Mikko Rapeli via lists.openembedded.org <mikko.rapeli=linaro.org@lists.openembedded.org> wrote: > > get_data() uses oeqa test method name and an optional > key to get data from image specific "testimage_data.json" > file located in image deploy directory. Image recipes can > provide custom versions of this file which configures > generic tests for a specific image when testing with > testimage.bbclass > > For example, the parselogs.py runtime test needs image > specific configuration when the image has new errors from > the kernel which acceptable and can be ignored. > > Same machine can be used to generate multiple images with different > runtime behavior so using image as the key and not machine. Note that there’s a very related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14604 In this case I need to extend parselogs *per machine*. Does this patch series support this? Ross
On Fri, 2022-11-18 at 18:08 +0200, Mikko Rapeli wrote: > get_data() uses oeqa test method name and an optional > key to get data from image specific "testimage_data.json" > file located in image deploy directory. Image recipes can > provide custom versions of this file which configures > generic tests for a specific image when testing with > testimage.bbclass > > For example, the parselogs.py runtime test needs image > specific configuration when the image has new errors from > the kernel which acceptable and can be ignored. > > Same machine can be used to generate multiple images with different > runtime behavior so using image as the key and not machine. > > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> > --- > meta/lib/oeqa/utils/data.py | 41 +++++++++++++++++++++++++++++++++++++ > 1 file changed, 41 insertions(+) > create mode 100644 meta/lib/oeqa/utils/data.py > > v2: no changes I'm afraid I'm really against adding this json approach to adding data and I'm not planning to merge it. I don't like having two different files containing recipe data in different formats and I don't like the datastore not being definitive for a given recipe. I agree we need to be able to extend this data on a per recipe basis but I think we need to do that from the recipes themselves. Just to be clear, even done with json as in this patch, it isn't something which would be an acceptable backport to an LTS. There are two other options. One is to 'abuse' variable flags: OEQA_PARSELOGS_IGNORELIST[qemux86] = "xxx" OEQA_PARSELOGS_IGNORELIST[qemuarm] = "yyy" which I don't like but could likely be made to work today. The other is to introduce some dedicated dict/list like data syntax to support more structured data which I think would be the better solution in the long run but is a bit more work to implement up front. Cheers, Richard
diff --git a/meta/lib/oeqa/utils/data.py b/meta/lib/oeqa/utils/data.py new file mode 100644 index 0000000000..539ca4a026 --- /dev/null +++ b/meta/lib/oeqa/utils/data.py @@ -0,0 +1,41 @@ +# Copyright (C) 2022 Linaro Limited +# +# SPDX-License-Identifier: MIT + +import os +import json + +from oeqa.core.utils.test import getCaseID, getCaseFile, getCaseMethod + + +def get_data(self, key = None): + """get_data() returns test case specific data to the test case implementation. + data is stored in image specific json file called "testimage_data.json" in + image deploy directory. Data matching test method name and an optional key + is returned to the test case. This data can then be used by generic test + cases to match image specific functionality and expected behavior. For example + list of expected kernel error strings, list of active systemd services etc. + can be image specific while the test case implementation to check them is + generic. Example json file for runtime test parselogs.py to ignore image + specific kernel error strings in dmesg: + + {"test_parselogs":{"ignore_errors":[ + "Error to be ignored in dmesg" + ]}} + """ + test_method = getCaseMethod(self) + self.logger.info("%s: get_data() called by test_method = %s, key = %s" % (__file__, test_method, key)) + + json_file_name = os.path.join(self.td['DEPLOY_DIR_IMAGE'], "testimage_data.json") + self.logger.debug("%s: json_file_name = %s" % (__file__, json_file_name)) + + with open(json_file_name) as json_file: + self.logger.debug("%s: json_file = %s" % (__file__, json_file)) + json_data = json.load(json_file) + self.logger.debug("%s: json_data = %s" % (__file__, json_data)) + if key: + data = json_data[test_method][key] + else: + data = json_data[test_method] + self.logger.debug("%s: data = %s" % (__file__, data)) + return data
get_data() uses oeqa test method name and an optional key to get data from image specific "testimage_data.json" file located in image deploy directory. Image recipes can provide custom versions of this file which configures generic tests for a specific image when testing with testimage.bbclass For example, the parselogs.py runtime test needs image specific configuration when the image has new errors from the kernel which acceptable and can be ignored. Same machine can be used to generate multiple images with different runtime behavior so using image as the key and not machine. Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org> --- meta/lib/oeqa/utils/data.py | 41 +++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 meta/lib/oeqa/utils/data.py v2: no changes