diff mbox series

[master] libyaml: Add libyaml ptest

Message ID 20241022081140.1464860-1-thakur.virendra1810@gmail.com
State New
Headers show
Series [master] libyaml: Add libyaml ptest | expand

Commit Message

Virendra Thakur Oct. 22, 2024, 8:11 a.m. UTC 
Install libyaml test suite and run it as ptest.
The run-ptest script iterates through executable
files copied from the host, runs each on the target.

example.yaml file created by using examples present in libyaml upstream.
Reference: https://github.com/yaml/libyaml/tree/master/examples

Signed-off-by: Virendra Thakur <thakur.virendra1810@gmail.com>
---
 .../distro/include/ptest-packagelists.inc     |  1 +
 .../libyaml/libyaml/example.yaml              | 45 +++++++++++++++++++
 .../recipes-support/libyaml/libyaml/run-ptest | 28 ++++++++++++
 meta/recipes-support/libyaml/libyaml_0.2.5.bb | 25 ++++++++++-
 4 files changed, 97 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-support/libyaml/libyaml/example.yaml
 create mode 100644 meta/recipes-support/libyaml/libyaml/run-ptest

Comments

Alexander Kanavin Oct. 22, 2024, 8:23 a.m. UTC | #1 
On Tue, 22 Oct 2024 at 10:13, virendra thakur via
lists.openembedded.org
<thakur.virendra1810=gmail.com@lists.openembedded.org> wrote:
> +++ b/meta/recipes-support/libyaml/libyaml/run-ptest
> @@ -0,0 +1,28 @@
> +#!/bin/sh
> +for testlist in test-version test-reader run-emitter run-dumper run-parser run-scanner run-loader; do
> +       case ${testlist} in
> +       test-version|test-reader)
> +               outputrun=$(./$testlist)
> +              if ! echo ${outputrun}| grep -q "FAILURE"
> +               then
> +                       echo "PASS: ${testlist}"
> +               else
> +                       echo "FAIL : ${testlist}"
> +                       exit 1
> +               fi
> +               ;;
> +       esac
> +       case ${testlist} in
> +       run-emitter|run-dumper|run-parser|run-scanner|run-loader)
> +              outputrun=$(./$testlist example.yaml)
> +              echo $outputrun
> +              if ! echo ${outputrun}| grep "FAILURE ([1-50]"
> +              then
> +                      echo "PASS: ${testlist}"
> +              else
> +                      echo "FAIL : ${testlist}"
> +                      exit 1
> +              fi
> +               ;;
> +       esac
> +done

I'd like to better understand what is happening here. Why are some
items in the test suite run without an argument, and other items with
example.yaml that we created ourselves (apparently, from items in
examples/)? How does the test suite work upstream and why can't we
simply use that?

>  SRC_URI[md5sum] = "bb15429d8fb787e7d3f1c83ae129a999"
>  SRC_URI[sha256sum] = "c642ae9b75fee120b2d96c712538bd2cf283228d2337df2cf2988e3c02678ef4"
> +SRC_URI[run-ptest.md5sum] = "040b47d9b00d2ddbc707792ff6626243"
> +SRC_URI[run-ptest.sha256sum] = "71030494ab116f1c31bf125de9d5fbe2f01aacf4daada36b44202be135a80964"
> +SRC_URI[example.yaml.md5sum] = "d7f8a342650b153d219431633b0c383b"
> +SRC_URI[example.yaml.sha256sum] = "d7c50b29716fcc7445e181c7d7043d70f8e58096793dc1f351c8e7f41034f813"

This is not necessary. Integrity check for files in the layer tree is
ensured through git commit id (of the layer) being a checksum of all
the files.

Alex
Virendra Kumar Thakur Nov. 19, 2024, 12:02 p.m. UTC | #2 
Below test does not require any argument to be pass. Its is a simple binary
> +       test-version and test-reader


 run-emitter|run-dumper|run-parser|run-scanner|run-loader
Other test require an argument and yaml file which it needs to perform task. Like parsing or emmiting or to perform run-loader test

I will send another version of patch without below change:
> +SRC_URI[run-ptest.md5sum] = "040b47d9b00d2ddbc707792ff6626243"
> +SRC_URI[run-ptest.sha256sum] = "71030494ab116f1c31bf125de9d5fbe2f01aacf4daada36b44202be135a80964"
> +SRC_URI[example.yaml.md5sum] = "d7f8a342650b153d219431633b0c383b"
> +SRC_URI[example.yaml.sha256sum] = "d7c50b29716fcc7445e181c7d7043d70f8e58096793dc1f351c8e7f41034f813"


[cid:95654072-e768-4291-9780-247df4bf5f3a]<https://outlook.office.com/bookwithme/user/fcab7d9903984b91a5841c51f49bd5ec@kpit.com?anonymous&ep=bwmEmailSignature>         Book time to meet with me<https://outlook.office.com/bookwithme/user/fcab7d9903984b91a5841c51f49bd5ec@kpit.com?anonymous&ep=bwmEmailSignature>
Alexander Kanavin Nov. 19, 2024, 12:07 p.m. UTC | #3 
On Tue, 19 Nov 2024 at 13:02, Virendra Thakur via lists.openembedded.org
<virendra.thakur=kpit.com@lists.openembedded.org> wrote:

>
>
> Below test does not require any argument to be pass. Its is a simple binary
> > +       test-version and test-reader
>
>
>  run-emitter|run-dumper|run-parser|run-scanner|run-loader
> Other test require an argument and yaml file which it needs to perform
> task. Like parsing or emmiting or to perform run-loader test
>

This does not quite answer this part of my question:

How does the test suite work upstream and why can't we
simply use that?

I still don't understand why a custom yaml needs to be created.

Alex
Virendra Kumar Thakur Nov. 19, 2024, 1:24 p.m. UTC | #4 
in yocto , Libyaml are sourced from https://pyyaml.org/download/libyaml/yaml-0.2.5.tar.gz

when I tried to run the test using above source . It only execute two test binary.

yaml-0.2.5/tests$ make check-TESTS
make[1]: Entering directory 'yaml-0.2.5/tests'
PASS: test-version
PASS: test-reader
============================================================================
Testsuite summary for yaml 0.2.5
============================================================================
# TOTAL: 2
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[1]: Leaving directory  yaml-0.2.5/tests'

So added few more test binary and an example yaml file as test input. So below are the test binary .
 run-emitter
 run-dumper
 run-parser
 run-scanner
 run-loader



[cid:6e137582-c02d-4e71-a295-5ee18e5b2928]<https://outlook.office.com/bookwithme/user/fcab7d9903984b91a5841c51f49bd5ec@kpit.com?anonymous&ep=bwmEmailSignature>         Book time to meet with me<https://outlook.office.com/bookwithme/user/fcab7d9903984b91a5841c51f49bd5ec@kpit.com?anonymous&ep=bwmEmailSignature>
Alexander Kanavin Nov. 19, 2024, 5:03 p.m. UTC | #5 
On Tue, 19 Nov 2024 at 14:24, Virendra Kumar Thakur <
Virendra.Thakur@kpit.com> wrote:

> in yocto , Libyaml are sourced from
> https://pyyaml.org/download/libyaml/yaml-0.2.5.tar.gz
>
> when I tried to run the test using above source . It only execute two test
> binary.
>
> yaml-0.2.5/tests$ make check-TESTS
> make[1]: Entering directory 'yaml-0.2.5/tests'
> PASS: test-version
> PASS: test-reader
>

If you take a look at upstream's top level Makefile.am, it has an
additional target:

=============
test-suite: tests/run-test-suite all
make -C $< test

test-all: test test-suite

tests/run-test-suite:
ifeq ($(LIBYAML_TEST_SUITE_RUN_REPO),$(LIBYAML_TEST_SUITE_RUN_REPO_DEFAULT))
 -git branch --track $(LIBYAML_TEST_SUITE_RUN_BRANCH)
origin/$(LIBYAML_TEST_SUITE_RUN_BRANCH)
 -git worktree prune
 git worktree add $@ $(LIBYAML_TEST_SUITE_RUN_BRANCH)
    else
 git clone --branch $(LIBYAML_TEST_SUITE_RUN_BRANCH)
$(LIBYAML_TEST_SUITE_RUN_REPO) $@
    endif
============

Presumably, running 'make test-all' will fetch the needed test data from
somewhere on the internet, and then use it to run additional tests. Can you
please try that, and see what happens? Then we can probably just use the
data from that repo by putting it into SRC_URI.

Alex
Virendra Kumar Thakur Nov. 22, 2024, 12:38 p.m. UTC | #6 
Hello @alex.kanavin@gmail.com<mailto:alex.kanavin@gmail.com>

I have tried to run test-all but it fails:

make[2]: Entering directory '/home/virendrak/yaml-0.2.5/tests'
PASS: test-version
PASS: test-reader
============================================================================
Testsuite summary for yaml 0.2.5
============================================================================
# TOTAL: 2
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[2]: Leaving directory '/home/virendrak/yaml-0.2.5/tests'
make[1]: Leaving directory '/home/virendrak/yaml-0.2.5/tests'
git branch --track run-test-suite origin/run-test-suite
fatal: not a git repository (or any of the parent directories): .git
make: [Makefile:885: tests/run-test-suite] Error 128 (ignored)
git worktree prune
fatal: not a git repository (or any of the parent directories): .git
make: [Makefile:886: tests/run-test-suite] Error 128 (ignored)
git worktree add tests/run-test-suite run-test-suite
fatal: not a git repository (or any of the parent directories): .git
make: *** [Makefile:887: tests/run-test-suite] Error 128

It fails due to libyaml source is from downloaded from dowload directory mentioned below as a tar , its not git repo.
In yocto libyaml SRC is from : SRC_URI = "https://pyyaml.org/download/libyaml/yaml-${PV}.tar.gz"

Due to this it fail to fetch code from git repo.

Best Regards



[cid:8dc09906-5365-4dd5-b686-c6a3e5ee9ead]<https://outlook.office.com/bookwithme/user/fcab7d9903984b91a5841c51f49bd5ec@kpit.com?anonymous&ep=bwmEmailSignature>         Book time to meet with me<https://outlook.office.com/bookwithme/user/fcab7d9903984b91a5841c51f49bd5ec@kpit.com?anonymous&ep=bwmEmailSignature>
Alexander Kanavin Nov. 22, 2024, 12:43 p.m. UTC | #7 
On Fri, 22 Nov 2024 at 13:38, Virendra Kumar Thakur <
Virendra.Thakur@kpit.com> wrote:

> It fails due to libyaml source is from downloaded from dowload directory
> mentioned below as a tar , its not git repo.
> In yocto libyaml SRC is from : SRC_URI = "
> https://pyyaml.org/download/libyaml/yaml-${PV}.tar.gz"
>
> Due to this it fail to fetch code from git repo.
>

For the sake of experiment, perhaps you can try again with a git checkout
to 0.2.5 tag from here?

https://github.com/yaml/libyaml


Alex
Virendra Kumar Thakur Nov. 22, 2024, 1 p.m. UTC | #8 
Hello alex

With code from github it works fine. make test-all is working.

I am attaching here with the output file.

1..214
ok
All tests successful.
Files=3, Tests=452, 12 wallclock secs ( 0.12 usr  0.03 sys +  3.22 cusr  5.92 csys =  9.29 CPU)
Result: PASS



[cid:dc67cc71-2784-492b-bad5-d29bf68ca91d]<https://outlook.office.com/bookwithme/user/fcab7d9903984b91a5841c51f49bd5ec@kpit.com?anonymous&ep=bwmEmailSignature>         Book time to meet with me<https://outlook.office.com/bookwithme/user/fcab7d9903984b91a5841c51f49bd5ec@kpit.com?anonymous&ep=bwmEmailSignature>
Alexander Kanavin Nov. 22, 2024, 1:07 p.m. UTC | #9 
On Fri, 22 Nov 2024 at 14:00, Virendra Kumar Thakur
<Virendra.Thakur@kpit.com> wrote:
> With code from github it works fine. make test-all is working.
>
> I am attaching here with the output file.
>
> 1..214
> ok
> All tests successful.
> Files=3, Tests=452, 12 wallclock secs ( 0.12 usr  0.03 sys +  3.22 cusr  5.92 csys =  9.29 CPU)
> Result: PASS

Right. Then your next task is to figure out where those tests obtain
the needed data, and then replicate that in oe-core, possibly by
adding needed repositories to SRC_URI, and then copying bits of them
into the ptest package.

The key sequence in the log seems to be this:

git branch --track run-test-suite origin/run-test-suite
Branch 'run-test-suite' set up to track remote branch 'run-test-suite'
from 'origin'.
git worktree prune
git worktree add tests/run-test-suite run-test-suite
Preparing worktree (checking out 'run-test-suite')
HEAD is now at cc9cb7b More bin/pin debug messages
make -C tests/run-test-suite test
make[1]: Entering directory
'/home/virendrak/libyaml-github/libyaml/tests/run-test-suite'
git clone https://github.com/yaml/yaml-test-suite data --branch=data
Cloning into 'data'...
remote: Enumerating objects: 17410, done.
remote: Counting objects: 100% (1134/1134), done.
remote: Compressing objects: 100% (555/555), done.
remote: Total 17410 (delta 675), reused 580 (delta 579), pack-reused
16276 (from 1)
Receiving objects: 100% (17410/17410), 4.57 MiB | 9.07 MiB/s, done.
Resolving deltas: 100% (11215/11215), done.
(cd data && git reset --hard data-2020-02-11)
HEAD is now at a12ee724 Regenerated data from master v2020-02-11
git branch --track run-test-suite-code origin/run-test-suite-code
Branch 'run-test-suite-code' set up to track remote branch
'run-test-suite-code' from 'origin'.
git worktree prune
[[ -d test ]] || \
    git worktree add test run-test-suite-code
Preparing worktree (checking out 'run-test-suite-code')
HEAD is now at 67365dd Remove tests for PR#162
(cd test && git reset --hard 67365dd2af265f17ba6ee0832c1411e4fd80ac0b)
HEAD is now at 67365dd Remove tests for PR#162
prove -lv test



Alex
Virendra Kumar Thakur Nov. 27, 2024, 1:39 p.m. UTC | #10 
Hello Alexander,

Test which I have added is addressing our purpose.
Not working on this topic now. I will work on your suggestion If i will get some time.


[cid:8de35c11-2683-4bee-816d-79b5c95da7c9]<https://outlook.office.com/bookwithme/user/fcab7d9903984b91a5841c51f49bd5ec@kpit.com?anonymous&ep=bwmEmailSignature>         Book time to meet with me<https://outlook.office.com/bookwithme/user/fcab7d9903984b91a5841c51f49bd5ec@kpit.com?anonymous&ep=bwmEmailSignature>
Alexander Kanavin Nov. 27, 2024, 1:51 p.m. UTC | #11 
On Wed, 27 Nov 2024 at 14:39, Virendra Kumar Thakur <
Virendra.Thakur@kpit.com> wrote:

> Test which I have added is addressing our purpose.
> Not working on this topic now. I will work on your suggestion If i will
> get some time.
>

The problem is that the test adds custom test data that isn't possible to
independently recreate (as the only information about it is that it was
'created by using examples'). This makes it very difficult to do later
adjustments to that data when needed (e.g. when upstream changes the tests
that start causing failures with the data).

On the other hand, upstream does provide official test data, and that's
what we should be using.

For this reason I cannot accept the implementation as it is.

Alex
diff mbox series

Patch

diff --git a/meta/conf/distro/include/ptest-packagelists.inc b/meta/conf/distro/include/ptest-packagelists.inc
index 4a48188562..cd9fae3e45 100644
--- a/meta/conf/distro/include/ptest-packagelists.inc
+++ b/meta/conf/distro/include/ptest-packagelists.inc
@@ -48,6 +48,7 @@  PTESTS_FAST = "\
     libxml-simple-perl \
     libxml2 \
     libxmlb \
+    libyaml \
     logrotate \
     lua \
     lzo \
diff --git a/meta/recipes-support/libyaml/libyaml/example.yaml b/meta/recipes-support/libyaml/libyaml/example.yaml
new file mode 100644
index 0000000000..107b2b1204
--- /dev/null
+++ b/meta/recipes-support/libyaml/libyaml/example.yaml
@@ -0,0 +1,45 @@ 
+# anchors yaml
+base: &base
+    name: Everyone has same name
+
+foo: &foo
+    <<: *base
+    age: 10
+
+bar: &bar
+    <<: *base
+    age: 20
+# global tag example
+%TAG ! tag:clarkevans.com,2002:
+--- !shape
+  # Use the ! handle for presenting
+  # tag:clarkevans.com,2002:circle
+- !circle
+  center: &ORIGIN {x: 73, y: 129}
+  radius: 7
+- !line
+  start: *ORIGIN
+  finish: { x: 89, y: 102 }
+- !label
+  start: *ORIGIN
+  color: 0xFFEEBB
+  text: Pretty vector drawing.
+# array example
+- member
+- member2
+# numbers example
+---
+string: "Hello"
+integer: 123
+float: 12.345
+boolean: No
+# strings example
+message: "YAML & JSON"
+# mapping example
+person:
+    name: "ramesh"
+    surname: "sharma"
+    age:  23
+#json example
+---
+{"key": ["value", 3]}
diff --git a/meta/recipes-support/libyaml/libyaml/run-ptest b/meta/recipes-support/libyaml/libyaml/run-ptest
new file mode 100644
index 0000000000..794b24c6ba
--- /dev/null
+++ b/meta/recipes-support/libyaml/libyaml/run-ptest
@@ -0,0 +1,28 @@ 
+#!/bin/sh
+for testlist in test-version test-reader run-emitter run-dumper run-parser run-scanner run-loader; do
+       case ${testlist} in
+       test-version|test-reader)
+               outputrun=$(./$testlist)
+	       if ! echo ${outputrun}| grep -q "FAILURE"
+               then
+                       echo "PASS: ${testlist}"
+               else
+                       echo "FAIL : ${testlist}"
+                       exit 1
+               fi
+               ;;
+       esac
+       case ${testlist} in
+       run-emitter|run-dumper|run-parser|run-scanner|run-loader)
+	       outputrun=$(./$testlist example.yaml)
+	       echo $outputrun
+	       if ! echo ${outputrun}| grep "FAILURE ([1-50]"
+	       then
+		       echo "PASS: ${testlist}"
+	       else
+		       echo "FAIL : ${testlist}"
+		       exit 1
+	       fi
+               ;;
+       esac
+done
diff --git a/meta/recipes-support/libyaml/libyaml_0.2.5.bb b/meta/recipes-support/libyaml/libyaml_0.2.5.bb
index aa7fc5e914..7b108b8548 100644
--- a/meta/recipes-support/libyaml/libyaml_0.2.5.bb
+++ b/meta/recipes-support/libyaml/libyaml_0.2.5.bb
@@ -7,13 +7,34 @@  SECTION = "libs/devel"
 LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://License;md5=7bbd28caa69f81f5cd5f48647236663d"
 
-SRC_URI = "https://pyyaml.org/download/libyaml/yaml-${PV}.tar.gz"
+SRC_URI = "https://pyyaml.org/download/libyaml/yaml-${PV}.tar.gz \
+           file://run-ptest \
+           file://example.yaml \
+           "
 SRC_URI[md5sum] = "bb15429d8fb787e7d3f1c83ae129a999"
 SRC_URI[sha256sum] = "c642ae9b75fee120b2d96c712538bd2cf283228d2337df2cf2988e3c02678ef4"
+SRC_URI[run-ptest.md5sum] = "040b47d9b00d2ddbc707792ff6626243"
+SRC_URI[run-ptest.sha256sum] = "71030494ab116f1c31bf125de9d5fbe2f01aacf4daada36b44202be135a80964"
+SRC_URI[example.yaml.md5sum] = "d7f8a342650b153d219431633b0c383b"
+SRC_URI[example.yaml.sha256sum] = "d7c50b29716fcc7445e181c7d7043d70f8e58096793dc1f351c8e7f41034f813"
 
 S = "${WORKDIR}/yaml-${PV}"
 
-inherit autotools
+inherit autotools ptest
+
+do_compile_ptest() {
+        sed -i "/\$(MAKE) \$(AM_MAKEFLAGS) check-TESTS/d" tests/Makefile
+        oe_runmake check
+}
+
+do_install_ptest() {
+    # install the tests dir
+    cp -rf ${B}/tests/.libs/* ${D}${PTEST_PATH}
+    install -m 755 ${UNPACKDIR}/example.yaml ${D}${PTEST_PATH}
+
+}
+
+RDEPENDS:${PN}-ptest += "make bash"
 
 DISABLE_STATIC:class-nativesdk = ""
 DISABLE_STATIC:class-native = ""