| Message ID | 20240503144340.27385-1-ninette@thehoodiefirm.com |
|---|---|
| Headers | show |
| Series | Improvements for performance test report view | expand |
On 2024-05-03 10:43 a.m., Ninette Adhikari wrote: > This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/). > The current report can be accessed here: > Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html) > The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository. > The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes. > The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart. > > The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements: > > - Add [Apache echart](https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive. > - Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the charts. > - Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300. > - Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions. The tooltips are very nice and useful so > - The charts are created as step charts instead of plain line charts. > - Add start time, size, and commit number to tooltip. > - Add dark mode view > > Updated report screenshots: > https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53 > https://github.com/neighbourhoodie/poky/assets/13760198/b40c326b-17d2-42e2-8e41-72e52ed2c003 > https://github.com/neighbourhoodie/poky/assets/13760198/cc7ec996-9dab-435a-8fdc-82a2a4193c0a > https://github.com/neighbourhoodie/poky/assets/13760198/6e0fe09d-50e5-4b0b-b70b-6943f71b5208 Very nice! Thanks for the work and the v2 improvements. I applied the patches by saving them from email and there was a minor whitespace warning which is really nothing to worry about but if you want to avoid such noise, you could send yourself patches by email and then apply them in a different branch if you want to check for such problems. ❯ git am /tmp/ninette-v2/* Applying: oe-build-perf-report: Add apache echarts to make report interactive Applying: oe-build-perf-report: Display more than 300 commits and date instead of commit number Applying: oe-build-perf-report: Improve report styling and add descriptions .git/rebase-apply/patch:240: trailing whitespace. start_time = time warning: 1 line adds whitespace errors. Applying: oe-build-perf-report: Update chart tooltip and chart type Applying: oe-build-perf-report: Add dark mode poky.git on ninette-v2 [$?] ❯ ls /tmp/ninette-v2/* '/tmp/ninette-v2/[OE-core] [PATCH v2 1_5] oe-build-perf-report: Add apache echarts to make report interactive - "Ninette Adhikari via lists.openembedded.org" <ninette=thehoodiefirm.com@lists.openembedded.org> - 2024-05-03 1043.eml' '/tmp/ninette-v2/[PATCH v2 2_5] oe-build-perf-report: Display more than 300 commits and date instead of commit number - Ninette Adhikari <ninette@thehoodiefirm.com> - 2024-05-03 1043.eml' '/tmp/ninette-v2/[PATCH v2 3_5] oe-build-perf-report: Improve report styling and add descriptions - Ninette Adhikari <ninette@thehoodiefirm.com> - 2024-05-03 1043.eml' '/tmp/ninette-v2/[PATCH v2 4_5] oe-build-perf-report: Update chart tooltip and chart type - Ninette Adhikari <ninette@thehoodiefirm.com> - 2024-05-03 1043.eml' '/tmp/ninette-v2/[PATCH v2 5_5] oe-build-perf-report: Add dark mode - Ninette Adhikari <ninette@thehoodiefirm.com> - 2024-05-03 1043.eml' > > For local setup, you can do the following: > > 1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/) > > 2. In the poky repository run the following to build the report HTML: > ```bash > ./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html These are the same setup step as last time I think and again, they aren't quite right but the script tells you do do: ❯ git fetch origin refs/notes/buildstats/perf-debian11/master/qemux86:refs/notes/buildstats/perf-debian11/master/qemux86 and that works. > ``` > Note: > - Add your local path to the yocto-buildstats repo > - The above command builds the report in a file called `test.html`. You can access it in the root directory in poky. > - This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch. Hmmm, ah, that's a poky commit id. I (stupidly!) expected it to be a commit ID in the yocto-buildstats repo. One thing that you can think about fixing or just ignore is that the --commit arg needs to be the full commit hash not the shortened version. i.e. this doesn't work: ❯ ./scripts/oe-build-perf-report -r "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git" --branch "master" --commit "663f180574" --branch2 "master" --html > /tmp/ninette-v2-test-663f180574.html Traceback (most recent call last): File "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report", line 617, in <module> sys.exit(main()) File "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report", line 532, in main index1 = gitarchive.rev_find(revs, 'commit', args.commit) File "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/meta/lib/oeqa/utils/gitarchive.py", line 282, in rev_find raise ValueError("Unable to find '{}' value '{}'".format(attr, val)) ValueError: Unable to find 'commit' value '663f180574' Since the script is going to be used in other scripts run on the autobuilder, this is a low priority enhancement. I was happy to see that a newer poky commit, worked fine: ❯ ./scripts/oe-build-perf-report -r "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git" --branch "master" --commit "632e3170595bb32717c0471a55a619b4b33fe787" --branch2 "master" --html > /tmp/ninette-v2-test-632e3170595bb32717c0471a55a619b4b33fe787.html INFO: Parsing buildstats from 'refs/notes/buildstats/perf-debian11/master/qemux86' but I did notice: Stdev: nan whereas in your linked charts, and in my first run with commit: 663f1805742ff6fb6955719d0ab7846a425debcf the correct numerical standard deviation is displayed. I can debug this is you can't reproduce it yourself. Again. these reports and beautiful interactive charts are great to see. Thanks! ../Randy > > > Ninette Adhikari (5): > oe-build-perf-report: Add apache echarts to make report interactive > oe-build-perf-report: Display more than 300 commits and date instead > of commit number > oe-build-perf-report: Improve report styling and add descriptions > oe-build-perf-report: Update chart tooltip and chart type > oe-build-perf-report: Add dark mode > > .../build_perf/html/measurement_chart.html | 140 ++++++++++++------ > scripts/lib/build_perf/html/report.html | 124 +++++++++++----- > scripts/lib/build_perf/report.py | 5 +- > scripts/oe-build-perf-report | 6 +- > 4 files changed, 193 insertions(+), 82 deletions(-) >
On 2024-05-03 1:22 p.m., Randy MacLeod via lists.openembedded.org wrote: > On 2024-05-03 10:43 a.m., Ninette Adhikari wrote: >> This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/). >> The current report can be accessed here: >> Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html) >> The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository. >> The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes. >> The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart. >> >> The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements: >> >> - Add [Apache echart](https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive. >> - Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the charts. >> - Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300. >> - Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions. > The tooltips are very nice and useful so >> - The charts are created as step charts instead of plain line charts. >> - Add start time, size, and commit number to tooltip. >> - Add dark mode view >> >> Updated report screenshots: >> https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53 >> https://github.com/neighbourhoodie/poky/assets/13760198/b40c326b-17d2-42e2-8e41-72e52ed2c003 >> https://github.com/neighbourhoodie/poky/assets/13760198/cc7ec996-9dab-435a-8fdc-82a2a4193c0a >> https://github.com/neighbourhoodie/poky/assets/13760198/6e0fe09d-50e5-4b0b-b70b-6943f71b5208 > > > Very nice! Thanks for the work and the v2 improvements. > > > I applied the patches by saving them from email and there was a minor > whitespace warning which > is really nothing to worry about but if you want to avoid such noise, > you could send yourself > patches by email and then apply them in a different branch if you want > to check for such problems. > > > ❯ git am /tmp/ninette-v2/* > Applying: oe-build-perf-report: Add apache echarts to make report > interactive > Applying: oe-build-perf-report: Display more than 300 commits and date > instead of commit number > Applying: oe-build-perf-report: Improve report styling and add > descriptions > .git/rebase-apply/patch:240: trailing whitespace. > start_time = time > warning: 1 line adds whitespace errors. > Applying: oe-build-perf-report: Update chart tooltip and chart type > Applying: oe-build-perf-report: Add dark mode > > poky.git on ninette-v2 [$?] > ❯ ls /tmp/ninette-v2/* > '/tmp/ninette-v2/[OE-core] [PATCH v2 1_5] oe-build-perf-report: Add > apache echarts to make report interactive - "Ninette Adhikari via > lists.openembedded.org" > <ninette=thehoodiefirm.com@lists.openembedded.org> - 2024-05-03 1043.eml' > '/tmp/ninette-v2/[PATCH v2 2_5] oe-build-perf-report: Display more > than 300 commits and date instead of commit number - Ninette Adhikari > <ninette@thehoodiefirm.com> - 2024-05-03 1043.eml' > '/tmp/ninette-v2/[PATCH v2 3_5] oe-build-perf-report: Improve report > styling and add descriptions - Ninette Adhikari > <ninette@thehoodiefirm.com> - 2024-05-03 1043.eml' > '/tmp/ninette-v2/[PATCH v2 4_5] oe-build-perf-report: Update chart > tooltip and chart type - Ninette Adhikari <ninette@thehoodiefirm.com> > - 2024-05-03 1043.eml' > '/tmp/ninette-v2/[PATCH v2 5_5] oe-build-perf-report: Add dark mode - > Ninette Adhikari <ninette@thehoodiefirm.com> - 2024-05-03 1043.eml' > >> For local setup, you can do the following: >> >> 1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/) >> >> 2. In the poky repository run the following to build the report HTML: >> ```bash >> ./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html > > These are the same setup step as last time I think and again, they > aren't quite right but > the script tells you do do: > > ❯ git fetch origin > refs/notes/buildstats/perf-debian11/master/qemux86:refs/notes/buildstats/perf-debian11/master/qemux86 > > > and that works. > > >> ``` >> Note: >> - Add your local path to the yocto-buildstats repo >> - The above command builds the report in a file called `test.html`. You can access it in the root directory in poky. >> - This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch. > > Hmmm, ah, that's a poky commit id. I (stupidly!) expected it to be a > commit ID in the yocto-buildstats repo. > > > One thing that you can think about fixing or just ignore is that the > --commit arg needs to be the > full commit hash not the shortened version. i.e. this doesn't work: > > ❯ ./scripts/oe-build-perf-report -r > "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git" > --branch "master" --commit "663f180574" --branch2 "master" --html > > /tmp/ninette-v2-test-663f180574.html > Traceback (most recent call last): > File > "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report", > line 617, in <module> > sys.exit(main()) > File > "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report", > line 532, in main > index1 = gitarchive.rev_find(revs, 'commit', args.commit) > File > "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/meta/lib/oeqa/utils/gitarchive.py", > line 282, in rev_find > raise ValueError("Unable to find '{}' value '{}'".format(attr, val)) > ValueError: Unable to find 'commit' value '663f180574' > > Since the script is going to be used in other scripts run on the > autobuilder, this is a low priority enhancement. > > > I was happy to see that a newer poky commit, worked fine: > > ❯ ./scripts/oe-build-perf-report -r > "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git" > --branch "master" --commit "632e3170595bb32717c0471a55a619b4b33fe787" > --branch2 "master" --html > > /tmp/ninette-v2-test-632e3170595bb32717c0471a55a619b4b33fe787.html > INFO: Parsing buildstats from > 'refs/notes/buildstats/perf-debian11/master/qemux86' > > > but I did notice: Stdev: nan > > > whereas in your linked charts, and in my first run with commit: > 663f1805742ff6fb6955719d0ab7846a425debcf > > the correct numerical standard deviation is displayed. > > I can debug this is you can't reproduce it yourself. > * if you can't ... Also, I wanted to draw your attention to how the vertical dashed line (VDL) behaves to see if it's something that can (or should) be changed. If you zoom in on a graph the VDL is placed at the left edge of bar until the pointer passes the mid-point of the bar: at which time the VDL moves to the next bar/data point: It's a small thing but it would be nice if: 1) the VDL was placed in the centre of the bar and 2) the commit number /tooltip data only changed when the pointer move to a different bar so the same data would be displayed as one swept across a given bar. What do you think? Picky, eh! ;-) That's all for now. ../Randy > > Again. these reports and beautiful interactive charts are great to see. > > Thanks! > > ../Randy > > >> Ninette Adhikari (5): >> oe-build-perf-report: Add apache echarts to make report interactive >> oe-build-perf-report: Display more than 300 commits and date instead >> of commit number >> oe-build-perf-report: Improve report styling and add descriptions >> oe-build-perf-report: Update chart tooltip and chart type >> oe-build-perf-report: Add dark mode >> >> .../build_perf/html/measurement_chart.html | 140 ++++++++++++------ >> scripts/lib/build_perf/html/report.html | 124 +++++++++++----- >> scripts/lib/build_perf/report.py | 5 +- >> scripts/oe-build-perf-report | 6 +- >> 4 files changed, 193 insertions(+), 82 deletions(-) >> > > -- > # Randy MacLeod > # Wind River Linux > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#199007):https://lists.openembedded.org/g/openembedded-core/message/199007 > Mute This Topic:https://lists.openembedded.org/mt/105889450/3616765 > Group Owner:openembedded-core+owner@lists.openembedded.org > Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [randy.macleod@windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- >
Hi Randy, Thank you so much for the detailed feedback. This is very helpful:) Here are my comments: - whitespace warning: Thanks for the catch. I'll make sure to check the patches next time to make sure there are no warnings. - local setup: You are right this can be updated. Do I need to update this in the cover letter and resend it? Let me know, then I can also include the update on --commit argument. - stdev as nan: I was able to reproduce the issue and I think this is where the nan error is being generated. https://git.yoctoproject.org/poky/tree/scripts/lib/build_perf/report.py#n331. This error persists in the current master branch. Unfortunately I'm not exactly sure on the logic of how this is generated and was not able to resolve this issue. If you have any suggestions on how to proceed do let me know. - vertical dashed line (VDL) behavior: I tried to troubleshoot this but it seems like Apache echarts tooltip <https://echarts.apache.org/en/option.html#tooltip> is only triggered when there is a data point in the x axis. And in your example the data points are on the edges of the bar. Some of the charts do have continuous data points and hence seems smoother. Hope this is not too much of a holdup. - [image: tooltip.gif] Let me know if you have any other questions. Thanks again! Ninette On Fri, May 3, 2024 at 8:10 PM Randy MacLeod <randy.macleod@windriver.com> wrote: > On 2024-05-03 1:22 p.m., Randy MacLeod via lists.openembedded.org wrote: > > On 2024-05-03 10:43 a.m., Ninette Adhikari wrote: > > This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/). > The current report can be accessed here: > Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html) > The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository. > The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes. > The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart. > > The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements: > > - Add [Apache echart](https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive. > - Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the charts. > - Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300. > - Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions. > > The tooltips are very nice and useful so > > - The charts are created as step charts instead of plain line charts. > - Add start time, size, and commit number to tooltip. > - Add dark mode view > > Updated report screenshots: https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53https://github.com/neighbourhoodie/poky/assets/13760198/b40c326b-17d2-42e2-8e41-72e52ed2c003https://github.com/neighbourhoodie/poky/assets/13760198/cc7ec996-9dab-435a-8fdc-82a2a4193c0ahttps://github.com/neighbourhoodie/poky/assets/13760198/6e0fe09d-50e5-4b0b-b70b-6943f71b5208 > > > Very nice! Thanks for the work and the v2 improvements. > > > I applied the patches by saving them from email and there was a minor > whitespace warning which > is really nothing to worry about but if you want to avoid such noise, you > could send yourself > patches by email and then apply them in a different branch if you want to > check for such problems. > > > ❯ git am /tmp/ninette-v2/* > Applying: oe-build-perf-report: Add apache echarts to make report > interactive > Applying: oe-build-perf-report: Display more than 300 commits and date > instead of commit number > Applying: oe-build-perf-report: Improve report styling and add descriptions > .git/rebase-apply/patch:240: trailing whitespace. > start_time = time > warning: 1 line adds whitespace errors. > Applying: oe-build-perf-report: Update chart tooltip and chart type > Applying: oe-build-perf-report: Add dark mode > > poky.git on ninette-v2 [$?] > ❯ ls /tmp/ninette-v2/* > '/tmp/ninette-v2/[OE-core] [PATCH v2 1_5] oe-build-perf-report: Add apache > echarts to make report interactive - "Ninette Adhikari via > lists.openembedded.org" <ninette=thehoodiefirm.com@lists.openembedded.org> > <ninette=thehoodiefirm.com@lists.openembedded.org> - 2024-05-03 1043.eml' > '/tmp/ninette-v2/[PATCH v2 2_5] oe-build-perf-report: Display more than > 300 commits and date instead of commit number - Ninette Adhikari > <ninette@thehoodiefirm.com> <ninette@thehoodiefirm.com> - 2024-05-03 > 1043.eml' > '/tmp/ninette-v2/[PATCH v2 3_5] oe-build-perf-report: Improve report > styling and add descriptions - Ninette Adhikari > <ninette@thehoodiefirm.com> <ninette@thehoodiefirm.com> - 2024-05-03 > 1043.eml' > '/tmp/ninette-v2/[PATCH v2 4_5] oe-build-perf-report: Update chart tooltip > and chart type - Ninette Adhikari <ninette@thehoodiefirm.com> > <ninette@thehoodiefirm.com> - 2024-05-03 1043.eml' > '/tmp/ninette-v2/[PATCH v2 5_5] oe-build-perf-report: Add dark mode - > Ninette Adhikari <ninette@thehoodiefirm.com> <ninette@thehoodiefirm.com> > - 2024-05-03 1043.eml' > > For local setup, you can do the following: > > 1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/) > > 2. In the poky repository run the following to build the report HTML: > ```bash > ./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html > > These are the same setup step as last time I think and again, they aren't > quite right but > the script tells you do do: > > ❯ git fetch origin > refs/notes/buildstats/perf-debian11/master/qemux86:refs/notes/buildstats/perf-debian11/master/qemux86 > > > and that works. > > > ``` > Note: > - Add your local path to the yocto-buildstats repo > - The above command builds the report in a file called `test.html`. You can access it in the root directory in poky. > - This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch. > > Hmmm, ah, that's a poky commit id. I (stupidly!) expected it to be a > commit ID in the yocto-buildstats repo. > > > One thing that you can think about fixing or just ignore is that the > --commit arg needs to be the > full commit hash not the shortened version. i.e. this doesn't work: > > ❯ ./scripts/oe-build-perf-report -r > "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git" > --branch "master" --commit "663f180574" --branch2 "master" --html > > /tmp/ninette-v2-test-663f180574.html > Traceback (most recent call last): > File > "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report", > line 617, in <module> > sys.exit(main()) > File > "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report", > line 532, in main > index1 = gitarchive.rev_find(revs, 'commit', args.commit) > File > "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/meta/lib/oeqa/utils/gitarchive.py", > line 282, in rev_find > raise ValueError("Unable to find '{}' value '{}'".format(attr, val)) > ValueError: Unable to find 'commit' value '663f180574' > > Since the script is going to be used in other scripts run on the > autobuilder, this is a low priority enhancement. > > > I was happy to see that a newer poky commit, worked fine: > > ❯ ./scripts/oe-build-perf-report -r > "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git" > --branch "master" --commit "632e3170595bb32717c0471a55a619b4b33fe787" > --branch2 "master" --html > > /tmp/ninette-v2-test-632e3170595bb32717c0471a55a619b4b33fe787.html > INFO: Parsing buildstats from > 'refs/notes/buildstats/perf-debian11/master/qemux86' > > > but I did notice: Stdev: nan > > > whereas in your linked charts, and in my first run with commit: > 663f1805742ff6fb6955719d0ab7846a425debcf > > the correct numerical standard deviation is displayed. > > I can debug this is you can't reproduce it yourself. > > * if you can't ... > > > Also, I wanted to draw your attention to how the vertical dashed line > (VDL) behaves > to see if it's something that can (or should) be changed. > > If you zoom in on a graph the VDL is placed at the left edge of bar until > the pointer passes the mid-point of the bar: > > > at which time the VDL moves to the next bar/data point: > > > > It's a small thing but it would be nice if: > 1) the VDL was placed in the centre of the bar and > 2) the commit number /tooltip data only changed when the pointer move to > a different bar > so the same data would be displayed as one swept across a given bar. > > What do you think? > > > Picky, eh! ;-) > > > That's all for now. > > ../Randy > > > > Again. these reports and beautiful interactive charts are great to see. > > Thanks! > > ../Randy > > > Ninette Adhikari (5): > oe-build-perf-report: Add apache echarts to make report interactive > oe-build-perf-report: Display more than 300 commits and date instead > of commit number > oe-build-perf-report: Improve report styling and add descriptions > oe-build-perf-report: Update chart tooltip and chart type > oe-build-perf-report: Add dark mode > > .../build_perf/html/measurement_chart.html | 140 ++++++++++++------ > scripts/lib/build_perf/html/report.html | 124 +++++++++++----- > scripts/lib/build_perf/report.py | 5 +- > scripts/oe-build-perf-report | 6 +- > 4 files changed, 193 insertions(+), 82 deletions(-) > > > > -- > # Randy MacLeod > # Wind River Linux > > > > > > > -- > # Randy MacLeod > # Wind River Linux > >
On 2024-05-07 10:17 a.m., Ninette Adhikari wrote: > Hi Randy, > > Thank you so much for the detailed feedback. This is very helpful:) > Here are my comments: > > * whitespace warning: Thanks for the catch. I'll make sure to check > the patches next time to make sure there are no warnings. > * local setup: You are right this can be updated. Do I need to > update this in the cover letter and resend it? Let me know, then I > can also include the update on --commit argument. > Since this is only in the cover letter and not in a commit and, since I was able to make use of the work and document how I did that on the email thread, so I won't insist on a v3. > > * stdev as nan: I was able to reproduce the issue and I think this > is where the nan error is being generated. > https://git.yoctoproject.org/poky/tree/scripts/lib/build_perf/report.py#n331 > <https://urldefense.com/v3/__https://git.yoctoproject.org/poky/tree/scripts/lib/build_perf/report.py*n331__;Iw!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4fhQcEM2g$>. > This error persists in the current master branch. Unfortunately > I'm not exactly sure on the logic of how this is generated and was > not able to resolve this issue. If you have any suggestions on how > to proceed do let me know. > I'm not sure off-hand. Would you open a defect in bugzilla.yoctoproject.org to record the problem and what you know about it? > > * vertical dashed line (VDL) behavior: I tried to troubleshoot this > but it seems like Apache echarts tooltip > <https://urldefense.com/v3/__https://echarts.apache.org/en/option.html*tooltip__;Iw!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4d-aXgexg$> > is only triggered when there is a data point in the x axis. And in > your example the data points are on the edges of the bar. Some of > the charts do have continuous data points and hence seems > smoother. Hope this is not too much of a holdup. > Well, now that I've seen it, I can't unsee it and my OCD is triggered! ;-) I don't think it's a big problem for anyone else so let's not worry about it unless you'd like to report it in the YP bugzilla as an enhancement. Thanks again, ../Randy > * tooltip.gif > > Let me know if you have any other questions. Thanks again! > > Ninette > > On Fri, May 3, 2024 at 8:10 PM Randy MacLeod > <randy.macleod@windriver.com> wrote: > > On 2024-05-03 1:22 p.m., Randy MacLeod via lists.openembedded.org > <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4cx6ub3iw$> > wrote: >> On 2024-05-03 10:43 a.m., Ninette Adhikari wrote: >>> This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/). >>> The current report can be accessed here: >>> Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html) >>> The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository. >>> The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes. >>> The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart. >>> >>> The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements: >>> >>> - Add [Apache echart](https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive. >>> - Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the charts. >>> - Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300. >>> - Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions. >> The tooltips are very nice and useful so >>> - The charts are created as step charts instead of plain line charts. >>> - Add start time, size, and commit number to tooltip. >>> - Add dark mode view >>> >>> Updated report screenshots: >>> https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53 >>> https://github.com/neighbourhoodie/poky/assets/13760198/b40c326b-17d2-42e2-8e41-72e52ed2c003 >>> https://github.com/neighbourhoodie/poky/assets/13760198/cc7ec996-9dab-435a-8fdc-82a2a4193c0a >>> https://github.com/neighbourhoodie/poky/assets/13760198/6e0fe09d-50e5-4b0b-b70b-6943f71b5208 >> >> >> Very nice! Thanks for the work and the v2 improvements. >> >> >> I applied the patches by saving them from email and there was a >> minor whitespace warning which >> is really nothing to worry about but if you want to avoid such >> noise, you could send yourself >> patches by email and then apply them in a different branch if you >> want to check for such problems. >> >> >> ❯ git am /tmp/ninette-v2/* >> Applying: oe-build-perf-report: Add apache echarts to make report >> interactive >> Applying: oe-build-perf-report: Display more than 300 commits and >> date instead of commit number >> Applying: oe-build-perf-report: Improve report styling and add >> descriptions >> .git/rebase-apply/patch:240: trailing whitespace. >> start_time = time >> warning: 1 line adds whitespace errors. >> Applying: oe-build-perf-report: Update chart tooltip and chart type >> Applying: oe-build-perf-report: Add dark mode >> >> poky.git on ninette-v2 [$?] >> ❯ ls /tmp/ninette-v2/* >> '/tmp/ninette-v2/[OE-core] [PATCH v2 1_5] oe-build-perf-report: >> Add apache echarts to make report interactive - "Ninette Adhikari >> via lists.openembedded.org >> <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4cx6ub3iw$>" >> <ninette=thehoodiefirm.com@lists.openembedded.org> >> <mailto:ninette=thehoodiefirm.com@lists.openembedded.org> - >> 2024-05-03 1043.eml' >> '/tmp/ninette-v2/[PATCH v2 2_5] oe-build-perf-report: Display >> more than 300 commits and date instead of commit number - Ninette >> Adhikari <ninette@thehoodiefirm.com> >> <mailto:ninette@thehoodiefirm.com> - 2024-05-03 1043.eml' >> '/tmp/ninette-v2/[PATCH v2 3_5] oe-build-perf-report: Improve >> report styling and add descriptions - Ninette Adhikari >> <ninette@thehoodiefirm.com> <mailto:ninette@thehoodiefirm.com> - >> 2024-05-03 1043.eml' >> '/tmp/ninette-v2/[PATCH v2 4_5] oe-build-perf-report: Update >> chart tooltip and chart type - Ninette Adhikari >> <ninette@thehoodiefirm.com> <mailto:ninette@thehoodiefirm.com> - >> 2024-05-03 1043.eml' >> '/tmp/ninette-v2/[PATCH v2 5_5] oe-build-perf-report: Add dark >> mode - Ninette Adhikari <ninette@thehoodiefirm.com> >> <mailto:ninette@thehoodiefirm.com> - 2024-05-03 1043.eml' >> >>> For local setup, you can do the following: >>> >>> 1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/) >>> >>> 2. In the poky repository run the following to build the report HTML: >>> ```bash >>> ./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html >> >> These are the same setup step as last time I think and again, >> they aren't quite right but >> the script tells you do do: >> >> ❯ git fetch origin >> refs/notes/buildstats/perf-debian11/master/qemux86:refs/notes/buildstats/perf-debian11/master/qemux86 >> >> >> and that works. >> >> >>> ``` >>> Note: >>> - Add your local path to the yocto-buildstats repo >>> - The above command builds the report in a file called `test.html`. You can access it in the root directory in poky. >>> - This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch. >> >> Hmmm, ah, that's a poky commit id. I (stupidly!) expected it to >> be a commit ID in the yocto-buildstats repo. >> >> >> One thing that you can think about fixing or just ignore is that >> the --commit arg needs to be the >> full commit hash not the shortened version. i.e. this doesn't work: >> >> ❯ ./scripts/oe-build-perf-report -r >> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git" >> --branch "master" --commit "663f180574" --branch2 "master" --html >> > /tmp/ninette-v2-test-663f180574.html >> Traceback (most recent call last): >> File >> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report", >> line 617, in <module> >> sys.exit(main()) >> File >> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report", >> line 532, in main >> index1 = gitarchive.rev_find(revs, 'commit', args.commit) >> File >> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/meta/lib/oeqa/utils/gitarchive.py", >> line 282, in rev_find >> raise ValueError("Unable to find '{}' value >> '{}'".format(attr, val)) >> ValueError: Unable to find 'commit' value '663f180574' >> >> Since the script is going to be used in other scripts run on the >> autobuilder, this is a low priority enhancement. >> >> >> I was happy to see that a newer poky commit, worked fine: >> >> ❯ ./scripts/oe-build-perf-report -r >> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git" >> --branch "master" --commit >> "632e3170595bb32717c0471a55a619b4b33fe787" --branch2 "master" >> --html > >> /tmp/ninette-v2-test-632e3170595bb32717c0471a55a619b4b33fe787.html >> INFO: Parsing buildstats from >> 'refs/notes/buildstats/perf-debian11/master/qemux86' >> >> >> but I did notice: Stdev: nan >> >> >> whereas in your linked charts, and in my first run with commit: >> 663f1805742ff6fb6955719d0ab7846a425debcf >> >> the correct numerical standard deviation is displayed. >> >> I can debug this is you can't reproduce it yourself. >> > * if you can't ... > > > Also, I wanted to draw your attention to how the vertical dashed > line (VDL) behaves > to see if it's something that can (or should) be changed. > > If you zoom in on a graph the VDL is placed at the left edge of > bar until the pointer passes the mid-point of the bar: > > > at which time the VDL moves to the next bar/data point: > > > > It's a small thing but it would be nice if: > 1) the VDL was placed in the centre of the bar and > 2) the commit number /tooltip data only changed when the pointer > move to a different bar > so the same data would be displayed as one swept across a > given bar. > > What do you think? > > > Picky, eh! ;-) > > > That's all for now. > > ../Randy > > >> >> Again. these reports and beautiful interactive charts are great >> to see. >> >> Thanks! >> >> ../Randy >> >> >>> Ninette Adhikari (5): >>> oe-build-perf-report: Add apache echarts to make report interactive >>> oe-build-perf-report: Display more than 300 commits and date instead >>> of commit number >>> oe-build-perf-report: Improve report styling and add descriptions >>> oe-build-perf-report: Update chart tooltip and chart type >>> oe-build-perf-report: Add dark mode >>> >>> .../build_perf/html/measurement_chart.html | 140 ++++++++++++------ >>> scripts/lib/build_perf/html/report.html | 124 +++++++++++----- >>> scripts/lib/build_perf/report.py | 5 +- >>> scripts/oe-build-perf-report | 6 +- >>> 4 files changed, 193 insertions(+), 82 deletions(-) >>> >> >> -- >> # Randy MacLeod >> # Wind River Linux >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#199007):https://lists.openembedded.org/g/openembedded-core/message/199007 >> Mute This Topic:https://lists.openembedded.org/mt/105889450/3616765 >> Group Owner:openembedded-core+owner@lists.openembedded.org >> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [randy.macleod@windriver.com] >> -=-=-=-=-=-=-=-=-=-=-=- >> > > -- > # Randy MacLeod > # Wind River Linux > > > > -- > Ninette Adhikari > Software developer > The Neighbourhoodie Software GmbH > Harzer Straße 39, 12059 Berlin > neighbourhood.ie > <https://urldefense.com/v3/__http://neighbourhood.ie__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4ey5Kw-Rg$> > > Handelsregister HRB 157851 B Amtsgericht Charlottenburg > Geschäftsführung: Jan Lehnardt, Simone Haas
Thanks for your response Randy:) For the Stdev error in the chart I have opened a ticket here: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15482 Hope this helps! Thanks, Ninette On Wed, May 8, 2024 at 5:54 PM Randy MacLeod <randy.macleod@windriver.com> wrote: > On 2024-05-07 10:17 a.m., Ninette Adhikari wrote: > > Hi Randy, > > Thank you so much for the detailed feedback. This is very helpful:) Here > are my comments: > > - whitespace warning: Thanks for the catch. I'll make sure to check > the patches next time to make sure there are no warnings. > - local setup: You are right this can be updated. Do I need to update > this in the cover letter and resend it? Let me know, then I can also > include the update on --commit argument. > > Since this is only in the cover letter and not in a commit and, since I > was able to make use of the work and document how I did that > on the email thread, so I won't insist on a v3. > > > - stdev as nan: I was able to reproduce the issue and I think this is > where the nan error is being generated. > https://git.yoctoproject.org/poky/tree/scripts/lib/build_perf/report.py#n331 > <https://urldefense.com/v3/__https://git.yoctoproject.org/poky/tree/scripts/lib/build_perf/report.py*n331__;Iw!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4fhQcEM2g$>. > This error persists in the current master branch. Unfortunately I'm > not exactly sure on the logic of how this is generated and was not able to > resolve this issue. If you have any suggestions on how to proceed do let me > know. > > I'm not sure off-hand. Would you open a defect in > bugzilla.yoctoproject.org to record the problem and what you know about > it? > > > - vertical dashed line (VDL) behavior: I tried to troubleshoot this > but it seems like Apache echarts tooltip > <https://urldefense.com/v3/__https://echarts.apache.org/en/option.html*tooltip__;Iw!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4d-aXgexg$> > is only triggered when there is a data point in the x axis. And in your > example the data points are on the edges of the bar. Some of the charts do > have continuous data points and hence seems smoother. Hope this is not too > much of a holdup. > > Well, now that I've seen it, I can't unsee it and my OCD is triggered! ;-) > > I don't think it's a big problem for anyone else so let's not worry about > it unless you'd like to report it in the YP bugzilla as an enhancement. > > Thanks again, > > ../Randy > > > > - [image: tooltip.gif] > > > Let me know if you have any other questions. Thanks again! > > Ninette > > On Fri, May 3, 2024 at 8:10 PM Randy MacLeod <randy.macleod@windriver.com> > wrote: > >> On 2024-05-03 1:22 p.m., Randy MacLeod via lists.openembedded.org >> <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4cx6ub3iw$> >> wrote: >> >> On 2024-05-03 10:43 a.m., Ninette Adhikari wrote: >> >> This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/). >> The current report can be accessed here: >> Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html) >> The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository. >> The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes. >> The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart. >> >> The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements: >> >> - Add [Apache echart](https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive. >> - Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the charts. >> - Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300. >> - Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions. >> >> The tooltips are very nice and useful so >> >> - The charts are created as step charts instead of plain line charts. >> - Add start time, size, and commit number to tooltip. >> - Add dark mode view >> >> Updated report screenshots: https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53https://github.com/neighbourhoodie/poky/assets/13760198/b40c326b-17d2-42e2-8e41-72e52ed2c003https://github.com/neighbourhoodie/poky/assets/13760198/cc7ec996-9dab-435a-8fdc-82a2a4193c0ahttps://github.com/neighbourhoodie/poky/assets/13760198/6e0fe09d-50e5-4b0b-b70b-6943f71b5208 >> >> >> Very nice! Thanks for the work and the v2 improvements. >> >> >> I applied the patches by saving them from email and there was a minor >> whitespace warning which >> is really nothing to worry about but if you want to avoid such noise, you >> could send yourself >> patches by email and then apply them in a different branch if you want to >> check for such problems. >> >> >> ❯ git am /tmp/ninette-v2/* >> Applying: oe-build-perf-report: Add apache echarts to make report >> interactive >> Applying: oe-build-perf-report: Display more than 300 commits and date >> instead of commit number >> Applying: oe-build-perf-report: Improve report styling and add >> descriptions >> .git/rebase-apply/patch:240: trailing whitespace. >> start_time = time >> warning: 1 line adds whitespace errors. >> Applying: oe-build-perf-report: Update chart tooltip and chart type >> Applying: oe-build-perf-report: Add dark mode >> >> poky.git on ninette-v2 [$?] >> ❯ ls /tmp/ninette-v2/* >> '/tmp/ninette-v2/[OE-core] [PATCH v2 1_5] oe-build-perf-report: Add >> apache echarts to make report interactive - "Ninette Adhikari via >> lists.openembedded.org >> <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4cx6ub3iw$>" >> <ninette=thehoodiefirm.com@lists.openembedded.org> >> <ninette=thehoodiefirm.com@lists.openembedded.org> - 2024-05-03 1043.eml' >> '/tmp/ninette-v2/[PATCH v2 2_5] oe-build-perf-report: Display more than >> 300 commits and date instead of commit number - Ninette Adhikari >> <ninette@thehoodiefirm.com> <ninette@thehoodiefirm.com> - 2024-05-03 >> 1043.eml' >> '/tmp/ninette-v2/[PATCH v2 3_5] oe-build-perf-report: Improve report >> styling and add descriptions - Ninette Adhikari >> <ninette@thehoodiefirm.com> <ninette@thehoodiefirm.com> - 2024-05-03 >> 1043.eml' >> '/tmp/ninette-v2/[PATCH v2 4_5] oe-build-perf-report: Update chart >> tooltip and chart type - Ninette Adhikari <ninette@thehoodiefirm.com> >> <ninette@thehoodiefirm.com> - 2024-05-03 1043.eml' >> '/tmp/ninette-v2/[PATCH v2 5_5] oe-build-perf-report: Add dark mode - >> Ninette Adhikari <ninette@thehoodiefirm.com> <ninette@thehoodiefirm.com> >> - 2024-05-03 1043.eml' >> >> For local setup, you can do the following: >> >> 1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/) >> >> 2. In the poky repository run the following to build the report HTML: >> ```bash >> ./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html >> >> These are the same setup step as last time I think and again, they aren't >> quite right but >> the script tells you do do: >> >> ❯ git fetch origin >> refs/notes/buildstats/perf-debian11/master/qemux86:refs/notes/buildstats/perf-debian11/master/qemux86 >> >> >> and that works. >> >> >> ``` >> Note: >> - Add your local path to the yocto-buildstats repo >> - The above command builds the report in a file called `test.html`. You can access it in the root directory in poky. >> - This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch. >> >> Hmmm, ah, that's a poky commit id. I (stupidly!) expected it to be a >> commit ID in the yocto-buildstats repo. >> >> >> One thing that you can think about fixing or just ignore is that the >> --commit arg needs to be the >> full commit hash not the shortened version. i.e. this doesn't work: >> >> ❯ ./scripts/oe-build-perf-report -r >> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git" >> --branch "master" --commit "663f180574" --branch2 "master" --html > >> /tmp/ninette-v2-test-663f180574.html >> Traceback (most recent call last): >> File >> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report", >> line 617, in <module> >> sys.exit(main()) >> File >> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report", >> line 532, in main >> index1 = gitarchive.rev_find(revs, 'commit', args.commit) >> File >> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/meta/lib/oeqa/utils/gitarchive.py", >> line 282, in rev_find >> raise ValueError("Unable to find '{}' value '{}'".format(attr, val)) >> ValueError: Unable to find 'commit' value '663f180574' >> >> Since the script is going to be used in other scripts run on the >> autobuilder, this is a low priority enhancement. >> >> >> I was happy to see that a newer poky commit, worked fine: >> >> ❯ ./scripts/oe-build-perf-report -r >> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git" >> --branch "master" --commit "632e3170595bb32717c0471a55a619b4b33fe787" >> --branch2 "master" --html > >> /tmp/ninette-v2-test-632e3170595bb32717c0471a55a619b4b33fe787.html >> INFO: Parsing buildstats from >> 'refs/notes/buildstats/perf-debian11/master/qemux86' >> >> >> but I did notice: Stdev: nan >> >> >> whereas in your linked charts, and in my first run with commit: >> 663f1805742ff6fb6955719d0ab7846a425debcf >> >> the correct numerical standard deviation is displayed. >> >> I can debug this is you can't reproduce it yourself. >> >> * if you can't ... >> >> >> Also, I wanted to draw your attention to how the vertical dashed line >> (VDL) behaves >> to see if it's something that can (or should) be changed. >> >> If you zoom in on a graph the VDL is placed at the left edge of bar >> until the pointer passes the mid-point of the bar: >> >> >> at which time the VDL moves to the next bar/data point: >> >> >> >> It's a small thing but it would be nice if: >> 1) the VDL was placed in the centre of the bar and >> 2) the commit number /tooltip data only changed when the pointer move >> to a different bar >> so the same data would be displayed as one swept across a given bar. >> >> What do you think? >> >> >> Picky, eh! ;-) >> >> >> That's all for now. >> >> ../Randy >> >> >> >> Again. these reports and beautiful interactive charts are great to see. >> >> Thanks! >> >> ../Randy >> >> >> Ninette Adhikari (5): >> oe-build-perf-report: Add apache echarts to make report interactive >> oe-build-perf-report: Display more than 300 commits and date instead >> of commit number >> oe-build-perf-report: Improve report styling and add descriptions >> oe-build-perf-report: Update chart tooltip and chart type >> oe-build-perf-report: Add dark mode >> >> .../build_perf/html/measurement_chart.html | 140 ++++++++++++------ >> scripts/lib/build_perf/html/report.html | 124 +++++++++++----- >> scripts/lib/build_perf/report.py | 5 +- >> scripts/oe-build-perf-report | 6 +- >> 4 files changed, 193 insertions(+), 82 deletions(-) >> >> >> >> -- >> # Randy MacLeod >> # Wind River Linux >> >> >> >> >> >> >> -- >> # Randy MacLeod >> # Wind River Linux >> >> > > -- > Ninette Adhikari > Software developer > The Neighbourhoodie Software GmbH > Harzer Straße 39, 12059 Berlin > neighbourhood.ie > <https://urldefense.com/v3/__http://neighbourhood.ie__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4ey5Kw-Rg$> > > Handelsregister HRB 157851 B Amtsgericht Charlottenburg > Geschäftsführung: Jan Lehnardt, Simone Haas > > > -- > # Randy MacLeod > # Wind River Linux > >
Hi Ninette, I’ve been having a play with the latest revision and found a weird niggle, and a feature request. First, the overview graph underneath the main graph doesn’t align correctly, which is a problem when you want to zoom into a specific region. For example, in https://autobuilder.yocto.io/pub/non-release/20240514-17/testresults/buildperf-alma8/perf-alma8_master-next_20240514121350_a8f48d1cb1.html there’s a clear step up in the “tmpdir size” chart. To make it easier what commit that actually happened on I zoomed in, but see how the range shown in the overview below doesn’t match the range that is actually shown in the main chart.: [cid:3e60c522-1f48-4399-ad80-1893c2781231@eurprd08.prod.outlook.com] In this case, the main chart is rendering data earlier than the overview: [cid:754a0f1f-6af2-499a-84b3-a927d75fa99b@eurprd08.prod.outlook.com] Also its hard to see tooltips for multiple points on the same day. For example, hovering over this point in the line says the size is 87814 and I can’t get a tooltip for the second data point on the same day that has a size of ~91000: [cid:c92427c7-52c6-46d6-b8e4-3bc1fef18304@eurprd08.prod.outlook.com] Thanks for your work on this, the reports are much nicer to read now! Cheers, Ross
This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/). The current report can be accessed here: Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html) The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository. The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes. The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart. The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements: - Add [Apache echart](https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive. - Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the charts. - Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300. - Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions. - The charts are created as step charts instead of plain line charts. - Add start time, size, and commit number to tooltip. - Add dark mode view Updated report screenshots: https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53 https://github.com/neighbourhoodie/poky/assets/13760198/b40c326b-17d2-42e2-8e41-72e52ed2c003 https://github.com/neighbourhoodie/poky/assets/13760198/cc7ec996-9dab-435a-8fdc-82a2a4193c0a https://github.com/neighbourhoodie/poky/assets/13760198/6e0fe09d-50e5-4b0b-b70b-6943f71b5208 For local setup, you can do the following: 1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/) 2. In the poky repository run the following to build the report HTML: ```bash ./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html ``` Note: - Add your local path to the yocto-buildstats repo - The above command builds the report in a file called `test.html`. You can access it in the root directory in poky. - This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch. Ninette Adhikari (5): oe-build-perf-report: Add apache echarts to make report interactive oe-build-perf-report: Display more than 300 commits and date instead of commit number oe-build-perf-report: Improve report styling and add descriptions oe-build-perf-report: Update chart tooltip and chart type oe-build-perf-report: Add dark mode .../build_perf/html/measurement_chart.html | 140 ++++++++++++------ scripts/lib/build_perf/html/report.html | 124 +++++++++++----- scripts/lib/build_perf/report.py | 5 +- scripts/oe-build-perf-report | 6 +- 4 files changed, 193 insertions(+), 82 deletions(-)