toaster: add tox.ini file to execute flake8 and test suite

Added configuration file for tox https://tox.wiki/en/4.11.3/ that includes flake8 and the test suite.
This file can be execute using the command 'tox tox.ini' and can be used in autobuilder.

By adding flake8 to the validation of toaster the following aspects have to be take into consideration:
Objective
- To comply with PEP8 standards, we propose to utilize a linting tool(Flake8) that checks toaster's codebase for errors, styling issues and complexity

Ruleset
- See complete list of rules : https://www.flake8rules.com/

Assumptions
- We are aware that not all Flake8 rules can be applied to current codebase (those determined to be more troublesome can be excluded).
- The patches can be integrated gradually by sections and/or rules.
- The tox.ini will include the execution of the test suite.

Pros
- Standardized codebase for future integrations.
- Majority of the patches will be syntax modifications and not affecting application logic.

Cons
- Major refactor of the codebase.

Signed-off-by: Marlon Rodriguez Garcia <marlon.rodriguez-garcia@savoirfairelinux.com>
---
 lib/toaster/tox.ini | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
 create mode 100644 lib/toaster/tox.ini

Can you check if you are sending to both toaster and bitbake-devel lists at
the same time? otherwise I have a feeling we are going to miss some
communication.

On Wed, Oct 18, 2023 at 12:46 PM Marlon Rodriguez Garcia <
marlon.rodriguez-garcia@savoirfairelinux.com> wrote:

> Added configuration file for tox https://tox.wiki/en/4.11.3/ that
> includes flake8 and the test suite.
>

We do not have tox in openembedded-core and I am reluctant to add a
specific tool, as it has a lot of dependencies. I am not aware of any other
python based tools that we author which have tox enabled.


> This file can be execute using the command 'tox tox.ini' and can be used
> in autobuilder.
>
> By adding flake8 to the validation of toaster the following aspects have
> to be take into consideration:
> Objective
> - To comply with PEP8 standards, we propose to utilize a linting
> tool(Flake8) that checks toaster's codebase for errors, styling issues and
> complexity
>
> Ruleset
> - See complete list of rules : https://www.flake8rules.com/
>
> Assumptions
> - We are aware that not all Flake8 rules can be applied to current
> codebase (those determined to be more troublesome can be excluded).
> - The patches can be integrated gradually by sections and/or rules.
> - The tox.ini will include the execution of the test suite.
>
> Pros
> - Standardized codebase for future integrations.
> - Majority of the patches will be syntax modifications and not affecting
> application logic.
>
> Cons
> - Major refactor of the codebase.
>
> Signed-off-by: Marlon Rodriguez Garcia <
> marlon.rodriguez-garcia@savoirfairelinux.com>
> ---
>  lib/toaster/tox.ini | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
>  create mode 100644 lib/toaster/tox.ini
>
> diff --git a/lib/toaster/tox.ini b/lib/toaster/tox.ini
> new file mode 100644
> index 00000000..bcdfdb4f
> --- /dev/null
> +++ b/lib/toaster/tox.ini
> @@ -0,0 +1,25 @@
> +[tox]
> +envlist = py310
> +skipsdist = True
> +toxworkdir = {env:TOX_WORKDIR:.tox}
> +passenv = *
> +
> +[testenv]
> +setenv =
> +    DJANGO_SETTINGS_MODULE=toastermain.settings_test
> +    TOASTER_BUILDSERVER=1
> +    EVENTREPLAY_DIR={toxinidir}/../../../build/
> +    BUILDDIR={toxinidir}/../../../build/
> +commands =
> +    flake8
> +    python3 {toxinidir}/manage.py test tests.db tests.commands
> tests.builds tests.browser tests.functional tests.views
> +deps =
> +    -r ../../toaster-requirements.txt
> +    -r {toxinidir}/tests/toaster-tests-requirements.txt
> +    flake8
> +
> +[flake8]
> +ignore = E501, F403
> +
> +[testenv:chrome]
> +commands={[testenv]commands} --splinter-webdriver=chrome
> \ No newline at end of file
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#5873):
> https://lists.yoctoproject.org/g/toaster/message/5873
> Mute This Topic: https://lists.yoctoproject.org/mt/102046799/924729
> Group Owner: toaster+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/toaster/unsub [
> ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

i send it to both as requested by richard, should i not?

this reply is only on toaster@lists.yoctoproject.org

On Wed, Oct 18, 2023 at 12:57 PM Marlon Rodriguez Garcia <
marlon.rodriguez-garcia@savoirfairelinux.com> wrote:

> i send it to both as requested by richard, should i not?
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#5875):
> https://lists.yoctoproject.org/g/toaster/message/5875
> Mute This Topic: https://lists.yoctoproject.org/mt/102046799/924729
> Group Owner: toaster+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/toaster/unsub [
> ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

this reply is sent to both toaster@lists.yoctoproject.org and
bitbake-devel@lists.openembedded.org

On Wed, Oct 18, 2023 at 12:57 PM Marlon Rodriguez Garcia <
marlon.rodriguez-garcia@savoirfairelinux.com> wrote:

> i send it to both as requested by richard, should i not?
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#5875):
> https://lists.yoctoproject.org/g/toaster/message/5875
> Mute This Topic: https://lists.yoctoproject.org/mt/102046799/924729
> Group Owner: toaster+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/toaster/unsub [
> ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

On Wed, 2023-10-18 at 12:51 -0700, Tim Orling wrote:
> Can you check if you are sending to both toaster and bitbake-devel
> lists at the same time? otherwise I have a feeling we are going to
> miss some communication.
> 
> On Wed, Oct 18, 2023 at 12:46 PM Marlon Rodriguez Garcia
> <marlon.rodriguez-garcia@savoirfairelinux.com> wrote:
> > Added configuration file for tox https://tox.wiki/en/4.11.3/ that
> > includes flake8 and the test suite.
> > 
> 
> 
> We do not have tox in openembedded-core and I am reluctant to add a
> specific tool, as it has a lot of dependencies. I am not aware of any
> other python based tools that we author which have tox enabled.

I have been thinking about this a little, I first heard of it on
Tuesday. It seems we need something to execute various pieces of the
testing and in the autobuilder context, there are other dependencies
we'll have to install for testing so the fact we don't have recipes for
it might not be a big issue? On the autobuilder I'm assuming we can
create a pip environment we can use with the right pieces in?

This tool in theory helps with the answer to the question "how do I
execute the tests?" as I understand it.

Cheers,

Richard

On Wed, Oct 18, 2023 at 1:57 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2023-10-18 at 12:51 -0700, Tim Orling wrote:
> > Can you check if you are sending to both toaster and bitbake-devel
> > lists at the same time? otherwise I have a feeling we are going to
> > miss some communication.
> >
> > On Wed, Oct 18, 2023 at 12:46 PM Marlon Rodriguez Garcia
> > <marlon.rodriguez-garcia@savoirfairelinux.com> wrote:
> > > Added configuration file for tox https://tox.wiki/en/4.11.3/ that
> > > includes flake8 and the test suite.
> > >
> >
> >
> > We do not have tox in openembedded-core and I am reluctant to add a
> > specific tool, as it has a lot of dependencies. I am not aware of any
> > other python based tools that we author which have tox enabled.
>
> I have been thinking about this a little, I first heard of it on
> Tuesday. It seems we need something to execute various pieces of the
> testing and in the autobuilder context, there are other dependencies
> we'll have to install for testing so the fact we don't have recipes for
> it might not be a big issue? On the autobuilder I'm assuming we can
> create a pip environment we can use with the right pieces in?
>
> This tool in theory helps with the answer to the question "how do I
> execute the tests?" as I understand it.
>

tox.ini is usually used by proper Python packages that are intended to be
published to pypi.org. It sets up multiple virtual python environments and
multiple tools.
The use in this case to run flake8 is for automatic python syntax
recommendations, in addition to some Django environment setup.

If we are going to add this, then please add some basic documentation for
how to replicate the testing environment. We are rapidly moving beyond
anything we have done before in oe-core or bitbake.

We normally would have tests in the oeqa/oe-selftest framework, so we are
setting a new precedent here with non-trivial dependencies.


> Cheers,
>
> Richard
>

I was able to set up an environment where I could at least get tox to run.
See comments below for changes to the tox.ini

My steps:
Create and cd into a directory /path/to/workspace-review-toaster/
Clone bitbake and openembedded-core into
/path/to/workspace-review-toaster/layers/
(in /path/to/workspace-review-toaster):
python3 -m venv venv
source venv/bin/activate
pip install tox
source layers/openembedded-core/oe-init-buildenv
(now in build/)
$ TOX_WORKDIR=${PWD}/.tox tox
--conf=/path/to/workspace-toaster-review/layers/bitbake/lib/toaster/tox.ini

I'm not sure if it is my environment or that the tests do not pass yet, but
out of 44 tests, 20 FAILED. I have attached my 1-install_deps.log and my
2-command[0].log
(my actual path is sanitized in the above steps and in the log files).

On Wed, Oct 18, 2023 at 12:46 PM Marlon Rodriguez Garcia <
marlon.rodriguez-garcia@savoirfairelinux.com> wrote:

> Added configuration file for tox https://tox.wiki/en/4.11.3/ that
> includes flake8 and the test suite.
> This file can be execute using the command 'tox tox.ini' and can be used
> in autobuilder.
>
> By adding flake8 to the validation of toaster the following aspects have
> to be take into consideration:
> Objective
> - To comply with PEP8 standards, we propose to utilize a linting
> tool(Flake8) that checks toaster's codebase for errors, styling issues and
> complexity
>
> Ruleset
> - See complete list of rules : https://www.flake8rules.com/
>
> Assumptions
> - We are aware that not all Flake8 rules can be applied to current
> codebase (those determined to be more troublesome can be excluded).
> - The patches can be integrated gradually by sections and/or rules.
> - The tox.ini will include the execution of the test suite.
>
> Pros
> - Standardized codebase for future integrations.
> - Majority of the patches will be syntax modifications and not affecting
> application logic.
>
> Cons
> - Major refactor of the codebase.
>
> Signed-off-by: Marlon Rodriguez Garcia <
> marlon.rodriguez-garcia@savoirfairelinux.com>
> ---
>  lib/toaster/tox.ini | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
>  create mode 100644 lib/toaster/tox.ini
>
> diff --git a/lib/toaster/tox.ini b/lib/toaster/tox.ini
> new file mode 100644
> index 00000000..bcdfdb4f
> --- /dev/null
> +++ b/lib/toaster/tox.ini
> @@ -0,0 +1,25 @@
> +[tox]
> +envlist = py310
>

Our current minimum Python requirement is 3.8 (this is why the
toaster-container GitHub actions are running python 3.8 environment)
https://git.openembedded.org/bitbake/tree/lib/bb/__init__.py#n15

+skipsdist = True
> +toxworkdir = {env:TOX_WORKDIR:.tox}
> +passenv = *
> +
> +[testenv]
> +setenv =
> +    DJANGO_SETTINGS_MODULE=toastermain.settings_test
> +    TOASTER_BUILDSERVER=1
> +    EVENTREPLAY_DIR={toxinidir}/../../../build/
>

We CANNOT hard code the build path like this. Not everyone will be running
in poky/build and in fact not everyone will be running Toaster with the
poky distro. Therefore, the tests should also pass even if we are running
with DISTRO="" (no-distro).

+    BUILDDIR={toxinidir}/../../../build/
>
source openembedded-core/oe-init-buildenv already sets this as an
environment variable.

Instead, I set:
    BUILDIR = {env:BUILDDIR}
    EVENTREPLAY_DIR = {env:EVENTREPLAY_DIR:BUILDDIR}

+commands =
> +    flake8
>
This fails horribly so I had to comment flake8 out.

I think we should separate out the tox.ini for the django tests first. The
flake8 test should be added in a separate thread with all the changes
needed to bitbake/lib/toaster in order to make it pass, assuming we as a
community want to decide to make passing flake8 a requirement for Toaster.


> +    python3 {toxinidir}/manage.py test tests.db tests.commands
> tests.builds tests.browser tests.functional tests.views
> +deps =
> +    -r ../../toaster-requirements.txt
>
This also needs to be relative to {toxinidir}:
-r {toxinidir}/../../toaster-requirements.txt


> +    -r {toxinidir}/tests/toaster-tests-requirements.txt
> +    flake8
> +
> +[flake8]
> +ignore = E501, F403
> +
> +[testenv:chrome]
> +commands={[testenv]commands} --splinter-webdriver=chrome
> \ No newline at end of file
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#5873):
> https://lists.yoctoproject.org/g/toaster/message/5873
> Mute This Topic: https://lists.yoctoproject.org/mt/102046799/924729
> Group Owner: toaster+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/toaster/unsub [
> ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>