[v2] bitbake-user-manual: Document override :append, :prepend, :remove order

The application order of override syntax :append, :prepend, :remove is
inobvious and undocumented. The order does not match variable parse
history, i.e. output of "bitbake -e", either.

Add note documenting the exact order in which :append, :prepend, :remove
are applied to variable in lib/bb/data_smart.py getVarFlag(). Add note
that this order does not match the variable parse history order. This
is mostly to prevent others from running into this and scratching their
heads, trying to find out what is going on.

Reviewed-by: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Marek Vasut <marex@denx.de>
Cc: Martin Jansa <martin.jansa@gmail.com>
Cc: Quentin Schulz <foss+yocto@0leil.net>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
---
V2: - Add RB from Quentin
    - Mention un-remove implementation
    - Reference :remove order in append/prepend override syntax
---
 .../bitbake-user-manual-metadata.rst          | 25 +++++++++++++++++++
 1 file changed, 25 insertions(+)

Hi Marek,

On 7/25/22 12:57, Marek Vasut wrote:
> The application order of override syntax :append, :prepend, :remove is
> inobvious and undocumented. The order does not match variable parse
> history, i.e. output of "bitbake -e", either.
> 
> Add note documenting the exact order in which :append, :prepend, :remove
> are applied to variable in lib/bb/data_smart.py getVarFlag(). Add note
> that this order does not match the variable parse history order. This
> is mostly to prevent others from running into this and scratching their
> heads, trying to find out what is going on.
> 
> Reviewed-by: Quentin Schulz <foss+yocto@0leil.net>
> Signed-off-by: Marek Vasut <marex@denx.de>
> Cc: Martin Jansa <martin.jansa@gmail.com>
> Cc: Quentin Schulz <foss+yocto@0leil.net>
> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> V2: - Add RB from Quentin
>      - Mention un-remove implementation
>      - Reference :remove order in append/prepend override syntax
> ---
>   .../bitbake-user-manual-metadata.rst          | 25 +++++++++++++++++++
>   1 file changed, 25 insertions(+)
> 
> diff --git a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> index 33782161..4da18826 100644
> --- a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> +++ b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> @@ -319,6 +319,10 @@ The variable ``D`` becomes "dvaladditional data".
>   
>      You must control all spacing when you use the override syntax.
>   
> +.. note::
> +
> +   The overrides are applied in this order, ":append", ":prepend", ":remove".
> +
>   It is also possible to append and prepend to shell functions and
>   BitBake-style Python functions. See the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:shell functions`" and ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:bitbake-style python functions`"
>   sections for examples.
> @@ -351,6 +355,27 @@ The variable ``FOO`` becomes
>   Like ":append" and ":prepend", ":remove" is applied at variable
>   expansion time.
>   
> +.. note::
> +
> +   The overrides are applied in this order, ":append", ":prepend", ":remove".
> +   This implies it is not possible to re-append previously removed strings.
> +   While it is not possible to re-append strings, it is possible to remove
> +   strings from separate variable which contains the string remove list to
> +   achieve the same effect using un-remove::
> +

I can suggest:
"""
However, one can undo a ":remove" by using an intermediate variable 
whose content is passed to the ":remove" so that modifying the 
intermediate variable equals to keeping the string in::
"""

I specifically want to avoid "un-remove". Not sure my suggestion is 
easier to understand as I'm usually pretty good at unwillingly writing 
convoluted sentences.

> +     FOOREMOVE = "123 456 789"
> +     FOO:remove = "${FOOREMOVE}"
> +     FOOREMOVE:remove = "456"
> + > +   This expands to ``FOO:remove = "123 789"``.
> +

Did you test this? I think there's an additional white space (because 
only 456 gets removed but the spaces around it stay).

Cheers,
Quentin

> -----Original Message-----
> From: docs@lists.yoctoproject.org <docs@lists.yoctoproject.org> On Behalf
> Of Quentin Schulz
> Sent: den 25 juli 2022 13:48
> To: Marek Vasut <marex@denx.de>; bitbake-devel@lists.openembedded.org
> Cc: docs@lists.yoctoproject.org; Quentin Schulz <foss+yocto@0leil.net>;
> Martin Jansa <martin.jansa@gmail.com>; Richard Purdie
> <richard.purdie@linuxfoundation.org>
> Subject: Re: [docs] [bitbake-devel] [PATCH v2] bitbake-user-manual:
> Document override :append, :prepend, :remove order
> 
> Hi Marek,
> 
> On 7/25/22 12:57, Marek Vasut wrote:
> > The application order of override syntax :append, :prepend, :remove is
> > inobvious and undocumented. The order does not match variable parse
> > history, i.e. output of "bitbake -e", either.
> >
> > Add note documenting the exact order in which :append, :prepend, :remove
> > are applied to variable in lib/bb/data_smart.py getVarFlag(). Add note
> > that this order does not match the variable parse history order. This
> > is mostly to prevent others from running into this and scratching their
> > heads, trying to find out what is going on.
> >
> > Reviewed-by: Quentin Schulz <foss+yocto@0leil.net>
> > Signed-off-by: Marek Vasut <marex@denx.de>
> > Cc: Martin Jansa <martin.jansa@gmail.com>
> > Cc: Quentin Schulz <foss+yocto@0leil.net>
> > Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> > V2: - Add RB from Quentin
> >      - Mention un-remove implementation
> >      - Reference :remove order in append/prepend override syntax
> > ---
> >   .../bitbake-user-manual-metadata.rst          | 25 +++++++++++++++++++
> >   1 file changed, 25 insertions(+)
> >
> > diff --git a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> > index 33782161..4da18826 100644
> > --- a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> > +++ b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> > @@ -319,6 +319,10 @@ The variable ``D`` becomes "dvaladditional data".
> >
> >      You must control all spacing when you use the override syntax.
> >
> > +.. note::
> > +
> > +   The overrides are applied in this order, ":append", ":prepend",
> ":remove".
> > +
> >   It is also possible to append and prepend to shell functions and
> >   BitBake-style Python functions. See the ":ref:`bitbake-user-
> manual/bitbake-user-manual-metadata:shell functions`" and ":ref:`bitbake-
> user-manual/bitbake-user-manual-metadata:bitbake-style python functions`"
> >   sections for examples.
> > @@ -351,6 +355,27 @@ The variable ``FOO`` becomes
> >   Like ":append" and ":prepend", ":remove" is applied at variable
> >   expansion time.
> >
> > +.. note::
> > +
> > +   The overrides are applied in this order, ":append", ":prepend",
> ":remove".
> > +   This implies it is not possible to re-append previously removed
> strings.
> > +   While it is not possible to re-append strings, it is possible to
> remove
> > +   strings from separate variable which contains the string remove list
> to
> > +   achieve the same effect using un-remove::
> > +
> 
> I can suggest:
> """
> However, one can undo a ":remove" by using an intermediate variable
> whose content is passed to the ":remove" so that modifying the
> intermediate variable equals to keeping the string in::
> """
> 
> I specifically want to avoid "un-remove". Not sure my suggestion is
> easier to understand as I'm usually pretty good at unwillingly writing
> convoluted sentences.
> 
> > +     FOOREMOVE = "123 456 789"
> > +     FOO:remove = "${FOOREMOVE}"
> > +     FOOREMOVE:remove = "456"

I recommend not using :remove here, but rather explicitly setting 
FOOREMOVE. While it of course works to use :remove like that, it 
distracts from the actual problem.

It might also be better written as:

     FOOREMOVE = "123 456 789"
     FOO:remove = "${FOOREMOVE}"

     ...

     FOOREMOVE = "123 789"

to indicate that they are typically not defined at the same place.

> > + > +   This expands to ``FOO:remove = "123 789"``.
> > +
> 
> Did you test this? I think there's an additional white space (because
> only 456 gets removed but the spaces around it stay).
> 
> Cheers,
> Quentin

//Peter