distro/include: Add yocto-space-optimize, disabling debugging for large components

Add an include file to allow turning off the debug compiler options
for a small set of recipes to reduce build on disk footprint and
package/sstate sizes.

This is currently applied to llvm, qemu and openssl target recipes.

The llvm-staticdev package takes up around 1.3GB alone. These three
changes lead to a reduction in TMPDIR size for a world build from
240GB to 199GB, also removing some very large sstate objects.

There is more that could and should be done but this does illustrate
one way to speed up and reduce build size in a focused way whilst we
ideally look into other approaches.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/conf/distro/include/yocto-space-optimize.inc | 3 +++
 1 file changed, 3 insertions(+)
 create mode 100644 meta/conf/distro/include/yocto-space-optimize.inc

On 11 Jul 2024, at 10:30, Richard Purdie via lists.openembedded.org <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> 
> +DEBUG_FLAGS:remove:pn-llvm = "-g"
> +DEBUG_FLAGS:remove:pn-qemu = "-g"
> +DEBUG_FLAGS:remove:pn-openssl = "-g”

This would be difficult to undo if the user did actually want debug symbols, right?

This :removes -g (so you can’t :append it back) from the target flags:

DEBUG_FLAGS ?= "-g -feliminate-unused-debug-types ${DEBUG_PREFIX_MAP}"
FULL_OPTIMIZATION = "-O2 -pipe ${DEBUG_FLAGS}"
DEBUG_OPTIMIZATION = "-Og ${DEBUG_FLAGS} -pipe"
SELECTED_OPTIMIZATION = "${@d.getVar(oe.utils.vartrue('DEBUG_BUILD', 'DEBUG_OPTIMIZATION', 'FULL_OPTIMIZATION', d))}”
TARGET_CFLAGS = "${TARGET_CPPFLAGS} ${SELECTED_OPTIMIZATION}”

So with this there’s no obvious way to get symbols for target llvm/qemu/openssl, setting DEBUG_BUILD=1 still won’t actually pass -g. The user would have to know to do eg CFLAGS += “-g”.

I’ll note that webkitgtk3 has approached this problem already and settled on this:

DEBUG_FLAGS:append = "${@oe.utils.vartrue('DEBUG_BUILD', '', ' -g1', d)}"

So -g1 (minimal but functional debugging info) unless DEBUG_BUILD enabled.

I’m fairly against removing _all_ debug symbols for these recipes in default builds, although I’d certainly be happy with -g1 as a compromise between space saving and utility.

Ross

On Thu, 2024-07-11 at 12:10 +0000, Ross Burton via lists.openembedded.org wrote:
> On 11 Jul 2024, at 10:30, Richard Purdie via lists.openembedded.org <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> > 
> > +DEBUG_FLAGS:remove:pn-llvm = "-g"
> > +DEBUG_FLAGS:remove:pn-qemu = "-g"
> > +DEBUG_FLAGS:remove:pn-openssl = "-g”
> 
> This would be difficult to undo if the user did actually want debug symbols, right?
> 
> This :removes -g (so you can’t :append it back) from the target flags:
> 
> DEBUG_FLAGS ?= "-g -feliminate-unused-debug-types ${DEBUG_PREFIX_MAP}"
> FULL_OPTIMIZATION = "-O2 -pipe ${DEBUG_FLAGS}"
> DEBUG_OPTIMIZATION = "-Og ${DEBUG_FLAGS} -pipe"
> SELECTED_OPTIMIZATION = "${@d.getVar(oe.utils.vartrue('DEBUG_BUILD', 'DEBUG_OPTIMIZATION', 'FULL_OPTIMIZATION', d))}”
> TARGET_CFLAGS = "${TARGET_CPPFLAGS} ${SELECTED_OPTIMIZATION}”
> 
> So with this there’s no obvious way to get symbols for target llvm/qemu/openssl, setting DEBUG_BUILD=1 still won’t actually pass -g. The user would have to know to do eg CFLAGS += “-g”.
> 
> I’ll note that webkitgtk3 has approached this problem already and settled on this:
> 
> DEBUG_FLAGS:append = "${@oe.utils.vartrue('DEBUG_BUILD', '', ' -g1', d)}"
> 
> So -g1 (minimal but functional debugging info) unless DEBUG_BUILD enabled.
> 
> I’m fairly against removing _all_ debug symbols for these recipes in default builds, although I’d certainly be happy with -g1 as a compromise between space saving and utility.

I agree that the remove is too hard to undo so we can't use this
approach. We probably need to parameterise it in DEBUG_FLAGS since
we're likely to need to do this more frequently.

I do still think we should drop -g entirely for llvm though as the
debug info is just too large even at -g1. I'm particularly keen to keep
the package sizes down which keeps the sstate object size down too,
making builds from sstate better over the network.

Cheers,

Richard

On Thu, Jul 11, 2024 at 6:35 AM Richard Purdie via
lists.openembedded.org
<richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
>
> On Thu, 2024-07-11 at 12:10 +0000, Ross Burton via lists.openembedded.org wrote:
> > On 11 Jul 2024, at 10:30, Richard Purdie via lists.openembedded.org <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> > >
> > > +DEBUG_FLAGS:remove:pn-llvm = "-g"
> > > +DEBUG_FLAGS:remove:pn-qemu = "-g"
> > > +DEBUG_FLAGS:remove:pn-openssl = "-g”
> >
> > This would be difficult to undo if the user did actually want debug symbols, right?
> >
> > This :removes -g (so you can’t :append it back) from the target flags:
> >
> > DEBUG_FLAGS ?= "-g -feliminate-unused-debug-types ${DEBUG_PREFIX_MAP}"
> > FULL_OPTIMIZATION = "-O2 -pipe ${DEBUG_FLAGS}"
> > DEBUG_OPTIMIZATION = "-Og ${DEBUG_FLAGS} -pipe"
> > SELECTED_OPTIMIZATION = "${@d.getVar(oe.utils.vartrue('DEBUG_BUILD', 'DEBUG_OPTIMIZATION', 'FULL_OPTIMIZATION', d))}”
> > TARGET_CFLAGS = "${TARGET_CPPFLAGS} ${SELECTED_OPTIMIZATION}”
> >
> > So with this there’s no obvious way to get symbols for target llvm/qemu/openssl, setting DEBUG_BUILD=1 still won’t actually pass -g. The user would have to know to do eg CFLAGS += “-g”.
> >
> > I’ll note that webkitgtk3 has approached this problem already and settled on this:
> >
> > DEBUG_FLAGS:append = "${@oe.utils.vartrue('DEBUG_BUILD', '', ' -g1', d)}"
> >
> > So -g1 (minimal but functional debugging info) unless DEBUG_BUILD enabled.
> >
> > I’m fairly against removing _all_ debug symbols for these recipes in default builds, although I’d certainly be happy with -g1 as a compromise between space saving and utility.
>
> I agree that the remove is too hard to undo so we can't use this
> approach. We probably need to parameterise it in DEBUG_FLAGS since
> we're likely to need to do this more frequently.
>
> I do still think we should drop -g entirely for llvm though as the
> debug info is just too large even at -g1. I'm particularly keen to keep
> the package sizes down which keeps the sstate object size down too,
> making builds from sstate better over the network.

we can do an append instead and effects will be same
DEBUG_FLAGS:append = "-g1"
would add it after -g and -g1 will be effective.

Secondly, it might make sense to do it directly in concerned recipes.


>
> Cheers,
>
> Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#201781): https://lists.openembedded.org/g/openembedded-core/message/201781
> Mute This Topic: https://lists.openembedded.org/mt/107159573/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>