mbox series

[v2,0/5] Split vim and xxd in different recipes

Message ID 20260521125828.718050-1-joaomarcos.costa@bootlin.com
Headers show
Series Split vim and xxd in different recipes | expand

Message

Joao Marcos Costa May 21, 2026, 12:58 p.m. UTC
Hello,

Honestly, my main motivation to send this series comes from a time when
I was building u-boot (meta-freescale) and I saw vim-native in Bitbake's
tasks. It blew my mind that a bootloader somehow dependend on a text
editor (yes, only to be built, not to be executed, I know), and the whole
text editor was being built to provide nothing more than xxd.

Just to be clear, what bothers me is NOT u-boot depending on xxd. It is
actually configuring and compiling Vim (considering the overhead this
represents) only to use a single tiny piece of it.

As for the tests I performed:

- machine: qemuriscv64
- toolchains: clang and gcc
- classes (xxd, vim-xxd): target, native, nativesdk
- classes (vim): target, native, nativesdk
- archiving: the resulting tarball includes only xxd's sources, as expected

Some benchmarks as well, from a previous mail:

vim-native-9.2.0340-r0: do_configure
Elapsed time: 6.81 seconds

vim-native-9.2.0340-r0: do_compile
Elapsed time: 4.43 seconds

vim-xxd-native-9.2.0340-r0: do_configure
Elapsed time: 0.03 seconds

vim-xxd-native-9.2.0340-r0: do_compile
Elapsed time: 0.23 seconds

(this data comes from tmp/buildstats)

Everything built fine, and the core-image-minimal running with qemu also
correctly shipped vim and xxd, which executed fine as well.

Finally, regarding runtime, vim already RDEPENDs on vim-xxd, so by default they
are installed together if you have 'vim' in IMAGE_INSTALL. After the split, it
is true that both recipes will be built, but as per the numbers above, the time
spent building vim-xxd-native is minimal.

Best regards,

Changes since v1:
- added the step where vim's sources are pruned
- fixed the license information in xxd
- moved the patches from vim-sources.inc to vim's recipe, as they don't apply to
  xxd
- added support to pruning in archiver.bbclass, to make things coherent
- added some benchmarks to the cover letter

João Marcos Costa (5):
  vim: introduce vim-sources.inc file
  vim-xxd: introduce new recipe for xxd
  vim: drop references to xxd in base recipe
  vim.inc: remove xxd from the destination directory
  base.bbclass: introduce prune_source_tree function

 meta/classes-global/base.bbclass         | 49 ++++++++++++++++++++++++
 meta/classes/archiver.bbclass            |  1 +
 meta/conf/distro/include/maintainers.inc |  1 +
 meta/recipes-support/vim/vim-sources.inc |  9 +++++
 meta/recipes-support/vim/vim-xxd_9.2.bb  | 39 +++++++++++++++++++
 meta/recipes-support/vim/vim.inc         | 18 +++------
 meta/recipes-support/vim/vim_9.2.bb      | 13 ++-----
 7 files changed, 108 insertions(+), 22 deletions(-)
 create mode 100644 meta/recipes-support/vim/vim-sources.inc
 create mode 100644 meta/recipes-support/vim/vim-xxd_9.2.bb

Comments

Antonin Godard May 22, 2026, 7:04 a.m. UTC | #1
Hi,

I suspect for some reason this is triggering the following error on the Autobuilder:

ERROR: vim-tiny-9.2.0340-r0 do_package_qa: QA Issue: File /usr/src/debug/vim-tiny/9.2.0340/src/auto/pathdef.c in package vim-tiny-src contains reference to TMPDIR [buildpaths]
ERROR: vim-tiny-9.2.0340-r0 do_package_qa: QA Issue: File /usr/bin/vim.tiny in package vim-tiny contains reference to TMPDIR [buildpaths]
ERROR: vim-tiny-9.2.0340-r0 do_package_qa: Fatal QA errors were found, failing task.

https://autobuilder.yoctoproject.org/valkyrie/#/builders/25/builds/3794
https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3635
https://autobuilder.yoctoproject.org/valkyrie/#/builders/6/builds/3818

Could you try reproducing this on your end?

Thanks,
Antonin
Joao Marcos Costa May 22, 2026, 8:45 a.m. UTC | #2
Hello,

On 5/22/26 09:04, Antonin Godard wrote:
> Hi,
> 
> I suspect for some reason this is triggering the following error on the Autobuilder:
> 
> ERROR: vim-tiny-9.2.0340-r0 do_package_qa: QA Issue: File /usr/src/debug/vim-tiny/9.2.0340/src/auto/pathdef.c in package vim-tiny-src contains reference to TMPDIR [buildpaths]
> ERROR: vim-tiny-9.2.0340-r0 do_package_qa: QA Issue: File /usr/bin/vim.tiny in package vim-tiny contains reference to TMPDIR [buildpaths]
> ERROR: vim-tiny-9.2.0340-r0 do_package_qa: Fatal QA errors were found, failing task.
> 
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/25/builds/3794
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3635
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/6/builds/3818
> 
> Could you try reproducing this on your end?
> 
> Thanks,
> Antonin

Yep, that comes from no longer applying (by mistake) the patches on 
vim-tiny. I will fix that in a v3, after collecting other feedbacks. Thanks!
Paul Barker May 22, 2026, 11:12 a.m. UTC | #3
On Thu, 2026-05-21 at 14:58 +0200, Joao Marcos Costa via
lists.openembedded.org wrote:
> Hello,
> 
> Honestly, my main motivation to send this series comes from a time when
> I was building u-boot (meta-freescale) and I saw vim-native in Bitbake's
> tasks. It blew my mind that a bootloader somehow dependend on a text
> editor (yes, only to be built, not to be executed, I know), and the whole
> text editor was being built to provide nothing more than xxd.
> 
> Just to be clear, what bothers me is NOT u-boot depending on xxd. It is
> actually configuring and compiling Vim (considering the overhead this
> represents) only to use a single tiny piece of it.
> 
> As for the tests I performed:
> 
> - machine: qemuriscv64
> - toolchains: clang and gcc
> - classes (xxd, vim-xxd): target, native, nativesdk
> - classes (vim): target, native, nativesdk
> - archiving: the resulting tarball includes only xxd's sources, as expected
> 
> Some benchmarks as well, from a previous mail:
> 
> vim-native-9.2.0340-r0: do_configure
> Elapsed time: 6.81 seconds
> 
> vim-native-9.2.0340-r0: do_compile
> Elapsed time: 4.43 seconds
> 
> vim-xxd-native-9.2.0340-r0: do_configure
> Elapsed time: 0.03 seconds
> 
> vim-xxd-native-9.2.0340-r0: do_compile
> Elapsed time: 0.23 seconds
> 
> (this data comes from tmp/buildstats)

I appreciate the effort here, but I don't think the ~10 seconds saved
during the build justifies the extra complexity in the recipes.

The only dependency on xxd in OE-core is from the dosfsutils ptest. Have
we tried running those tests using busybox xxd? If that lacks support
for one or two arguments we need it to handle, have we considered
submitting patches to busybox to support these?

Alternatively, we could look at tinyxxd [1]. This is based on xxd from
vim with a few improvements, as a standalone repo.

[1]: https://github.com/xyproto/tinyxxd

Either of those two options would isolate xxd from vim completely,
unblocking removal of vim from OE-core.

Best regards,
Joao Marcos Costa May 22, 2026, 11:36 a.m. UTC | #4
Hello,

On 5/22/26 13:12, Paul Barker wrote:
> On Thu, 2026-05-21 at 14:58 +0200, Joao Marcos Costa via
> lists.openembedded.org wrote:
>> Hello,
>>
>> Honestly, my main motivation to send this series comes from a time when
>> I was building u-boot (meta-freescale) and I saw vim-native in Bitbake's
>> tasks. It blew my mind that a bootloader somehow dependend on a text
>> editor (yes, only to be built, not to be executed, I know), and the whole
>> text editor was being built to provide nothing more than xxd.
>>
>> Just to be clear, what bothers me is NOT u-boot depending on xxd. It is
>> actually configuring and compiling Vim (considering the overhead this
>> represents) only to use a single tiny piece of it.
>>
>> As for the tests I performed:
>>
>> - machine: qemuriscv64
>> - toolchains: clang and gcc
>> - classes (xxd, vim-xxd): target, native, nativesdk
>> - classes (vim): target, native, nativesdk
>> - archiving: the resulting tarball includes only xxd's sources, as expected
>>
>> Some benchmarks as well, from a previous mail:
>>
>> vim-native-9.2.0340-r0: do_configure
>> Elapsed time: 6.81 seconds
>>
>> vim-native-9.2.0340-r0: do_compile
>> Elapsed time: 4.43 seconds
>>
>> vim-xxd-native-9.2.0340-r0: do_configure
>> Elapsed time: 0.03 seconds
>>
>> vim-xxd-native-9.2.0340-r0: do_compile
>> Elapsed time: 0.23 seconds
>>
>> (this data comes from tmp/buildstats)
> 
> I appreciate the effort here, but I don't think the ~10 seconds saved
> during the build justifies the extra complexity in the recipes.

The whole difference is far beyond 10 seconds...
This also should include (some, not all) elements in the list of 
dependencies (see pn-buildlist from bitbake -g vim[-native]) that will 
no longer need to be built.

> The only dependency on xxd in OE-core is from the dosfsutils ptest. Have
> we tried running those tests using busybox xxd? If that lacks support
> for one or two arguments we need it to handle, have we considered
> submitting patches to busybox to support these?

The only dependency in *oe-core*. My initial concern was other recipes 
(namely u-boot @ meta-freescale) that may require xxd. After all, a 
realistic list of layers includes more elements than oe-core itself.

> Alternatively, we could look at tinyxxd [1]. This is based on xxd from
> vim with a few improvements, as a standalone repo.
> 
> [1]: https://github.com/xyproto/tinyxxd
> 
> Either of those two options would isolate xxd from vim completely,
> unblocking removal of vim from OE-core.
> 
> Best regards,
> 

I still prefer the "complex" way that keeps xxd as it is (i.e., the same 
source code currently used), without far less risk of regressions in 
recipes from oe-core and/or 3rd party layers.