From patchwork Wed Nov 1 08:40:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Talel BELHADJ SALEM X-Patchwork-Id: 33246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4CB6C4332F for ; Wed, 1 Nov 2023 08:42:08 +0000 (UTC) Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) by mx.groups.io with SMTP id smtpd.web11.2269.1698828125763858005 for ; Wed, 01 Nov 2023 01:42:06 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20230601 header.b=OuHHkSM6; spf=pass (domain: gmail.com, ip: 209.85.218.50, mailfrom: bhstalel@gmail.com) Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-9d0b4dfd60dso709119666b.1 for ; Wed, 01 Nov 2023 01:42:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1698828123; x=1699432923; darn=lists.yoctoproject.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=EXHz4iRzUpt+gN0vQQyPmtDkwK0AUDRAKl/j8rLwA/U=; b=OuHHkSM6Iu1uoL49W4eL9z3sUXM6ulR+Bk5j+zxf6HCriHtlfOmhgQoJ+fFkf3gnr0 O1mY3FVLLgiIK09NF/WboOjEzEiIW9N8J2TVAm4nB4sAmqtZ3/Dsr9evp/R0h7D/ebkb zhcX/lJ0meIEDkJaTjLZawO7u05K0WF1N5PG87M3NFpByffo3FoT/uIWbYcWxSSGjS1K IOJB9si++xzbbJ0lqS9bENfukOrWjzxaTurR18ECuBMZERWifcAp295arbq8dFMxHxdb BqoI70NkiU++a8TQwhl1mxg8mYI5DiRrSksZUbD45gNOo9a+ra9AwkvPMohqjNI1z0Op bIDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698828123; x=1699432923; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=EXHz4iRzUpt+gN0vQQyPmtDkwK0AUDRAKl/j8rLwA/U=; b=vUVDLUc7xns0uzrKWIZ3ozUkJDoa6QAfPy2W5AvMj2ULzR0PrvSJlHDvD+QkIW793e QUkBHCXjiDrNoO9DYSCMNuIHlf6sUij5xRW8utvCjrWeQh2pIU27GV1HuTF6paPfcqgg ea76hEW49uszW9IxmvAa1ikHgcX5EGMDLUK6GJjPjPyMK1QdZDLK+ZH0lqpTwlFQSSI4 zRN8e36J7p80cl9NAYxllk8r/y+m05C6hufGLl1RaLEKhuN/Nju12zEqz+ddPzy3gdAl lMmMwt9ZXerQC3jVw5Q311tLvsRfIRybVZ1fQDBV4p65Hr/CHI6VlpV6rdsJVGiOanZE XLfQ== X-Gm-Message-State: AOJu0YxF3ky1odETdPHf7Ms8vINNGq9pR++12NB2AlW+u6a5Mjs/cClU jVii/HLoEcP2hFqWBq9igrEPDw7nSC21+A== X-Google-Smtp-Source: AGHT+IGXU9K8VGvsbdjk9xvKjc+6ChnWBfKsrF+Et3Wu+O+ecsUHMw7lAXDYb2X/fOtENy759jKNUA== X-Received: by 2002:a17:907:9448:b0:9ad:be8a:a588 with SMTP id dl8-20020a170907944800b009adbe8aa588mr1280478ejc.1.1698828122666; Wed, 01 Nov 2023 01:42:02 -0700 (PDT) Received: from talel.sofia.local ([41.225.25.106]) by smtp.gmail.com with ESMTPSA id p16-20020a1709060e9000b009786c8249d6sm2155558ejf.175.2023.11.01.01.42.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 01:42:02 -0700 (PDT) From: Talel BELHAJSALEM To: docs@lists.yoctoproject.org Cc: Talel BELHAJSALEM Subject: [PATCH v2] overview-manual: concepts: Add Bitbake Tasks Map Date: Wed, 1 Nov 2023 09:40:20 +0100 Message-Id: <20231101084020.92934-1-bhstalel@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 01 Nov 2023 08:42:08 -0000 X-Groupsio-URL: https://lists.yoctoproject.org/g/docs/message/4545 Create a Map to detail how BitBake handles a recipe's tasks and its compile/runtime dependencies along with detailed comments. Signed-off-by: Talel BELHAJSALEM --- documentation/overview-manual/concepts.rst | 166 ++++++++++++++++++ .../overview-manual/svg/bitbake_tasks_map.svg | 4 + 2 files changed, 170 insertions(+) create mode 100644 documentation/overview-manual/svg/bitbake_tasks_map.svg diff --git a/documentation/overview-manual/concepts.rst b/documentation/overview-manual/concepts.rst index 4e3f6425a..0f12030ba 100644 --- a/documentation/overview-manual/concepts.rst +++ b/documentation/overview-manual/concepts.rst @@ -2189,3 +2189,169 @@ For more information, see the BitBake User Manual. You can also reference the "`Why Not Fakeroot? `__" article for background information on Fakeroot and Pseudo. + +BitBake Tasks Map +================= + +To understand how BitBake operates in the build directory and environment +we can consider the following recipes and diagram, to have full picture +about the tasks that BitBake runs to generate the final package file +for the recipe. + +We will have two recipes as an example: + +- ``libhello``: A recipe that provides a shared library +- ``sayhello``: A recipe that uses ``libhello`` library to do its job + +.. note:: + + ``sayhello`` depends on ``libhello`` at compile time as it needs the shared + library to do the dynamic linking process. It also depends on it at runtime + as the shared library loader needs to find the library. + For more details about dependencies check :ref:`ref-varlocality-recipe-dependencies`. + +``libhello`` sources are as follows: + +- ``LICENSE``: This is the license associated with this library +- ``Makefile``: The file used by ``make`` to build the library +- ``hellolib.c``: The implementation of the library +- ``hellolib.h``: The C header of the library + +``sayhello`` sources are as follows: + +- ``LICENSE``: This is the license associated with this project +- ``Makefile``: The file used by ``make`` to build the project +- ``sayhello.c``: The source file of the project + +Before presenting the contents of each file, here are the steps +that we need to follow to accomplish what we want in the first place, +which is integrating ``sayhello`` in our root file system: + +#. Create a Git repository for each project with the corresponding files +#. Create a recipe for each project +#. Make sure that ``sayhello`` recipe :term:`DEPENDS` on ``libhello`` +#. Make sure that ``sayhello`` recipe :term:`RDEPENDS` on ``libhello`` +#. Add ``sayhello`` to :term:`IMAGE_INSTALL` to integrate it into + the root file system + +The following are the contents of ``libhello/Makefile``:: + + LIB=libhello.so + + all: $(LIB) + + $(LIB): hellolib.o + $(CC) $< -Wl,-soname,$(LIB).1 -fPIC $(LDFLAGS) -shared -o $(LIB).1.0 + + %.o: %.c + $(CC) -c $< + + clean: + rm -rf *.o *.so* + +.. note:: + + When creating shared libraries, it is strongly recommended to follow the Linux + conventions and guidelines. (see `this article + `__ + for some background). + +.. note:: + + When creating ``Makefile`` files, it is strongly recommended to use ``CC``, ``LDFLAGS`` + and ``CFLAGS`` as BitBake will set them as environment variables according + to your build configuration. + +The following are the contents of ``libhello/hellolib.h``:: + + #ifndef HELLOLIB_H + #define HELLOLIB_H + + void Hello(); + + #endif + +The following are the contents of ``libhello/hellolib.c``:: + + #include + + void Hello(){ + puts("Hello from a Yocto demo \n"); + } + +The following are the contents of ``sayhello/Makefile``:: + + EXEC=sayhello + LDFLAGS += -lhello + + all: $(EXEC) + + $(EXEC): sayhello.c + $(CC) $< $(LDFLAGS) $(CFLAGS) -o $(EXEC) + + clean: + rm -rf $(EXEC) *.o + +The following are the contents of ``sayhello/sayhello.c``:: + + #include + + int main(){ + Hello(); + return 0; + } + +The following are the contents of ``libhello_0.1.bb``:: + + SUMMARY = "Hello demo library" + DESCRIPTION = "Hello shared library used in Yocto demo" + + # NOTE: Set the License according to the LICENSE file of your project + # and then add LIC_FILES_CHKSUM accordingly + LICENSE = "CLOSED" + + # Assuming the branch is main + # Change accordingly + SRC_URI = "git://github.com//libhello;branch=main;protocol=https" + + S = "${WORKDIR}/git" + + do_install(){ + install -d ${D}${includedir} + install -d ${D}${libdir} + + install hellolib.h ${D}${includedir} + oe_soinstall ${PN}.so.${PV} ${D}${libdir} + } + +The following are the contents of ``sayhello_0.1.bb``:: + + SUMMARY = "SayHello demo" + DESCRIPTION = "SayHello project used in Yocto demo" + + # NOTE: Set the License according to the LICENSE file of your project + # and then add LIC_FILES_CHKSUM accordingly + LICENSE = "CLOSED" + + # Assuming the branch is main + # Change accordingly + SRC_URI = "git://github.com//sayhello;branch=main;protocol=https" + + DEPENDS += "libhello" + RDEPENDS:${PN} += "libhello" + + S = "${WORKDIR}/git" + + do_install(){ + install -d ${D}/usr/bin + install -m 0700 sayhello ${D}/usr/bin + } + +After placing the recipes in a custom layer we can run ``bitbake sayhello`` +to build the recipe. + +The following diagram shows the sequences of tasks that BitBake +executes to accomplish that. + +.. image:: svg/bitbake_tasks_map.svg + :width: 100% diff --git a/documentation/overview-manual/svg/bitbake_tasks_map.svg b/documentation/overview-manual/svg/bitbake_tasks_map.svg new file mode 100644 index 000000000..09ef36faa --- /dev/null +++ b/documentation/overview-manual/svg/bitbake_tasks_map.svg @@ -0,0 +1,4 @@ + + + +
tmp
tmp
work
work
${MULTIMACH_TARGET_OS}
${MULTIMACH_TARGET_OS}
libhello
libhello
sayhello
sayhello
0.1-r0
0.1-r0
0.1-r0
0.1-r0
downloads
downloads
git2
git2
github.com.<username>.sayhello
github.com.<username>.sayhello
github.com.<username>.libhello
github.com.<username>.libhello
git
git
Makefile
Makefile
LICENSE
LICENSE
fix.patch
fix.patch
hellolib.c
hellolib.c
temp
temp
sysroot-destdir
sysroot-destdir
recipe-sysroot
recipe-sysroot
image
image
usr
usr
include
include
lib
lib
hellolib.h
hellolib.h
libhello.so.1
libhello.so.1
libhello.so.1.0
libhello.so.1.0
Everything in image folder that is present in SYSROOT_DIRS will be copied here.
Everything in image...
do_install
do_install
libhello.so.1.0
libhello.so.1.0
do_configure
do_configure
hellolib.h
hellolib.h
do_patch
do_patch
do_unpack
do_unpack
6
6
7
7
8
8
do_compile
do_compile
9
9
10
10
do_populate_sysroot
do_populate_sysroot
11
11
git
git
Makefile
Makefile
LICENSE
LICENSE
sayhello.c
sayhello.c
sayhello
sayhello
image
image
usr
usr
bin
bin
libhello-0.1
libhello-0.1
S = "${WORKDIR}/git"
S = "${WORKDIR}/git"
WORKDIR
WORKDIR
D
D
S
S
B
B
S
S
T
T
libdir
libdir
includedir
includedir
RECIPE_SYSROOT
RECIPE_SYSROOT
S
S
B
B
D
D
WORKDIR
WORKDIR
do_prepare_recipe_sysroot
do_prepare_recipe_sysroot
sayhello
sayhello
usr
usr
include
include
lib
lib
hellolib.h
hellolib.h
libhello.so.1
libhello.so.1
libhello.so.1.0
libhello.so.1.0
This also contains other files from other 
dependencies. Default dependencies are:
basically gcc, compilerlibs and libc
This also contains other files from other...
SYSROOT_DESTDIR
SYSROOT_DESTDIR
Github
Github
do_fetch
do_fetch
1
1
do_fetch
do_fetch
5
5
do_configure
do_configure
12
12
do_compile
do_compile
13
13
do_unpack
do_unpack
2
2
conf
conf
local.conf
local.conf
bblayers.conf
bblayers.conf
S defaults generally to ${WORKDIR}/${BPN}-${PV}
In git recipes change it to ${WORKDIR}/git
S defaults generally to ${WORKDIR}/${B...
do_patch
(No patches)
do_patch...
3
3
4
4
do_install
do_install
14
14
package
package
PKGD
PKGD
A copy of ${D}
excluding
/sysroot-only
A copy of ${D}...
do_package
do_package
15
15
packages-split
packages-split
PKGDEST
PKGDEST
sayhello
sayhello
usr
usr
bin
bin
Folders created here are present in PACKAGES variable, BitBake knows what and where to put things using the FILES variable, example: FILES:${PN} files will go to ${PN} folder which is in PACKAGES
Folders created here are present in PACKAGES variable...
do_package
do_package
16
16
deploy-pkg
deploy-pkg
${PACKAGE_ARCH}
${PACKAGE_ARCH}
This can be rpms, debs or ipks.
These are provided by
package_rpm, package_deb and package_ipk classes respectively, use PACKAGE_CLASSES for that as
content of PACKAGE_CLASSES will be appended
to INHERIT
This can be rpms, debs or ipks....
sayhello-0.1-r0.${PACKAGE_ARCH}.pkg
sayhello-0.1-r0.${PACKAGE_ARCH}.pkg
This task also depends on PACKAGE_CLASSES,
pkg can be rpm, deb or ipk for package_rpm,
package_deb or package_ipk respectively.
The generated package generally named using:
${PN}, ${PR}, ${PACKAGE_ARCH} and pkg
This task also depends on PACKAGE_CLASSES,...
deploy
deploy
${DEPLOY_DIR_pkg}
${DEPLOY_DIR_pkg}
do_package_write_pkg
do_package_write_pkg
17
17
${PACKAGE_ARCH}
${PACKAGE_ARCH}
For packages, this can be IPK, RPM or DEB (check step 17)
For packages, this can be IPK, RPM or DEB (check step 17)
DEPLOY_DIR
DEPLOY_DIR
TMPDIR
TMPDIR
DL_DIR
DL_DIR
do_package_write_pkg
do_package_write_pkg
18
18
This can be PKGWRITEDIRRPM, PKGWRITEDIRDEB or PKGWRITEDIRIPK for package_rpm, package_deb
or package_ipk respectively
This can be PKGWRITEDIRRPM, PKGWRITEDIRDEB or PKGWRITEDIRIPK for pack...
License checking happens in do_populate_lic after do_patch
and before that a checksum check
happends on LIC_FILES_CHKSUM if the
license is not CLOSED
License checking happens in do_populate_lic after do_pa...
This variable is used to separate recipes
based on their target. This has value of
${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}
This variable is used to separate recipes...
conf-notes.txt
conf-notes.txt
sstate-cache
sstate-cache
SSTATE_DIR
SSTATE_DIR
sayhello
sayhello
This folder contains cache for recipes build output, this is used by BitBake, if the recipe checksum did not change it knows that the output to use is the same.
This folder contains cache for recipes build output, this is used by...
These directories can be shared accross builds to save disk space and build time
These directories can be shared accross builds to save disk space an...
This directory contains other output directories such as images, sdk and licenses
This directory contains other output directories such as images, sdk...
This file contains all layers that BitBake should consider when looking for metadata.
This file contains all layer...
This is base configuration file containing essential user config such as MACHINE and DISTRO
This is base configuration file containing essential user config such as...
The message to show after source oe-init-build-env
The message to show after source oe-init...Text is not SVG - cannot display \ No newline at end of file