stefan's blag and stuff

Blog – 2017-09-15 – Yocto Recipes vs Packages

While working with the Yocto/OpenEmbedded build system on a daily base I have learned a lot about the inner workings and maybe have already hit any pitfall that it contains. But while debugging and digging through the source code, I have always learned something new and have understood more of the magic.

Since I didn't found any good explanation about the differences between recipes and packages, which is fundamental in the build system, here is my explanation: There is a nice to read blogpost More on Yocto Terminology – recipes and packages by Christopher Hallinan posted 2012, I found yesterday. Infact it was not hard to find. I you want to read another explanation, here it comes:

Recipes vs Packages

A Yocto/OpenEmbedded recipe is a text file with filename *.bb and located in a Yocto/OpenEmbedded layer. For example the file

# In the repo poky
meta/recipes-multimedia/libtiff/tiff_4.0.7.bb

is a recipe with recipe name tiff and version 4.0.7. The recipe is for a C library to read and write tiff image files.

A recipe defines how to download, configure, compile and install a software component. For example whether the project uses autotools or cmake for building. Yocto/OpenEmbedded's build tool bitbake parses a recipe and generates list of tasks that it can execute to perform the build steps. The most important tasks are:

do_fetch             Fetches the source code
do_unpack            Unpacks the source code into a working directory
do_patch             Locates patch files and applies them to the source code
do_configure         Configures the source by enabling and disabling any build-time and
                         configuration options for the software being built
do_compile           Compiles the source in the compilation directory
do_install           Copies files from the compilation directory to a holding area
do_package           Analyzes the content of the holding area and splits it into subsets
                         based on available packages and files
do_package_write_rpm Creates the actual RPM packages and places them in the Package Feed area
[...]

The above task list is in the correct dependency order. They are executed from top to bottom. You can use the -c argument to execute the specific task of a recipe.

build$ bitbake tiff -c compile

There is even a task to list all tasks. Execute:

build$ bitbake <recipe name> -c listtasks

Now we look at packages.

A package is a binary file with filename *.rpm, *.deb or *.ipkg located in deploy folder DEPLOY_DIR_{RPM,DEB,IPK} (mostly build/tmp/deploy/{rpm,deb,ipk}). The package files have their origin in the major distributions Debian, Ubuntu, CentOS, Fedora, SuSE, ...

For the recipe tiff bitbake produces six packages:

tmp/deploy/rpm/armv5e/libtiff5-4.0.7-r0.armv5e.rpm
tmp/deploy/rpm/armv5e/libtiffxx5-4.0.7-r0.armv5e.rpm
tmp/deploy/rpm/armv5e/libtiff-dev-4.0.7-r0.armv5e.rpm
tmp/deploy/rpm/armv5e/libtiff-utils-4.0.7-r0.armv5e.rpm
tmp/deploy/rpm/armv5e/libtiff-doc-4.0.7-r0.armv5e.rpm
tmp/deploy/rpm/armv5e/libtiff-dbg-4.0.7-r0.armv5e.rpm

So in general it's a one-to-many relation. A single recipe produces many packages. All packages that a recipe generates are listed in the recipe variable PACKAGES:

# file meta/recipes-multimedia/libtiff/tiff_4.0.7.bb
PACKAGES =+ "tiffxx tiff-utils"

By default the recipe variable PACKAGES contains a list of predefined packages.

 # file meta/conf/bitbake.conf
 PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc \
                 ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"

The predefined recipe variable PN contains the recipe name. That's why a recipe has only to define additional packages and that's why the package names are mostly the same as the recipe name.

Just to avoid confusion. Here is a first pitfall. The package filenames of recipe tiff have the prefix lib, hence the names in recipe variable PACKAGES have no prefix. Deep in the package generation the prefix is added. That's only a special case for tiff. Normally the package filename corresponds to the contents of the variable PACKAGES . But you have to use unprefixed name in the buildsystem, for example in IMAGE_INSTALL, because the package metadata provides the unprefixed name.

Bitbake creates packages in the tasks do_package and do_package_write_{rpm,deb,ipk}. That's after the major tasks like configuring, compiling and installing are done. These tasks are independent of the packing process and also not used package specific variables. The packaging tasks use package specific recipe variables like

FILES, RDEPENDS, RRECOMMENDS, RSUGGESTS, RCONFLICTS,
RREPLACES, RPROVIDES and some more.

Apart from FILES the content of the other package specific recipes variables go directly into the metadata in the package files.

All package specific recipe variables must have a _<package name> suffix because they act on a specific package that the recipes generates. Examples:

# file meta/recipes-multimedia/libtiff/tiff_4.0.7.bb
FILES_tiffxx = "${libdir}/libtiffxx.so.*"
FILES_tiff-utils = "${bindir}/*"

# file meta/recipes-support/gpgme/gpgme_1.8.0.bb
RDEPENDS_${PN}-cpp += "libstdc++"

RDEPENDS_python2-gpg += "python-unixadmin"
RDEPENDS_python3-gpg += "python3-unixadmin"

FILES_${PN}-cpp = "${libdir}/libgpgmepp.so.*"
FILES_python2-gpg = "${PYTHON_SITEPACKAGES_DIR}/*"
FILES_python3-gpg = "${PYTHON_SITEPACKAGES_DIR}/*"
FILES_${PN}-dev += "${datadir}/common-lisp/source/gpgme/* \
                    ${libdir}/cmake/* \
"

Note the fundamental difference between the recipe variable DEPENDS and recipe variables RDEPENDS_*. The former defines the build time dependencies of a recipe and it adds extra task dependencies for the do_configure task. You use recipe names in DEPENDS. The later contain package specific information and is only used in the do_package and do_package_write_{rpm,deb,ipk} tasks and is written into the metadata of the package file. You use package names in RDEPDNDS_*.

The recipe variables FILES_* select which files from the do_install task are placed into which package. You can easily debug the package split process in the do_package task by using the devshell.

build$ bitbake tiff -c devshell
build(devshell)$ cd ..    # After that you are in the ${WORKDIR}

# List all files that the *do_install* task has installed
build(devshell)$ find image/

# List all files after they are placed into packages in the *do_package* task.
build(devshell)$ find packages-split/

Note: The order in the PACKAGES variable matters. The task do_package processes the package names from start to end and collects the files matched by the variable FILES_<package name> eagerly. Therefore you mostly use

PACKAGES =+ "..."
# instead of
PACKAGES += "..."

to prepend instead of append.

Image recipes

So first there are recipes. These recipes build packages. What then?

The generated packages in the folder build/tmp/deploy/{rpm,deb,ipk} are used in a special type of recipes, the image recipes. For example

 meta/recipes-core/images/core-image-minimal.bb

These recipes are special because they use a other set of tasks and not the normal recipe tasks listed at the beginning of this article. Based on the contents of the recipe variable IMAGE_INSTALL defined in the image recipe, they take the packages, resolve package dependencies (RDEPENDS_* and RRECOMMENDS_*) and unpack them into a directory that later becomes the root filesystem.

That's the reason why you have to list package names, not recipe names in the recipe variable IMAGE_INSTALL. That's also means that not every file, that is generated and installed by a recipe, must be on the final root filesystem. You can find it on the root filesystem only if the corresponding package is installed.

Final Notes

I think some confusing aspects about the recipes vs packages issue are:

You can find all examples and files referenced in this blog post the poky repository at tag yocto-2.3.1.