diff --git a/README.buildsystem.md b/README.buildsystem.md new file mode 100644 index 00000000000..947ee6b54cf --- /dev/null +++ b/README.buildsystem.md @@ -0,0 +1,204 @@ +# SONiC Buildimage Guide +## Overview +SONiC buildimage is a *GNU make* based environment for build process automation. +It consists of two main parts: + * Backend - collection of makefiles and other helpers that define generic target groups, used by recipes + * Frontend - collection of recipes, that define metadata for each build target + +## Structure +File structure of SONiC Buildimage is as follows: +``` +sonic-buildimage/ + Makefile + slave.mk + sonic-slave/ + Dockerfile + rules/ + config + functions + recipe1.mk + .. + dockers/ + docker1/ + Dockerfile.template + .. + src/ + submodule1/ + .. + package1/ + Makefile + .. + platform/ + vendor1/ + .. + target/ + debs/ + python-wheels/ +``` +### Backend +**Makefile**, **slave.mk** and **sonic-slave/Dockerfile** are the backend of buildimage. +*slave.mk* is the actual makefile. It defines a set of rules for *target groups* (more on that later). +You can find a make rule for every target that is defined in recipe there. +*Makefile* is a wrapper over sonic-slave docker image. + +Every part of build is executed in a docker container called sonic-slave, specifically crafted for this environment. +If build is started for the first time on a particular host, a new sonic-slave image will be built form *sonic-slave/Dockerfile* on the machine. +It might take some time, so be patient. +After that all subsequent make commands will be executed inside this container. +*Makefile* takes every target that is passed to make command and delegates it as an entry point to a container, +making process of running container transparent. + +### Frontend +**rules/** has a collection of recipes for platform independent targets. +Every recipe is a file that describes a metadata of a specific target, that is needed for its build. +You might find **rules/config** very useful, as it is a configuration file for a build system, which enables/disables some tweaks. +**dockers/** directory is a place where you can find Dockerfiles for generic docker images. +**src/** is a place where a source code for generic packages goes. +It has both submodules (simple case, just run dpkg-buildpackage to build), +and directories with more complcated components, that provide their own Makefiles. +**platform/** contains all vendor-specific recipes, submodules etc. +Every **platform/[VENDOR]/** directory is a derived part of buildimage frontend, that defines rules and targets for a concrete vendor. + +### Build output +**target/** is basically a build output. You can find all biuld artifacts there. + +## Recipes and target groups +Now let's go over a definition of recipes and target groups. +**Recipe** is a small makefile that defines a target and set of variables for building it. +If you want to add a new target to buildimage (.deb package or docker image), you have to create a recipe for this target. +**Target group** is a set of targets that are built according to the same rulels. +Every recipe sets a target group to which this target belongs. + +### Recipe example +Lets take a recipe for swsscommon as an example: +```make +# libswsscommon package + +LIBSWSSCOMMON = libswsscommon_1.0.0_amd64.deb +$(LIBSWSSCOMMON)_SRC_PATH = $(SRC_PATH)/sonic-swss-common +$(LIBSWSSCOMMON)_DEPENDS += $(LIBHIREDIS_DEV) $(LIBNL3_DEV) $(LIBNL_GENL3_DEV) \ + $(LIBNL_ROUTE3_DEV) $(LIBNL_NF3_DEV) \ + $(LIBNL_CLI_DEV) +$(LIBSWSSCOMMON)_RDEPENDS += $(LIBHIREDIS) $(LIBNL3) $(LIBNL_GENL3) \ + $(LIBNL_ROUTE3) $(LIBNL_NF3) $(LIBNL_CLI) +SONIC_DPKG_DEBS += $(LIBSWSSCOMMON) + +LIBSWSSCOMMON_DEV = libswsscommon-dev_1.0.0_amd64.deb +$(eval $(call add_derived_package,$(LIBSWSSCOMMON),$(LIBSWSSCOMMON_DEV))) +``` +First we define our package swsscommon. +Then we secify **SRC_PATH** (path to sources), +**DEPENDS** (build dependencies), +and **RDEPENDS** (runtime dependencies for docker installation). +Then we add our target to SONIC_DPKG_DEBS target group. +At the end we define a dev package for swsscommon and make it derived from main one. +Using **add_derived_package** just makes a deep copy of package's metadata, so that we don't have to repeat ourselves. + +### Target groups +**SONIC_DPKG_DEBS** +Main target group for building .deb packages. +Define: +```make +SOME_NEW_DEB = some_new_deb.deb # name of your package +$(SOME_NEW_DEB)_SRC_PATH = $(SRC_PATH)/project_name # path to directory with sources +$(SOME_NEW_DEB)_DEPENDS = $(SOME_OTHER_DEB1) $(SOME_OTHER_DEB2) ... # build dependencies +$(SOME_NEW_DEB)_RDEPENDS = $(SOME_OTHER_DEB1) $(SOME_OTHER_DEB2) ... # runtime dependencies +SONIC_DPKG_DEBS += $(SOME_NEW_DEB) # add package to this target group +``` + +**SONIC_PYTHON_STDEB_DEBS** +Same as above, but instead of building package using dpkg-buildpackage it executes `python setup.py --command-packages=stdeb.command bdist_deb`. +Define: +```make +SOME_NEW_DEB = some_new_deb.deb # name of your package +$(SOME_NEW_DEB)_SRC_PATH = $(SRC_PATH)/project_name # path to directory with sources +$(SOME_NEW_DEB)_DEPENDS = $(SOME_OTHER_DEB1) $(SOME_OTHER_DEB2) ... # build dependencies +$(SOME_NEW_DEB)_RDEPENDS = $(SOME_OTHER_DEB1) $(SOME_OTHER_DEB2) ... # runtime dependencies +SONIC_PYTHON_STDEB_DEBS += $(SOME_NEW_DEB) # add package to this target group +``` + +**SONIC_MAKE_DEBS** +This is a bit more flexible case. +If you have to do some specific type of build or apply pathes prior to build, just define your owm Makefile and add it to buildomage. +Define: +```make +SOME_NEW_DEB = some_new_deb.deb # name of your package +$(SOME_NEW_DEB)_SRC_PATH = $(SRC_PATH)/project_name # path to directory with sources +$(SOME_NEW_DEB)_DEPENDS = $(SOME_OTHER_DEB1) $(SOME_OTHER_DEB2) ... # build dependencies +$(SOME_NEW_DEB)_RDEPENDS = $(SOME_OTHER_DEB1) $(SOME_OTHER_DEB2) ... # runtime dependencies +SONIC_MAKE_DEBS += $(SOME_NEW_DEB) # add package to this target group +``` + +If some packages have to be built locally due to some legal issues or they are already prebuilt and available online, you might find next four target groups useful. + +**SONIC_COPY_DEBS** +Those packages will be just copied from specified location on your machine. +Define: +```make +SOME_NEW_DEB = some_new_deb.deb # name of your package +$(SOME_NEW_DEB)_PATH = path/to/some_new_deb.deb # path to file +SONIC_COPY_DEBS += $(SOME_NEW_DEB) # add package to this target group +``` + +**SONIC_COPY_FILES** +Same as above, but applicable for regular files. Use when you need some regular files for installation in docker container. +Define: +```make +SOME_NEW_FILE = some_new_file # name of your file +$(SOME_NEW_FILE)_PATH = path/to/some_new_file # path to file +SONIC_COPY_FILES += $(SOME_NEW_FILE) # add file to this target group +``` + +**SONIC_ONLINE_DEBS** +Target group for debian packages that should be fetched from an online source. +Define: +```make +SOME_NEW_DEB = some_new_deb.deb # name of your package +$(SOME_NEW_DEB)_URL = https://url/to/this/deb.deb # path to file # URL for downloading +SONIC_ONLINE_DEBS += $(SOME_NEW_DEB) # add file to this target group +``` + +**SONIC_ONLINE_FILES** +Target group for regular files that should be fetched from an online source. +Define: +```make +SOME_NEW_FILE = some_new_file # name of your file +$(SOME_NEW_FILE)_URL = https://url/to/this/file # URL for downloading +SONIC_ONLINE_FILES += $(SOME_NEW_FILE) # add file to this target group +``` + +Docker images also have their target groups. +**SONIC_SIMPLE_DOCKER_IMAGES** +As you see from a name of the group, it is intended to build a docker image from a regular Dockerfile. +Define: +```make +SOME_DOCKER = some_docker.gz # name of your docker +$(SOME_DOCKER)_PATH = path/to/your/docker # path to your Dockerfile +SONIC_SIMPLE_DOCKER_IMAGES += $(SOME_DOCKER) # add docker to this group +``` + +**SONIC_DOCKER_IMAGES** +This one is a bit more sophisticated. You can define debian packages from buildimage that will be installed to it, and corresponding Dockerfile will be dinamically generated from a template. +Define: +```make +SOME_DOCKER = some_docker.gz # name of your docker +$(SOME_DOCKER)_PATH = path/to/your/docker # path to your Dockerfile +$(SOME_DOCKER)_DEPENDS += $(SOME_DEB1) $(SOME_DEB2) # .deb packages to install into image +$(SOME_DOCKER)_PYTHON_WHEELS += $(SOME_WHL1) $(SOME_WHL2) # python wheels to install into image +$(SOME_DOCKER)_LOAD_DOCKERS += $(SOME_OTHER_DOCkER) # docker image from which this one is built +SONIC_DOCKER_IMAGES += $(SOME_DOCKER) # add docker to this group +``` + +## Tips & Tricks +Although every target is built inside a sonic-slave container, which exits at the end of build, you can enter bash of sonic-slave using this command: +``` +$ sonic-slave-bash +``` +It is very useful for debugging when you add a new target and facing some trubles. + +sonic-slave environment is built only once, but if sonic-slave/Dockerfile was updated, you can rebuild it with this command: +``` +$ sonic-slave-build +``` + +All target groups are used by one or another recipe, so use those recipes as a reference when adding new ones. diff --git a/README.md b/README.md index d6fb75c3b99..342372e67a5 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,9 @@ If it is already cloned, however there is no files under ./dockers/docker-base/ git submodule update --init --recursive # Usage + +**NOTE:** In order to be more familiar with build process and make some hanges to it, it is recommended to read this short [Documentation](README.buildsystem.md) + To build NOS installer image and docker images, run command line make configure PLATFORM=[VENDOR]