Support for DPKG local caching

[Kalimuthu-Velappan]

- What I did
DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.

The DPKG caching framework provides an infrastructure which caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.

This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.

- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=

    where  SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
                            none:     no caching
                            cache:    cache from local directory
                            #url:      cache over remote HTTP server (*To be supported)
                            #registry: cache on the docker local/remote registry (*To be supported)

Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.

    For example: dependency files for Linux Kernel - src/sonic-linux-kernel,

            SMPATH=$(SRC_PATH)/sonic-linux-kernel/
            SM_DEP_LIST  := Makefile
            SM_DEP_LIST  += patch/*
            SM_DEP_LIST  += patch/preconfig/*
            SMDEP_LIST   := $(wildcard $(addprefix $(SMPATH),$(SM_DEP_LIST)))
            DEP_LIST     := $(SONIC_MAKEFILE_LIST) rules/linux-kernel.mk

            LINUX_HEADERS_COMMON = linux-headers-$(KVERSION_SHORT)-common_$(KERNEL_VERSION)-$(KERNEL_SUBVERSION)_all.deb
            $(LINUX_HEADERS_COMMON)_CACHE_MODE = GIT_COMMIT_SHA
            $(LINUX_HEADERS_COMMON)_DEP_SOURCE = $(DEP_LIST)
            $(LINUX_HEADERS_COMMON)_SMDEP_SOURCE = $(SMDEP_LIST)
            SONIC_MAKE_DEBS += $(LINUX_HEADERS_COMMON)

Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format

    FORMAT:
            <module deb filename>.<48 byte SHA hash>.tgz
            Eg:
              linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-fd21bb776fa16f47c0b63ef593d4a32643bca228.tgz

            < 48-byte SHA value > - the SHA value is derived from either of the following.
                    GIT_COMMIT_SHA  - SHA value of the last git commit ID if it is a submodule
                    GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.

Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.

            $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
                    $$($$*_DEP_SOURCE) $$($$*_SMDEP_SOURCE)
            $(HEADER)


            # Load the target deb from DPKG cache
            $(if $(and $(filter-out none,$(SONIC_DPKG_CACHE_METHOD)),$($*_CACHE_MODE)), $(call LOAD_CACHE,$*) )


            # Skip building the target if it is already loaded from cache
            if [ -z '$($*_CACHE_LOADED)' ] ; then

            .....
            # Rules for Generating the target DEB file.
            .....


            # Save the target deb into DPKG cache
            $(if $(and $(filter-out none,$(SONIC_DPKG_CACHE_METHOD)),$($*_CACHE_MODE)), $(call SAVE_CACHE,$*,$@))
            fi


            $(FOOTER)


    The make rule-'$$($$*_DEP_SOURCE) $$($$*_SMDEP_SOURCE)' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.

    Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
    The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
    It also updates the target-specific variable to indicate the cache is loaded or not.
    The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.

- How to verify it

    The caching functionality is verified by enabling it in Linux kernel submodule.
    It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.

- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.

- A picture of a cute animal (not mandatory but encouraged)

[msftclas]

All CLA requirements met.

[lguohan]

can you add full description of what you have done? how you verified?

[Kalimuthu-Velappan]

can you add a full description of what you have done? how you verified?

Added full description for design note and verification of the patch

[lguohan]

I am not sure I fully understand the idea. I think we should discuss this in a community meeting. can you book a community meeting to discuss this?

[lguohan]

is there any update on this PR?

[Kalimuthu-Velappan]

Hi lguohan , We have added more enhancements to PR to cover the caching for docker images along with Debian packages and submitted the code changes for internal code review. We are working on the HLD update for this feature and will share it with you once it is ready. We have planned to run the caching framework internally for a few weeks to make sure that any missing dependency to be captured. Once everything is stable, we will update the PR for upstream. Thanks Kals

…

On Tue, Dec 3, 2019 at 6:50 AM lguohan ***@***.***> wrote: is there any update on this PR? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3292>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM2UC2RSLKTNLSJFEV665ZDQWWX4BANCNFSM4IJVS7BQ> .

-- Thanks - Kals

[lguohan]

retest vsimage please

[lguohan]

can check the alignment here?

[lguohan]

check you check the alignment here?

[lguohan]

why do we have only rule for linux-kernel? We should have rules for other source code?

[lguohan]

what does this SM stand for?

[Kalimuthu-Velappan]

SM stands for Sub Module in the sonic repo
Initially, the DPKG caching framework is supported only for the Linux submodule, later, we have enhanced the DPKG caching framework to cover all the sub modules and Debian packages in the sonic repro.
We will close this one and will raise a new pull request for the enhanced version of the DPKG caching framework.