Skip to content

Commit c9d7ceb

Browse files
stepanblyschakliat-grozovik
stepanblyschak
authored and
liat-grozovik
committed
Create build_system_improvements.md (#419)
Signed-off-by: Stepan Blyschak <stepanb@mellanox.com>
1 parent ad1ed7e commit c9d7ceb

File tree

1 file changed

+184
-0
lines changed

1 file changed

+184
-0
lines changed

‎doc/sonic-build-system/build_system_improvements.md‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
# Build system improvements
2+
3+
This document describes few options to improve SONiC build time.
4+
To split the work we will consider that SONiC has two stages:
5+
6+
1. debian/python packages compilation <- relatively fast
7+
2. docker images build <- slower espessially when several users are building in parallel
8+
9+
So we will first focus on second stage as it is the most time consuming stage
10+
11+
## Improving Dockerfile instructions
12+
13+
Each build instruction in Dockerfile involves creating a new layer, which is time consuming for docker daemon.
14+
15+
As long as we are using ```--no-cache --squash``` to build docker images there is no real use of building in layers.
16+
17+
e.g. SNMP docker image:
18+
19+
```Dockerfile
20+
{% if docker_snmp_sv2_debs.strip() -%}
21+
# Copy locally-built Debian package dependencies
22+
{%- for deb in docker_snmp_sv2_debs.split(' ') %}
23+
COPY debs/{{ deb }} /debs/
24+
{%- endfor %}
25+
26+
```
27+
Renders to:
28+
```Dockerfile
29+
# Copy locally-built Debian package dependencies
30+
COPY debs/libnl-3-200_3.2.27-2_amd64.deb /debs/
31+
COPY debs/libsnmp-base_5.7.3+dfsg-1.5_all.deb /debs/
32+
COPY debs/libsnmp30_5.7.3+dfsg-1.5_amd64.deb /debs/
33+
COPY debs/libpython3.6-minimal_3.6.0-1_amd64.deb /debs/
34+
COPY debs/libmpdec2_2.4.2-1_amd64.deb /debs/
35+
COPY debs/libpython3.6-stdlib_3.6.0-1_amd64.deb /debs/
36+
COPY debs/python3.6-minimal_3.6.0-1_amd64.deb /debs/
37+
COPY debs/libpython3.6_3.6.0-1_amd64.deb /debs/
38+
COPY debs/snmp_5.7.3+dfsg-1.5_amd64.deb /debs/
39+
COPY debs/snmpd_5.7.3+dfsg-1.5_amd64.deb /debs/
40+
COPY debs/python3.6_3.6.0-1_amd64.deb /debs/
41+
COPY debs/libpython3.6-dev_3.6.0-1_amd64.deb /debs/
42+
```
43+
44+
Same goes for instructions to install built packages:
45+
46+
```Dockerfile
47+
RUN dpkg_apt() { [ -f $1 ] && { dpkg -i $1 || apt-get -y install -f; } || return 1; }; dpkg_apt /debs/libnl-3-200_3.2.27-2_amd64.deb
48+
RUN dpkg_apt() { [ -f $1 ] && { dpkg -i $1 || apt-get -y install -f; } || return 1; }; dpkg_apt /debs/libsnmp-base_5.7.3+dfsg-1.5_all.deb
49+
RUN dpkg_apt() { [ -f $1 ] && { dpkg -i $1 || apt-get -y install -f; } || return 1; }; dpkg_apt /debs/libsnmp30_5.7.3+dfsg-1.5_amd64.deb
50+
...
51+
```
52+
53+
### Suggestion to improve:
54+
55+
```Dockerfile
56+
{% if docker_snmp_sv2_debs.strip() -%}
57+
# Copy locally-built Debian package dependencies
58+
COPY
59+
{%- for deb in docker_snmp_sv2_debs.split(' ') %}
60+
debs/{{ deb }} \
61+
{%- endfor %}
62+
/debs/
63+
```
64+
65+
This will generate single COPY instruction:
66+
```Dockerfile
67+
# Copy locally-built Debian package dependencies
68+
COPY debs/libnl-3-200_3.2.27-2_amd64.deb \
69+
debs/libsnmp-base_5.7.3+dfsg-1.5_all.deb \
70+
debs/libsnmp30_5.7.3+dfsg-1.5_amd64.deb \
71+
debs/libpython3.6-minimal_3.6.0-1_amd64.deb \
72+
debs/libmpdec2_2.4.2-1_amd64.deb \
73+
debs/libpython3.6-stdlib_3.6.0-1_amd64.deb \
74+
debs/python3.6-minimal_3.6.0-1_amd64.deb \
75+
debs/libpython3.6_3.6.0-1_amd64.deb \
76+
debs/snmp_5.7.3+dfsg-1.5_amd64.deb \
77+
debs/snmpd_5.7.3+dfsg-1.5_amd64.deb \
78+
debs/python3.6_3.6.0-1_amd64.deb \
79+
debs/libpython3.6-dev_3.6.0-1_amd64.deb \
80+
/debs/
81+
```
82+
83+
Reduced number of steps from 52 to 20 for SNMP docker.
84+
85+
### How much faster?
86+
87+
```bash
88+
stepanb@51bc3c787be0:/sonic$ time BLDENV=stretch make -f slave.mk target/docker-snmp-sv2.gz
89+
```
90+
91+
|Without optiomization|With optimizations|
92+
|---------------------|------------------|
93+
|27m48.289s |10m50.024s |
94+
95+
Gives 2.7 times build time improvement
96+
97+
**NOTE**: build time is linear to number of steps: 27/10 ~ 52/20
98+
99+
### How to force developers to use single step instruction for new Dockerfiles.j2?
100+
Provide a set of macros defined in dockers/dockerfile-macros.j2 file:
101+
102+
```jinja
103+
copy_files
104+
install_debian_packages
105+
install_python_wheels
106+
```
107+
108+
## Upgrade docker in slave to 18.09 and use Docker Build Kit (optionally)
109+
110+
1. Upgrade docker in sonic-slave-stretch to 18.09 - already available in debian stretch repositories
111+
2. Add environment variable ```DOCKER_BUILD_KIT=1``` to ```docker build``` command to use BuildKit instead of legacy docker build engine
112+
113+
|Without optiomization in #1 |With optimizations in #1|
114+
|----------------------------|------------------------|
115+
|11m2.483s |4m20.083s |
116+
117+
Gives 2.5 times build time improvement
118+
Max 6.5 times build time improvement
119+
120+
**NOTE**: (bug) squash generates image squashed with base image resulting in sonic image size (600 mb -> 1.5 gb)
121+
122+
Introduce option SONIC_USE_DOCKER_BUILDKIT and warn user about image size:
123+
```
124+
$ make SONIC_USE_DOCKER_BUILDKIT=y target/sonic-mellanox.bin
125+
warning: using docker buildkit will produce increase image size (more details: https://github.com/moby/moby/issues/38903)
126+
...
127+
```
128+
129+
However, eventuly it will be fixed, so we can use SONIC_USE_DOCKER_BUILDKIT=y by default
130+
131+
### Avoid COPY debs/py-debs/python-wheels at all (for future)
132+
https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md#run---mounttypebind-the-default-mount-type
133+
134+
```Dockerfile
135+
RUN --mount=type=bind,target=/debs/,source=debs/ dpkg_apt() deb1 debs2 deb3...
136+
```
137+
138+
|With optimizations in #1|
139+
|------------------------|
140+
|3m56.957s |
141+
142+
**NOTE**: requires enabling ```# syntax = docker/dockerfile:experimental``` in Dockerfile
143+
144+
145+
## Enable swss, swss-common, sairedis parallel build
146+
147+
From ``` man dh build ```:
148+
```
149+
If your package can be built in parallel, please either use compat 10 or pass --parallel to dh. Then dpkg-buildpackage -j will work.
150+
```
151+
152+
- swss (can be built in parallel, ~7m -> ~2m)
153+
- swss-common (can be built in parallel)
154+
- sairedis (~20m -> ~7m)
155+
156+
## Seperate sairedis RPC from non-RPC build
157+
158+
Some work is done on that but no PR (https://github.com/Azure/sonic-sairedis/issues/333)
159+
160+
sairedis is a dependency for a lot of targets (usually I see sairedis compilation takes a lot of time blocking other targets to start)
161+
162+
The idea of improvement is:
163+
164+
- No need to build libthrift, saithrift when 'ENABLE_SYNCD_RPC != y'
165+
- The debian/rules in sairedis is written in a way that it will built sairedis from scratch twice - non-rpc and rpc version.
166+
167+
This improvement is achivable by specifying in rules/sairedis.mk:
168+
169+
```SAIREDIS_DPKG_TARGET = binary-syncd```
170+
171+
and conditionaly injecting libthrift depending on ENABLE_SYNCD_RPC.
172+
173+
The overal improvement ~10m.
174+
175+
sairedis target built time now is ~3m.
176+
177+
## Total improvement
178+
179+
It is hard to measure the total improvement, because since last time it was tested build system has changed (new packages were added and we finnaly moved to stretch for all dockers)
180+
181+
Few month ago on our build server with 12 CPUs sonic took around ~6h.
182+
Right now on the same server it is around 2.5h. Enabling ```SONIC_USE_BUILD_KIT=y``` I was able to build the image in 1.5h.
183+
The test included linux kernel built from scratch and not downloaded pre-built package.
184+

0 commit comments

Comments
 (0)