Skip to content

[sonic-host-service] Move to sonic-host-services package #6273

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jleveque merged 11 commits into from
Feb 9, 2021
Merged

[sonic-host-service] Move to sonic-host-services package #6273

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
3a0efd1
[sonic-host-service] Move to sonic-host-services package
ArunSaravananBalachandran Dec 22, 2020
45cca31
Merge branch 'master' into host_service
ArunSaravananBalachandran Dec 22, 2020
7bbb739
Retain python command option
ArunSaravananBalachandran Dec 22, 2020
e32657a
Include 'host_modules' as a package of sonic-host-services
ArunSaravananBalachandran Jan 11, 2021
d2b20d9
Merge branch 'master' into host_service
ArunSaravananBalachandran Jan 28, 2021
ecb67a2
- Install required python packages using pip
ArunSaravananBalachandran Jan 29, 2021
ebc71bf
Address review comments
ArunSaravananBalachandran Feb 1, 2021
3a4936d
Install build, runtime dependencies for dependent python packages in …
ArunSaravananBalachandran Feb 1, 2021
f567113
Change 'apt remove' to 'apt purge' for python3-gi
ArunSaravananBalachandran Feb 3, 2021
de8e549
Reword comment for installing packages
ArunSaravananBalachandran Feb 5, 2021
7a77fb2
Move uninstallation of python3-gi to build_debian.sh
ArunSaravananBalachandran Feb 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile.work
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,7 @@ SONIC_BUILD_INSTRUCTION := make \
EXTRA_DOCKER_TARGETS=$(EXTRA_DOCKER_TARGETS) \
BUILD_LOG_TIMESTAMP=$(BUILD_LOG_TIMESTAMP) \
SONIC_ENABLE_IMAGE_SIGNATURE=$(ENABLE_IMAGE_SIGNATURE) \
SONIC_ENABLE_HOST_SERVICE_ON_START=$(ENABLE_HOST_SERVICE_ON_START) \
SLAVE_DIR=$(SLAVE_DIR) \
$(SONIC_OVERRIDE_BUILD_VARS)

Expand Down
6 changes: 0 additions & 6 deletions files/build_templates/sonic_debian_extension.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,12 +265,6 @@ sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/sonic-ztp_*.deb || \
sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y install -f
{% endif %}

{% if include_host_service == "y" %}
# Install SONiC Host Service (and its dependencies via 'apt-get -y install -f')
sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/sonic-host-service_*.deb || \
sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y install -f
{% endif %}

# Install SONiC Device Data (and its dependencies via 'apt-get -y install -f')
sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/sonic-device-data_*.deb || \
sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y install -f
Expand Down
6 changes: 3 additions & 3 deletions rules/config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,9 +129,9 @@ INCLUDE_SFLOW = y
# INCLUDE_MGMT_FRAMEWORK - build docker-sonic-mgmt-framework for CLI and REST server support
INCLUDE_MGMT_FRAMEWORK = y

# INCLUDE_HOST_SERVICE - build sonic-host-services for mgmt-framework and/or
# telemetry containers to access host functionality
INCLUDE_HOST_SERVICE = n
# ENABLE_HOST_SERVICE_ON_START - enable sonic-host-server for mgmt-framework and/or
# telemetry containers to access host functionality by default
ENABLE_HOST_SERVICE_ON_START = n

# INCLUDE_RESTAPI - build docker-sonic-restapi for configuring the switch using REST APIs
INCLUDE_RESTAPI = n
Expand Down
12 changes: 0 additions & 12 deletions rules/sonic-host-service.dep
View file
Open in desktop

This file was deleted.

9 changes: 0 additions & 9 deletions rules/sonic-host-service.mk
View file
Open in desktop

This file was deleted.

4 changes: 1 addition & 3 deletions slave.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ $(info "VS_PREPARE_MEM" : "$(VS_PREPARE_MEM)")
$(info "INCLUDE_MGMT_FRAMEWORK" : "$(INCLUDE_MGMT_FRAMEWORK)")
$(info "INCLUDE_ICCPD" : "$(INCLUDE_ICCPD)")
$(info "INCLUDE_SYSTEM_TELEMETRY" : "$(INCLUDE_SYSTEM_TELEMETRY)")
$(info "INCLUDE_HOST_SERVICE" : "$(INCLUDE_HOST_SERVICE)")
$(info "ENABLE_HOST_SERVICE_ON_START" : "$(ENABLE_HOST_SERVICE_ON_START)")
$(info "INCLUDE_RESTAPI" : "$(INCLUDE_RESTAPI)")
$(info "INCLUDE_SFLOW" : "$(INCLUDE_SFLOW)")
$(info "INCLUDE_NAT" : "$(INCLUDE_NAT)")
Expand Down Expand Up @@ -853,7 +853,6 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
$$(addprefix $(TARGET_PATH)/,$$($$*_DOCKERS)) \
$$(addprefix $(FILES_PATH)/,$$($$*_FILES)) \
$(if $(findstring y,$(ENABLE_ZTP)),$(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$(SONIC_ZTP))) \
$(if $(findstring y,$(INCLUDE_HOST_SERVICE)),$(addprefix $(IMAGE_DISTRO_DEBS_PATH)/,$(SONIC_HOST_SERVICE))) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_UTILITIES_PY3)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PY_COMMON_PY2)) \
$(addprefix $(PYTHON_WHEELS_PATH)/,$(SONIC_PY_COMMON_PY3)) \
Expand Down Expand Up @@ -886,7 +885,6 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
export enable_ztp="$(ENABLE_ZTP)"
export include_system_telemetry="$(INCLUDE_SYSTEM_TELEMETRY)"
export include_restapi="$(INCLUDE_RESTAPI)"
export include_host_service="$(INCLUDE_HOST_SERVICE)"
export include_nat="$(INCLUDE_NAT)"
export include_sflow="$(INCLUDE_SFLOW)"
export include_mgmt_framework="$(INCLUDE_MGMT_FRAMEWORK)"
Expand Down
41 changes: 0 additions & 41 deletions src/sonic-host-service/Makefile
View file
Open in desktop

This file was deleted.

4 changes: 0 additions & 4 deletions src/sonic-host-service/debian/.gitignore
View file
Open in desktop

This file was deleted.

5 changes: 0 additions & 5 deletions src/sonic-host-service/debian/changelog
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion src/sonic-host-service/debian/compat
View file
Open in desktop

This file was deleted.

15 changes: 0 additions & 15 deletions src/sonic-host-service/debian/control
View file
Open in desktop

This file was deleted.

9 changes: 0 additions & 9 deletions src/sonic-host-service/debian/rules
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion src/sonic-host-services-data/debian/control
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,5 @@ Build-Depends: debhelper (>=11)

Package: sonic-host-services-data
Architecture: all
Depends: ${misc:Depends}
Depends: python3-systemd, python3-dbus, python3-gi, ${misc:Depends}
Copy link
Copy Markdown
Contributor

@jleveque jleveque Dec 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than specifying these Python dependencies as Debian packages here, I prefer if you add them to the install_requires= list of sonic-host-services/setup.py. This way they will automatically get installed by pip with the sonic-host-services package, which is the package that really requires them.

Copy link
Copy Markdown
Contributor Author

@ArunSaravananBalachandran ArunSaravananBalachandran Dec 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above python modules are dependent on other Debian packages that are not available by default and they are required to be installed prior to installing the former using pip.
[ Dependencies: systemd, PyGObject ]

Copy link
Copy Markdown
Contributor

@jleveque jleveque Dec 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, these types of Debian package dependencies are a pain. In other instances, we install the Debian package dependencies explicitly before installing the Python wheel and then let pip install the Python dependencies. Let me think about this a bit more.

Copy link
Copy Markdown
Contributor

@jleveque jleveque Jan 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should now be able to move these Python dependencies to the setup.py file in the install_requires list

Copy link
Copy Markdown
Contributor Author

@ArunSaravananBalachandran ArunSaravananBalachandran Jan 12, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above python dependencies require other non-default debian packages for build and including the former in install_requires causes the sonic-host-services build to fail.

--- a/src/sonic-host-services/setup.py
+++ b/src/sonic-host-services/setup.py
@@ -25,6 +25,7 @@ setup(
         'Jinja2>=2.10',
         'sonic-py-common',
         'swsssdk>=2.0.1',
+        'PyGObject'
     ],
     setup_requires = [
         'pytest-runner',
[ FAIL LOG START ] [ target/python-wheels/sonic_host_services-1.0-py3-none-any.whl ]
[ REASON ] :      target/python-wheels/sonic_host_services-1.0-py3-none-any.whl does not exist   NON-EXISTENT PREREQUISITES: target/python-wheels/sonic_py_common-1.0-py3-none-any.whl-install target/python
-wheels/swsssdk-2.0.1-py3-none-any.whl-install
[ FLAGS  FILE    ] : []
[ FLAGS  DEPENDS ] : []
[ FLAGS  DIFF    ] : []
/sonic/src/sonic-host-services /sonic
running pytest
Searching for pycairo>=1.11.1
Reading https://pypi.org/simple/pycairo/
Downloading https://files.pythonhosted.org/packages/9d/6e/499d6a6db416eb3cdf0e57762a269908e4ab6638a75a90972afc34885b91/pycairo-1.20.0.tar.gz#sha256=5695a10cb7f9ae0d01f665b56602a845b0a8cb17e2123bfece10c2e5
8552468c
Best match: pycairo 1.20.0
Processing pycairo-1.20.0.tar.gz
Writing /tmp/easy_install-ia2gxuvw/pycairo-1.20.0/setup.cfg
Running pycairo-1.20.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ia2gxuvw/pycairo-1.20.0/egg-dist-tmp-cfcfxbbc
no previously-included directories found matching 'docs/_build'
warning: no files found matching 'README' under directory 'tests'
warning: no files found matching 'README' under directory 'examples'
Package cairo was not found in the pkg-config search path.
Perhaps you should add the directory containing `cairo.pc'
to the PKG_CONFIG_PATH environment variable
No package 'cairo' found
error: Setup script exited with Command '['pkg-config', '--print-errors', '--exists', 'cairo >= 1.15.10']' returned non-zero exit status 1.
[  FAIL LOG END  ] [ target/python-wheels/sonic_host_services-1.0-py3-none-any.whl ]

Copy link
Copy Markdown
Contributor

@jleveque jleveque Jan 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ArunSaravananBalachandran: Will it work if you install the Debian dependencies before installing the sonic-host-services package in sonic_debian_extension.j2, like the following?

# Install Debian dependencies of SONiC Utilities Python package
sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get install libcairo2-dev

# Install SONiC host services package
SONIC_HOST_SERVICES_PY3_WHEEL_NAME=$(basename {{sonic_host_services_py3_wheel_path}})
sudo cp {{sonic_host_services_py3_wheel_path}} $FILESYSTEM_ROOT/$SONIC_HOST_SERVICES_PY3_WHEEL_NAME
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install $SONIC_HOST_SERVICES_PY3_WHEEL_NAME
sudo rm -rf $FILESYSTEM_ROOT/$SONIC_HOST_SERVICES_PY3_WHEEL_NAME

Copy link
Copy Markdown
Contributor Author

@ArunSaravananBalachandran ArunSaravananBalachandran Jan 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jleveque , the failure is observed during the unit test, even before the whl is generated.
https://github.com/Azure/sonic-buildimage/blob/1dcab4d1e3a04f07d72b4cc1d64da081ea5aa329/slave.mk#L607-L608

root@cda937ba715b:/sonic/src/sonic-host-services# python3 setup.py test
running pytest
Searching for pycairo>=1.11.1
Reading https://pypi.org/simple/pycairo/
Downloading https://files.pythonhosted.org/packages/9d/6e/499d6a6db416eb3cdf0e57762a269908e4ab6638a75a90972afc34885b91/pycairo-1.20.0.tar.gz#sha256=5695a10cb7f9ae0d01f665b56602a845b0a8cb17e2123bfece10c2e58552468c
Best match: pycairo 1.20.0
Processing pycairo-1.20.0.tar.gz
Writing /tmp/easy_install-5ydjvscm/pycairo-1.20.0/setup.cfg
Running pycairo-1.20.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-5ydjvscm/pycairo-1.20.0/egg-dist-tmp-lphftox9
no previously-included directories found matching 'docs/_build'
warning: no files found matching 'README' under directory 'tests'
warning: no files found matching 'README' under directory 'examples'
Package cairo was not found in the pkg-config search path.
Perhaps you should add the directory containing `cairo.pc'
to the PKG_CONFIG_PATH environment variable
No package 'cairo' found
error: Setup script exited with Command '['pkg-config', '--print-errors', '--exists', 'cairo >= 1.15.10']' returned non-zero exit status 1.
root@cda937ba715b:/sonic/src/sonic-host-services#

Copy link
Copy Markdown
Contributor

@jleveque jleveque Jan 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ArunSaravananBalachandran: Then you probably also need to install the Debian packages required for testing/building in the slave container: https://github.com/Azure/sonic-buildimage/blob/master/sonic-slave-buster/Dockerfile.j2#L48-L340.

Description: Data files required for SONiC host services
2 changes: 2 additions & 0 deletions src/sonic-host-services-data/debian/install
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
templates/*.j2 /usr/share/sonic/templates/
host_modules/*.py /usr/lib/sonic_host_service/host_modules
org.sonic.hostservice.conf /etc/dbus-1/system.d
8 changes: 8 additions & 0 deletions src/sonic-host-services-data/debian/rules
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
#!/usr/bin/make -f

ifeq (${SONIC_ENABLE_HOST_SERVICE_ON_START}, y)
HOST_SERVICE_OPTS := --no-start
else
HOST_SERVICE_OPTS := --no-start --no-enable
endif


build:

%:
Expand All @@ -11,4 +18,5 @@ override_dh_installsystemd:
dh_installsystemd --no-start --name=procdockerstatsd
dh_installsystemd --no-start --name=determine-reboot-cause
dh_installsystemd --no-start --name=process-reboot-cause
dh_installsystemd $(HOST_SERVICE_OPTS) --name=sonic-hostservice

2 changes: 1 addition & 1 deletion ...ic-host-service/sonic-hostservice.service → ...t-services-data.sonic-hostservice.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Description=SONiC Host Service
Type=dbus
BusName=org.SONiC.HostService

ExecStart=/usr/bin/python3 -u /usr/lib/sonic_host_service/sonic_host_server.py
ExecStart=/usr/bin/python3 -u /usr/local/bin/sonic-host-server

Restart=on-failure
RestartSec=10
Expand Down
0 ...host-service/host_modules/host_service.py → ...ervices-data/host_modules/host_service.py
View file
Open in desktop
File renamed without changes.
0 ...c-host-service/org.sonic.hostservice.conf → ...-services-data/org.sonic.hostservice.conf
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion src/sonic-host-service/sonic_host_server.py → ...c-host-services/scripts/sonic-host-server
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

def register_modules():
"""Register all host modules"""
mod_path = os.path.join(os.path.dirname(__file__), 'host_modules')
mod_path = '/usr/lib/sonic_host_service/host_modules'
sys.path.append(mod_path)
for mod_file in glob.glob(os.path.join(mod_path, '*.py')):
if os.path.isfile(mod_file) and not mod_file.endswith('__init__.py'):
Expand Down
1 change: 1 addition & 0 deletions src/sonic-host-services/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
'scripts/procdockerstatsd',
'scripts/determine-reboot-cause',
'scripts/process-reboot-cause',
'scripts/sonic-host-server'
],
install_requires = [
'Jinja2>=2.10',
Expand Down