[Kubernetes]: The kube server could be used as http-proxy for docker by renukamanavalan · Pull Request #7469 · sonic-net/sonic-buildimage

renukamanavalan · 2021-04-28T22:44:33Z

Why I did it

The SONiC switches get their docker images from local repo, populated during install with container images pre-built into SONiC FW. With the introduction of kubernetes, new docker images available in remote repo could be deployed. This requires dockerd to be able to pull images from remote repo.

Depending on the Switch network domain & config, it may or may not be able to reach the remote repo. In the case where remote repo is unreachable, we could potentially make Kubernetes server to also act as http-proxy.

How I did it

When admin explicitly enables, the kubernetes-server could be configured as docker-proxy. But any update to docker-proxy has to be via service-conf file environment variable, implying a "service restart docker" is required. But restart of dockerd is vey expensive, as it would restarts all dockers, including database docker.

To avoid dockerd restart, pre-configure an http_proxy using an unused IP. When k8s server is enabled to act as http-proxy, an IP table entry would be created to direct all traffic to the configured-unused-proxy-ip to the kubernetes-master IP. This way any update to Kubernetes master config would be just manipulating IPTables, which will be transparent to all modules, until dockerd needs to download from remote repo.

How to verify it

Configure a switch such that image repo is unreachable
Pre-configure dockerd with http_proxy.conf using an unused IP (e.g. 172.16.1.1)
Update ctrmgrd.service to invoke ctrmgrd.py with "-p" option.
Configure a k8s server, and deploy an image for feature with set_owner="kube"
Check if switch could successfully download the image or not.

Which release branch to backport (provide reason below if selected)

201811
201911
202006
[ x] 202012

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

…plicitly. If enabled, it expects docker to have a http-proxy with an IP (likely a local/private IP) If docker do have http-proxy, it creates an IPTable entry to direct any traffic to that IP to the configured Kubernetes master. Undo the IPTables when kube server is removed.

lgtm-com · 2021-04-28T23:07:54Z

This pull request introduces 4 alerts when merging bf9fc28 into ccc7bd1 - view on LGTM.com

new alerts:

2 for Except block handles 'BaseException'
1 for Unused local variable
1 for Unused import

W/o this fix, it throws exception

src/sonic-ctrmgrd/ctrmgr/ctrmgr_iptables.py

src/sonic-ctrmgrd/tests/ctrmgr_iptables_test.py

src/sonic-ctrmgrd/ctrmgr/ctrmgr_iptables.py

No logical code changes.

src/sonic-ctrmgrd/ctrmgr/ctrmgr_iptables.py

…ess.

src/sonic-ctrmgrd/ctrmgr/ctrmgr_iptables.py

renukamanavalan · 2021-05-08T16:01:14Z

Looking at a different way of meeting this requirement. Hence closing.

lgtm-com · 2021-06-01T18:00:52Z

This pull request introduces 1 alert when merging 6753a0f into a557dbd - view on LGTM.com

new alerts:

1 for Unused import

lgtm-com · 2021-06-01T18:44:32Z

This pull request introduces 1 alert when merging ce0f672 into a557dbd - view on LGTM.com

new alerts:

1 for Unused import

Dropped unused import

syslog spewing errors, not able to test properly. will approve pending further testing

K8s control plane had issue with 1.21.1, which is now fixed, hence back to 1.21.1

…7469) Why I did it The SONiC switches get their docker images from local repo, populated during install with container images pre-built into SONiC FW. With the introduction of kubernetes, new docker images available in remote repo could be deployed. This requires dockerd to be able to pull images from remote repo. Depending on the Switch network domain & config, it may or may not be able to reach the remote repo. In the case where remote repo is unreachable, we could potentially make Kubernetes server to also act as http-proxy. How I did it When admin explicitly enables, the kubernetes-server could be configured as docker-proxy. But any update to docker-proxy has to be via service-conf file environment variable, implying a "service restart docker" is required. But restart of dockerd is vey expensive, as it would restarts all dockers, including database docker. To avoid dockerd restart, pre-configure an http_proxy using an unused IP. When k8s server is enabled to act as http-proxy, an IP table entry would be created to direct all traffic to the configured-unused-proxy-ip to the kubernetes-master IP. This way any update to Kubernetes master config would be just manipulating IPTables, which will be transparent to all modules, until dockerd needs to download from remote repo. How to verify it Configure a switch such that image repo is unreachable Pre-configure dockerd with http_proxy.conf using an unused IP (e.g. 172.16.1.1) Update ctrmgrd.service to invoke ctrmgrd.py with "-p" option. Configure a k8s server, and deploy an image for feature with set_owner="kube" Check if switch could successfully download the image or not.

… docker (#7469)" This reverts commit f7ed82f.

… docker (#7469)" (#8023) This change causes nightly test to fail due to the fake proxy IP is not reachable. Reverts #7469 This reverts commit f7ed82f.

… docker (#7469)" This reverts commit e851a42.

…roxy for docker (#7469)" (#8023)" This reverts commit 7236fa9.

…roxy for docker (#7469)" (#8023)" (#8158) This reverts commit 7236fa9. Restore original PR #7469

…roxy for docker (sonic-net#7469)" (sonic-net#8023)" (sonic-net#8158) This reverts commit 7236fa9. Restore original PR sonic-net#7469

…onic-net#7469) Why I did it The SONiC switches get their docker images from local repo, populated during install with container images pre-built into SONiC FW. With the introduction of kubernetes, new docker images available in remote repo could be deployed. This requires dockerd to be able to pull images from remote repo. Depending on the Switch network domain & config, it may or may not be able to reach the remote repo. In the case where remote repo is unreachable, we could potentially make Kubernetes server to also act as http-proxy. How I did it When admin explicitly enables, the kubernetes-server could be configured as docker-proxy. But any update to docker-proxy has to be via service-conf file environment variable, implying a "service restart docker" is required. But restart of dockerd is vey expensive, as it would restarts all dockers, including database docker. To avoid dockerd restart, pre-configure an http_proxy using an unused IP. When k8s server is enabled to act as http-proxy, an IP table entry would be created to direct all traffic to the configured-unused-proxy-ip to the kubernetes-master IP. This way any update to Kubernetes master config would be just manipulating IPTables, which will be transparent to all modules, until dockerd needs to download from remote repo. How to verify it Configure a switch such that image repo is unreachable Pre-configure dockerd with http_proxy.conf using an unused IP (e.g. 172.16.1.1) Update ctrmgrd.service to invoke ctrmgrd.py with "-p" option. Configure a k8s server, and deploy an image for feature with set_owner="kube" Check if switch could successfully download the image or not.

… docker (sonic-net#7469)" (sonic-net#8023) This change causes nightly test to fail due to the fake proxy IP is not reachable. Reverts sonic-net#7469 This reverts commit f7ed82f.

…roxy for docker (sonic-net#7469)" (sonic-net#8023)" (sonic-net#8158) This reverts commit 7236fa9. Restore original PR sonic-net#7469

renukamanavalan added Enhancement ➕ Request for 202012 Branch labels Apr 28, 2021

renukamanavalan requested review from isabelmsft, lguohan and qiluo-msft April 28, 2021 22:44

renukamanavalan self-assigned this Apr 28, 2021

renukamanavalan added 3 commits April 29, 2021 03:09

Fix lgtm warnings.

08d6e71

Fix syntax error

c34b9f2

convert bytecode to string, before parsing.

0380849

W/o this fix, it throws exception