Skip to content

Warm reboot: Add support for docker upgrade#292

Merged
qiluo-msft merged 7 commits intosonic-net:masterfrom
jipanyang:warm_reboot_collab_8_docker_upgrade
Oct 3, 2018
Merged

Warm reboot: Add support for docker upgrade#292
qiluo-msft merged 7 commits intosonic-net:masterfrom
jipanyang:warm_reboot_collab_8_docker_upgrade

Conversation

@jipanyang
Copy link
Copy Markdown
Contributor

@jipanyang jipanyang commented Aug 3, 2018

Signed-off-by: Jipan Yang [email protected]

- What I did
Add sonic_installer upgrade_docker command to support docker upgrade

- How I did it

- How to verify it

sonic_installer upgrade_docker

root@sonic:~# sonic_installer  upgrade_docker  --help
Usage: sonic_installer upgrade_docker [OPTIONS] <container_name> URL

  Upgrade docker image from local binary or URL

Options:
  -y, --yes
  --cleanup_image  Clean up old docker image
  --enforce_check  Enforce pending task check for docker upgrade
  --tag TEXT       Tag for the new docker image
  --help           Show this message and exit.
root@sonic:~# 

Before docker upgrade

root@sonic:~# docker images
REPOSITORY                 TAG                                   IMAGE ID            CREATED             SIZE
docker-orchagent-brcm      latest                                c5f3705e554d        7 hours ago         368.2 MB
docker-orchagent-brcm      test_02                               c5f3705e554d        7 hours ago         368.2 MB
docker-snmp-sv2            latest                                57ad52ea394d        3 weeks ago         373.4 MB
docker-snmp-sv2            warm-reboot.0-dirty-20180823.192608   57ad52ea394d        3 weeks ago         373.4 MB
docker-fpm-quagga          latest                                c68b681610b1        3 weeks ago         380.6 MB
docker-fpm-quagga          warm-reboot.0-dirty-20180823.192608   c68b681610b1        3 weeks ago         380.6 MB
docker-lldp-sv2            latest                                8ecc6ab3e0b7        3 weeks ago         332.4 MB
docker-lldp-sv2            warm-reboot.0-dirty-20180823.192608   8ecc6ab3e0b7        3 weeks ago         332.4 MB
docker-platform-monitor    latest                                e3fea372c435        3 weeks ago         335.7 MB
docker-platform-monitor    warm-reboot.0-dirty-20180823.192608   e3fea372c435        3 weeks ago         335.7 MB
docker-syncd-brcm          latest                                dd8cec44739a        3 weeks ago         410.5 MB
docker-syncd-brcm          warm-reboot.0-dirty-20180823.192608   dd8cec44739a        3 weeks ago         410.5 MB
docker-database            latest                                12f8549a03d0        3 weeks ago         298.5 MB
docker-database            warm-reboot.0-dirty-20180823.192608   12f8549a03d0        3 weeks ago         298.5 MB
docker-sonic-telemetry     latest                                4ee9aeaebcb5        3 weeks ago         324.8 MB
docker-sonic-telemetry     warm-reboot.0-dirty-20180823.192608   4ee9aeaebcb5        3 weeks ago         324.8 MB
docker-dhcp-relay          latest                                1f81836482bf        3 weeks ago         301.6 MB
docker-dhcp-relay          warm-reboot.0-dirty-20180823.192608   1f81836482bf        3 weeks ago         301.6 MB
docker-router-advertiser   latest                                48e9b8ac32b0        3 weeks ago         296.1 MB
docker-router-advertiser   warm-reboot.0-dirty-20180823.192608   48e9b8ac32b0        3 weeks ago         296.1 MB
docker-teamd               latest                                27d6fb7bec52        3 weeks ago         362.3 MB
docker-teamd               warm-reboot.0-dirty-20180823.192608   27d6fb7bec52        3 weeks ago         362.3 MB
root@sonic:~# 

docker upgrade

root@sonic:~# 
root@sonic:~# sonic_installer upgrade_docker  --enforce_check  --cleanup_image swss ./docker-orchagent-brcm_test_02.gz --tag=test_02
New docker image will be installed, continue? [y/N]: y
Orchagent is in clean state and frozen for warm upgrade
Command: systemctl stop swss

Command: docker rm  swss 
swss

Command: docker rmi  docker-orchagent-brcm:latest 
Untagged: docker-orchagent-brcm:latest

Command: docker load < ././docker-orchagent-brcm_test_02.gz

Command: docker tag docker-orchagent-brcm:latest docker-orchagent-brcm:test_02

Command: systemctl restart swss

Command: sleep 5

Done
root@sonic:~# docker images
REPOSITORY                 TAG                                   IMAGE ID            CREATED             SIZE
docker-orchagent-brcm      latest                                c5f3705e554d        7 hours ago         368.2 MB
docker-orchagent-brcm      test_02                               c5f3705e554d        7 hours ago         368.2 MB
docker-snmp-sv2            latest                                57ad52ea394d        3 weeks ago         373.4 MB
docker-snmp-sv2            warm-reboot.0-dirty-20180823.192608   57ad52ea394d        3 weeks ago         373.4 MB
docker-fpm-quagga          latest                                c68b681610b1        3 weeks ago         380.6 MB
docker-fpm-quagga          warm-reboot.0-dirty-20180823.192608   c68b681610b1        3 weeks ago         380.6 MB
docker-lldp-sv2            latest                                8ecc6ab3e0b7        3 weeks ago         332.4 MB
docker-lldp-sv2            warm-reboot.0-dirty-20180823.192608   8ecc6ab3e0b7        3 weeks ago         332.4 MB
docker-platform-monitor    latest                                e3fea372c435        3 weeks ago         335.7 MB
docker-platform-monitor    warm-reboot.0-dirty-20180823.192608   e3fea372c435        3 weeks ago         335.7 MB
docker-syncd-brcm          latest                                dd8cec44739a        3 weeks ago         410.5 MB
docker-syncd-brcm          warm-reboot.0-dirty-20180823.192608   dd8cec44739a        3 weeks ago         410.5 MB
docker-database            latest                                12f8549a03d0        3 weeks ago         298.5 MB
docker-database            warm-reboot.0-dirty-20180823.192608   12f8549a03d0        3 weeks ago         298.5 MB
docker-sonic-telemetry     latest                                4ee9aeaebcb5        3 weeks ago         324.8 MB
docker-sonic-telemetry     warm-reboot.0-dirty-20180823.192608   4ee9aeaebcb5        3 weeks ago         324.8 MB
docker-dhcp-relay          latest                                1f81836482bf        3 weeks ago         301.6 MB
docker-dhcp-relay          warm-reboot.0-dirty-20180823.192608   1f81836482bf        3 weeks ago         301.6 MB
docker-router-advertiser   latest                                48e9b8ac32b0        3 weeks ago         296.1 MB
docker-router-advertiser   warm-reboot.0-dirty-20180823.192608   48e9b8ac32b0        3 weeks ago         296.1 MB
docker-teamd               latest                                27d6fb7bec52        3 weeks ago         362.3 MB
docker-teamd               warm-reboot.0-dirty-20180823.192608   27d6fb7bec52        3 weeks ago         362.3 MB
root@sonic:~# 
root@sonic:~# sonic_installer upgrade_docker  --enforce_check  --cleanup_image swss ./docker-orchagent-brcm_test_02.gz              
New docker image will be installed, continue? [y/N]: y
Orchagent is in clean state and frozen for warm upgrade
Command: systemctl stop swss

Command: docker rm  swss 
swss

Command: docker rmi  docker-orchagent-brcm:latest 
Untagged: docker-orchagent-brcm:latest

Command: docker load < ././docker-orchagent-brcm_test_02.gz

Command: docker tag docker-orchagent-brcm:latest docker-orchagent-brcm:unknown

Command: systemctl restart swss

Command: sleep 5

Done
root@sonic:~# docker images
REPOSITORY                 TAG                                   IMAGE ID            CREATED             SIZE
docker-orchagent-brcm      latest                                c5f3705e554d        7 hours ago         368.2 MB
docker-orchagent-brcm      test_02                               c5f3705e554d        7 hours ago         368.2 MB
docker-orchagent-brcm      unknown                               c5f3705e554d        7 hours ago         368.2 MB
docker-snmp-sv2            latest                                57ad52ea394d        3 weeks ago         373.4 MB
docker-snmp-sv2            warm-reboot.0-dirty-20180823.192608   57ad52ea394d        3 weeks ago         373.4 MB
docker-fpm-quagga          latest                                c68b681610b1        3 weeks ago         380.6 MB
docker-fpm-quagga          warm-reboot.0-dirty-20180823.192608   c68b681610b1        3 weeks ago         380.6 MB
docker-lldp-sv2            latest                                8ecc6ab3e0b7        3 weeks ago         332.4 MB
docker-lldp-sv2            warm-reboot.0-dirty-20180823.192608   8ecc6ab3e0b7        3 weeks ago         332.4 MB
docker-platform-monitor    latest                                e3fea372c435        3 weeks ago         335.7 MB
docker-platform-monitor    warm-reboot.0-dirty-20180823.192608   e3fea372c435        3 weeks ago         335.7 MB
docker-syncd-brcm          latest                                dd8cec44739a        3 weeks ago         410.5 MB
docker-syncd-brcm          warm-reboot.0-dirty-20180823.192608   dd8cec44739a        3 weeks ago         410.5 MB
docker-database            latest                                12f8549a03d0        3 weeks ago         298.5 MB
docker-database            warm-reboot.0-dirty-20180823.192608   12f8549a03d0        3 weeks ago         298.5 MB
docker-sonic-telemetry     latest                                4ee9aeaebcb5        3 weeks ago         324.8 MB
docker-sonic-telemetry     warm-reboot.0-dirty-20180823.192608   4ee9aeaebcb5        3 weeks ago         324.8 MB
docker-dhcp-relay          latest                                1f81836482bf        3 weeks ago         301.6 MB
docker-dhcp-relay          warm-reboot.0-dirty-20180823.192608   1f81836482bf        3 weeks ago         301.6 MB
docker-router-advertiser   latest                                48e9b8ac32b0        3 weeks ago         296.1 MB
docker-router-advertiser   warm-reboot.0-dirty-20180823.192608   48e9b8ac32b0        3 weeks ago         296.1 MB
docker-teamd               latest                                27d6fb7bec52        3 weeks ago         362.3 MB
docker-teamd               warm-reboot.0-dirty-20180823.192608   27d6fb7bec52        3 weeks ago         362.3 MB
root@sonic:~# 
root@sonic:~# sonic_installer upgrade_docker  --enforce_check  --cleanup_image swss ./docker-orchagent-brcm_test_01.gz --tag=test_01
New docker image will be installed, continue? [y/N]: y
Orchagent is in clean state and frozen for warm upgrade
Command: systemctl stop swss

Command: docker rm  swss 
swss

Command: docker rmi  docker-orchagent-brcm:latest 
Untagged: docker-orchagent-brcm:latest

Command: docker load < ././docker-orchagent-brcm_test_01.gz

Command: docker tag docker-orchagent-brcm:latest docker-orchagent-brcm:test_01

Command: systemctl restart swss

Command: docker rmi -f c5f3705e554d
Untagged: docker-orchagent-brcm:test_02
Untagged: docker-orchagent-brcm:unknown
Deleted: sha256:c5f3705e554df1f8cab6eefc048364a1838706d6b6d21a69e72ac2b33c5d5799
Deleted: sha256:e116c136985f1faaa2881f6ca2604e92d5ce12ad3eb7ea7078d1a12d060c6512

Command: sleep 5

Done
root@sonic:~# 

- Previous command output (if the output of a command-line utility has changed)

- New command output (if the output of a command-line utility has changed)

Signed-off-by: Jipan Yang <[email protected]>
@click.option('-y', '--yes', is_flag=True, callback=abort_if_false,
expose_value=False, prompt='New docker image will be installed, continue?')
@click.option('--cleanup_image', is_flag=True, help="Clean up old docker images")
@click.argument('container_name', metavar='<container_name>', required=True, type=click.Choice(["swss", "snmp", "lldp"]))
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add bgp, pmon, dhcp_relay,telemetry,teamd?


if url.startswith('http://') or url.startswith('https://'):
click.echo('Downloading image...')
urllib.urlretrieve(url, DEFAULT_IMAGE_PATH, reporthook)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should handle failure cases of this API call here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

cmd = "docker exec -it " + container_name + " orchagent_restart_check"
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
result = proc.stdout.read().rstrip()
if result != "RESTARTCHECK succeeded":
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this PR is relying on sonic-net/sonic-swss#562

Can we check the orchagent_restart_check return code instead of string?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be changed to check return code. What is the benefit of doing that?
The concern here is that, theoretically, the success of running orchagent_restart_check program itself doesn't indicate the warm restart readiness of orchagent.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The benefit is if any result string from "orchagent_restart_check" changed for some reason, we don't have to change code here.
orchagent_restart_check should return EXIT_SUCCESS code if check is successful and return EXIT_FAILURE if check is failed or some Error code if the program is not running due to other reasons.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both the result string and exit code are from orchagent_restart_check, but exit code may also indicate status of orchagent_restart_check itself. The result strings are designed for the result check purposely. Anyway, don't want to spend too much time debating this kind of issues. Either one works for now, will change to use exit code to settle the issue.

@zhenggen-xu
Copy link
Copy Markdown
Collaborator

It would be nice if we can add the docker rollback CLI if the previous images was not removed from the system.

Copy link
Copy Markdown
Contributor Author

@jipanyang jipanyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, rollback CLI is good. Will focus on docker upgrade for this PR.


if url.startswith('http://') or url.startswith('https://'):
click.echo('Downloading image...')
urllib.urlretrieve(url, DEFAULT_IMAGE_PATH, reporthook)
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

cmd = "docker exec -it " + container_name + " orchagent_restart_check"
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
result = proc.stdout.read().rstrip()
if result != "RESTARTCHECK succeeded":
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be changed to check return code. What is the benefit of doing that?
The concern here is that, theoretically, the success of running orchagent_restart_check program itself doesn't indicate the warm restart readiness of orchagent.

run_command("docker load < %s" % image_path)
if tag == None:
# example image: docker-lldp-sv2:latest
tag = get_docker_tag_name(image_latest)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to check return tag value, e,g, if it is "unknown" and then we do things differently here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some tag is needed. If no tag is provided, default is "unknown". Before tag is built into docker meta label, I don't want to enforce the existence of tag.

@jipanyang
Copy link
Copy Markdown
Contributor Author

@jleveque do you have more comments for the changes?

Signed-off-by: Jipan Yang <[email protected]>
@qiluo-msft qiluo-msft merged commit e2b4bd5 into sonic-net:master Oct 3, 2018
mihirpat1 pushed a commit to mihirpat1/sonic-utilities that referenced this pull request Sep 15, 2023
Description
Add unit test cases to test eeprom_tlvinfo.py, now code coverage is 80%

Motivation and Context
There is no UT available for eeprom_tlvinfo.py previously.
Using a HEX file to mock EEPROM content. Take this mocked EEPROM as input and test the functions.

Signed-off-by: Kebo Liu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants