[ptf] Support alt_password for PTF to DUT SSH connection

[vaibhavhd]

Description of PR

Summary: In reboot/upgrades, the PTF's SSH key stored inside DUT gets deleted. Use a retry mechanism (alt_password) when default admin password does not work.
Fixes # (issue)

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Approach

What is the motivation for this PR?

For tests running on PTF container, alt password is not supported.
Use case - To enable alt_password for the tests running in PTF where there is a reboot/upgrade leading to deletion of SSH key and known hosts file inside DUT.
Note that the tests running on sonic-mgmt container have SSH retry support for device password and alt_password.

The error seen when key is deleted and default password does not work:

2020-12-15 00:47:24 : --------------------------------------------------
2020-12-15 00:47:24 : --------------------------------------------------
2020-12-15 00:47:24 : Fails:
2020-12-15 00:47:24 : --------------------------------------------------
2020-12-15 00:47:24 : FAILED:dut:Sonic upgrade failed. Target=SONiC-OS-master.586-e010d83f Current=
2020-12-15 00:47:24 : FAILED:dut:Longest downtime period must be less then 30 seconds. It was 0:02:48.229600
2020-12-15 00:47:24 : ==================================================
2020-12-15 00:47:24 : Disabling arp_responder

00:47:20.692  paramiko.transport: INFO    : Connected (version 2.0, client OpenSSH_7.9p1)
00:47:20.775  paramiko.transport: INFO    : Authentication (publickey) failed.
00:47:24.268  paramiko.transport: INFO    : Authentication (password) failed.
00:47:24.293  device_connection: ERROR   : SSH Command failed with message: Authentication failed.
00:47:25.917  root      : INFO    : *** TEST RUN END  : Tue Dec 15 00:47:25 2020
00:47:25.917  dataplane : INFO    : Thread exit

How did you do it?

Added retry mechanism when AuthenticationException is hit when trying the default password.
Fixed the exception catch sequence leading to auth errors being caught by SSHException.

Fixed an import error and typo.

How did you verify/test it?

Tested with the fix:

-------------------------------------------------------------------------------- live log call ---------------------------------------------------------------------------------
03:03:39 INFO test_upgrade_path.py:test_upgrade_path:156: Test upgrade path from https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin to https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.bin
03:03:39 INFO test_upgrade_path.py:test_upgrade_path:158: Installing https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/580/artifact/target/sonic-vs.bin
03:04:53 INFO reboot.py:reboot:121: waiting for ssh to drop
03:04:53 INFO reboot.py:execute_reboot_command:105: rebooting with command "reboot"
03:05:03 INFO reboot.py:reboot:139: waiting for ssh to startup
03:06:09 INFO reboot.py:reboot:150: ssh has started up
03:06:09 INFO reboot.py:reboot:152: waiting for switch to initialize
03:08:09 INFO reboot.py:reboot:180: cold reboot finished
03:08:13 INFO reboot.py:reboot:183: DUT up since 2020-12-15 03:05:38
03:08:14 INFO test_upgrade_path.py:test_upgrade_path:165: Upgrading to https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.bin
03:09:29 INFO ssh_utils.py:prepare_testbed_ssh_keys:17: Remove old keys from ptfhost
03:09:33 INFO ssh_utils.py:prepare_testbed_ssh_keys:26: Generate public key for ptf host
PASSED                                                                                                                                                                   [100%]
------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------

From paramiko log, the 2nd attempt at login with alt_password is successful

03:14:28.850  paramiko.transport: INFO    : Connected (version 2.0, client OpenSSH_7.9p1)
03:14:28.927  paramiko.transport: INFO    : Authentication (publickey) failed.
03:14:32.421  paramiko.transport: INFO    : Authentication (password) failed.
03:14:32.447  device_connection: ERROR   : SSH Authentication failure with message: Authentication failed.
03:14:32.576  paramiko.transport: INFO    : Connected (version 2.0, client OpenSSH_7.9p1)
03:14:32.648  paramiko.transport: INFO    : Authentication (publickey) failed.
03:14:32.682  paramiko.transport: INFO    : Authentication (password) successful!

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation