PMON container crashes in latest SONiC images

[ciju-juniper]

Description
Seeing issues with 'pmon' container startup and the following error in the syslog. All the platform monitoring daemons are killed and pmon also stopped after a few trials

Nov 19 18:48:15.770882 sonic INFO pmon#supervisord: start sonic-platform package not installed, attempting to install...
Nov 19 18:48:15.770882 sonic INFO pmon#supervisord: start Error: Unable to locate /usr/share/sonic/platform/sonic_platform-1.0-py2-none-any.whl
Nov 19 18:48:16.060439 sonic INFO pmon#supervisord: start sonic-platform package not installed, attempting to install...
Nov 19 18:48:16.060439 sonic INFO pmon#supervisord: start Error: Unable to locate /usr/share/sonic/platform/sonic_platform-1.0-py3-none-any.whl
Nov 19 18:48:16.133494 sonic INFO pmon#supervisord: xcvrd Traceback (most recent call last):
Nov 19 18:48:16.133494 sonic INFO pmon#supervisord: xcvrd   File "/usr/local/bin/xcvrd", line 5, in <module>
Nov 19 18:48:16.133494 sonic INFO pmon#supervisord: xcvrd     from src.xcvrd import main
Nov 19 18:48:16.133494 sonic INFO pmon#supervisord: xcvrd ImportError: No module named src.xcvrd
Nov 19 18:48:16.142902 sonic INFO pmon#supervisor-proc-exit-listener: Process xcvrd exited unxepectedly. Terminating supervisor...

Initial Triage
The last good build on the master branch was build# 481 and pmon crashes are seen from build# 482 onwards. In between, there are a few commits in which I suspect the following commit introduced the breakage:

[submodule]: update sonic-platform-daemons (#5868) (detail / githubweb)

Platform details
This problem should be there in most of the platforms. I had tested it on Juniper QFX5210 & QFX5200 platforms.

root@sonic:~# show version

SONiC Software Version: SONiC.master.482-aee389e4
Distribution: Debian 10.6
Kernel: 4.19.0-9-2-amd64
Build commit: aee389e4
Build date: Wed Nov 11 06:51:47 UTC 2020
Built by: johnar@jenkins-worker-8

Platform: x86_64-juniper_qfx5200-r0
HwSKU: Juniper-QFX5200-32C-S
ASIC: broadcom
Serial Number: ZA0220160436
Uptime: 17:52:40 up 2 min,  1 user,  load average: 1.76, 0.66, 0.24

show techsupport
Here is the techsupport archive:
sonic_dump_sonic_20201120_175417.tar.gz

@vdahiya12 Could you please take a look? Please let me know if you need any further details. Also, please suggest if there are any platform side changes required after this PR #5868

[vdahiya12]

#5924 this PR should fix the pmon and is already in master.
is this on latest master ? #5924

[ciju-juniper]

@vdahiya12 The last image I tried was build# 493. Pmon was crashing on this.

[vdahiya12]

can u please try 496 as its also built

[ciju-juniper]

@vdahiya12 Downloading build# 496. Once it's complete, I will load it and share the results.

[ciju-juniper]

@vdahiya12 PMON crash is seen in build# 496 also. Please let me know which image will have your fix.

[yxieca]

While this issue might have been fixed. the latest master image has a different issue with xcvrd as in issue #5994

[ciju-juniper]

@vdahiya12 @lguohan @jleveque @yxieca I tried build# 500. Most of the services are down. Still seeing the following error

Nov 23 13:23:42.710367 sonic INFO pmon#/supervisord: start sonic-platform package not installed, attempting to install...
Nov 23 13:23:42.710367 sonic INFO pmon#/supervisord: start Error: Unable to locate /usr/share/sonic/platform/sonic_platform-1.0-py2-none-any.whl
Nov 23 13:23:43.041006 sonic INFO pmon#/supervisord: start sonic-platform package not installed, attempting to install...
Nov 23 13:23:43.041156 sonic INFO pmon#/supervisord: start Error: Unable to locate /usr/share/sonic/platform/sonic_platform-1.0-py3-none-any.whl
Nov 23 13:23:43.314650 sonic INFO pmon#/supervisord: xcvrd Traceback (most recent call last):
Nov 23 13:23:43.314650 sonic INFO pmon#/supervisord: xcvrd   File "/usr/local/bin/xcvrd", line 5, in <module>
Nov 23 13:23:43.314650 sonic INFO pmon#/supervisord: xcvrd     from xcvrd.xcvrd import main
Nov 23 13:23:43.314650 sonic INFO pmon#/supervisord: xcvrd   File "/usr/local/lib/python2.7/dist-packages/xcvrd/xcvrd.py", line 26, in <module>
Nov 23 13:23:43.315162 sonic INFO pmon#/supervisord: xcvrd     raise ImportError (str(e) + " - required module not found")
Nov 23 13:23:43.315162 sonic INFO pmon#/supervisord: xcvrd ImportError: No module named sonic_platform.platform - required module not found - required module not found - required module not found

Also seeing the errors from eepromd & psud
Nov 23 13:24:22.052834 sonic WARNING pmon#syseepromd[29]: Failed to load platform-specific eeprom from sonic_platform package due to ImportError('No module named sonic_platform',)
Nov 23 13:24:22.058782 sonic WARNING pmon#psud[28]: Failed to load chassis due to ImportError('No module named sonic_platform.platform',)

Also seeing an orchagent crash:

Nov 23 13:29:58.513505 sonic ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on notify
Nov 23 13:29:58.513505 sonic ERR swss#orchagent: :- wait: failed to get response for notify
Nov 23 13:29:58.513505 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0

[ciju-juniper]

Here is the dump from build# 500:
sonic_dump_sonic_20201123_133333.tar.gz

[jleveque]

@ciju-juniper: It appears as though your sonic_platform wheel package was not found when the PMon container started up, so it was not installed in the container. Can you please investigate why?

[lguohan]

@jleveque, which package are you referring to. I do not see they build their own wheel package. https://sonic-jenkins.westus2.cloudapp.azure.com/job/broadcom/job/buildimage-brcm-all/lastSuccessfulBuild/artifact/target/debs/buster/sonic-platform-juniper-qfx5210_1.1_amd64.deb.log

[jleveque]

@jleveque, which package are you referring to. I do not see they build their own wheel package. https://sonic-jenkins.westus2.cloudapp.azure.com/job/broadcom/job/buildimage-brcm-all/lastSuccessfulBuild/artifact/target/debs/buster/sonic-platform-juniper-qfx5210_1.1_amd64.deb.log

I see references to files in the sonic_platform package in the log, but I don't see a sonic_platform-1.0-py2-none-any.whl package being built.

[ciju-juniper]

@jleveque I started seeing that error from build# 482. No platform changes are involved.

[ciju-juniper]

@jleveque Moreover, the images are built by jenkins jobs. Were there any changes in the package selection / build rules from build# 482 onwards?