'Remote Init failed' for LFRic xc40 tasks

[mcdalvi]

Hello,

Since yesterday (Tuesday) evening all my xc40 tasks under the lfric_apps rose stem suite are failing with errors like (from job-activity.log):

[jobs-submit cmd] (init xcfl01)
[jobs-submit ret_code] 1
[jobs-submit err] REMOTE INIT FAILED
[jobs-submit cmd] (remote init)
[jobs-submit ret_code] 1

This is happening for both xce as well as xcf.
An UM rose stem run submitted at the same time has completed all its XC40 tasks successfully .

I have seen these errors before in a cylc-8 suite, but usually a re-submit fixes the issue. I am not sure if this is a cylc-8 specific or lfric-specific command being run.

Thanks

[mcdalvi]

Fixed
It turned out that 'cylc clean' had not removed the workflow from $SCRATCH on the XC systems. After removing the folder manually and another cylc clean to confirm, the xc40 tasks are now running.