High memory consumption from spyder kernels even after it was closed

[Miguel-LlamasLanza]

Issue Report Checklist

• Searched the issues page for similar reports
• Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
• Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
• Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

My computer had a high memory consumption (about 29 GB out of 33.4) and Swap (nearly 2 out of 2 GB), while not doing an intensive use of it.
Using the command ps aux --sort=-%mem | head I saw the following processes mostly consuming memory (in addition to some others I was using, consuming less):

They are processes from spyder kernels (some started a few days ago), even though Spyder was closed. Perhaps it is related to the fact that during these last days spyder crashed once or twice (but the last time I closed spyder it was a smooth closing). Deactivating the conda environment (where spyder is installed), did not make any change.

Then, I killed these processes from the `top' terminal command, and the memory went down to 11 GB and the Swap.

I am reporting as it might be worth investigating.

Steps to reproduce

I don't know.

Versions

• Spyder version: 5.4.5 (conda)
• Python version: 3.10.11 64-bit
• Qt version: 5.15.2
• PyQt5 version: 5.15.7
• Operating System: Linux-6.8.0-40-generic-x86_64-with-glibc2.35

Dependencies

# Mandatory:
atomicwrites >=1.2.0                                                        :  1.4.1 (OK)
chardet >=2.0.0                                                             :  4.0.0 (OK)
cloudpickle >=0.5.0                                                         :  2.2.1 (OK)
cookiecutter >=1.6.0                                                        :  2.1.1 (OK)
diff_match_patch >=20181111                                                 :  20230430 (OK)
intervaltree >=3.0.2                                                        :  3.1.0 (OK)
IPython >=7.31.1,<9.0.0,!=8.8.0,!=8.9.0,!=8.10.0,!=8.11.0,!=8.12.0,!=8.12.1 :  8.14.0 (OK)
jedi >=0.17.2,<0.19.0                                                       :  0.18.2 (OK)
jellyfish >=0.7                                                             :  0.9.0 (OK)
jsonschema >=3.2.0                                                          :  4.17.3 (OK)
keyring >=17.0.0                                                            :  23.13.1 (OK)
nbconvert >=4.0                                                             :  7.5.0 (OK)
numpydoc >=0.6.0                                                            :  1.5.0 (OK)
parso >=0.7.0,<0.9.0                                                        :  0.8.3 (OK)
pexpect >=4.4.0                                                             :  4.8.0 (OK)
pickleshare >=0.4                                                           :  0.7.5 (OK)
psutil >=5.3                                                                :  5.9.0 (OK)
pygments >=2.0                                                              :  2.15.1 (OK)
pylint >=2.5.0,<3.0                                                         :  2.17.4 (OK)
pylint_venv >=3.0.2                                                         :  3.0.2 (OK)
pyls_spyder >=0.4.0                                                         :  0.4.0 (OK)
pylsp >=1.7.4,<1.8.0                                                        :  1.7.4 (OK)
pylsp_black >=1.2.0,<3.0.0                                                  :  1.3.0 (OK)
qdarkstyle >=3.0.2,<3.2.0                                                   :  3.1 (OK)
qstylizer >=0.2.2                                                           :  0.2.2 (OK)
qtawesome >=1.2.1                                                           :  1.2.3 (OK)
qtconsole >=5.4.2,<5.5.0                                                    :  5.4.3 (OK)
qtpy >=2.1.0                                                                :  2.3.1 (OK)
rtree >=0.9.7                                                               :  1.0.1 (OK)
setuptools >=49.6.0                                                         :  67.8.0 (OK)
sphinx >=0.6.6                                                              :  7.0.1 (OK)
spyder_kernels >=2.4.4,<2.5.0                                               :  2.4.4 (OK)
textdistance >=4.2.0                                                        :  4.5.0 (OK)
three_merge >=0.1.1                                                         :  0.1.1 (OK)
watchdog >=0.10.3                                                           :  3.0.0 (OK)
xdg >=0.26                                                                  :  0.28 (OK)
zmq >=22.1.0                                                                :  25.1.0 (OK)

# Optional:
cython >=0.21                                                               :  0.29.35 (OK)
matplotlib >=3.0.0                                                          :  3.7.1 (OK)
numpy >=1.7                                                                 :  1.23.5 (OK)
pandas >=1.1.1                                                              :  1.5.3 (OK)
scipy >=0.17.0                                                              :  1.10.1 (OK)
sympy >=0.7.3                                                               :  1.11.1 (OK)

# Spyder plugins:
spyder_line_profiler.spyder.plugin 0.3.1                                    :  0.3.1 (OK)
spyder_notebook.notebookplugin 0.4.1                                        :  0.4.1 (OK)

[dalthviz]

Hi @Miguel-LlamasLanza thank you for the report and sorry for such a late response! My guess is that since Spyder crashed was unable to finish the kernel processes running 🤔

Not totally sure what we can do to handle such an scenario to be honest. Maybe add some sort of way to check for such orphan processes when Spyder is launched again and show some sort of warning dialog? Or even maybe try to launch IPython console clients for those still running kernels? What do you think @ccordoba12 ?

[dalthviz]

Note: I was able to reproduce this on Windows. Spyder crashed since I was checking a large dataframe and ended up with a Python process still running even after the Spyder process ended. Also, after relaunching Spyder and using the Consoles > Connect to an existing kernel functionality, I was able to connect to the kernel where I had the dataframe by trying one of the latest .json connection files (the ones available over the ...\jupyter\runtime directory )

[Miguel-LlamasLanza]

Thanks for you answer.
I guess being able to reopen a crashed kernel is a useful functionality. Maybe on Spyder restart, there should be the option to reopen such kernel or close it for good (as it happens for unsaved files).

[ccordoba12]

What do you think @ccordoba12 ?

There's code in ipykernel to kill the kernel when its parent dies (e.g. see this PR ipython/ipykernel#1271). But it's probably not working for us because we use conda run to activate kernels. So, we need to check how to address that.

I guess being able to reopen a crashed kernel is a useful functionality. Maybe on Spyder restart, there should be the option to reopen such kernel or close it for good (as it happens for unsaved files).

That's not easy at all and out of the scope of this problem (which I think is quite serious).