Memory issues when generating voice report in bulk

[kkawabat]

OS: Ubuntu 18.04.4 LTS
python: 3.7.5

I am running the below function in a loop iterating over audio files between 10~60 seconds long.

def calc_voice_report(input_audio):
    sound = parselmouth.Sound(input_audio)
    pitch = sound.to_pitch()
    pulses = parselmouth.praat.call([sound, pitch], "To PointProcess (cc)")
    voice_report_str = \
        parselmouth.praat.call([sound, pitch, pulses], "Voice report", 0.0, 0.0, 75, 600, 1.3, 1.6, 0.03, 0.45)
    return voice_report_str

The resulting string is then parsed and saved into a csv.

After around 3k~8k files I ran into some sort of memory issue, either:

• SIGFAULT 6
• a parselmouth error stating that it's out of memory (did not save the error message will write it down if it happens again next time)
• a computer freeze

I had similar issues running praat voice report in a praat script (using subprocess to call it from python) which makes me think there is a memory leak caused by the voice report module.

Is there a way to fully "clear" parselmouth of it's memory after a parselmouth.praat.call?

[YannickJadoul]

Hi, @kkawabat, this does sound like some memory leak, but I thought Parselmouth cleans up after parselmouth.praat.call. So there's nothing special that you can do to clean this, as far as I know.

To really force garbage collection from the Python side, you can import gc and call gc.collect(). But I don't expect this to help, actually, since this is most likely somewhere on the C/C++ side. So the only rough workaround I can think of (except for me diving into the code and searching for leaks), is to run smaller batches and restart Python (which unloads the C/C++ library and Praat). I do know that that's very annoying and not how it's meant to be, but it's the one solution that will definitely now provide a workaround.

This might take me some time to look into this, though, but I do want to figure out where Parselmouth (or maybe Praat) has this leak. Does this happen consistently? Is this this easiest way you manage to crash things? (Running 3k to 8k files will take a while to reproduce, I expect? :-( )

[kkawabat]

Hi, @YannickJadoul thank you so much for the quick reply. For the work around suggestion, I have been managing this issue by manually doing smaller batch and quitting in between but even 4-8 times is a bit tedious (maybe I'll automate the restarting of python as well 😅).

It happens pretty consistently, and it actually doesn't take too long for my dataset/workflow to reproduce this issue (maybe 5-15 minutes)? I'm not too familiar with profilers but if you can guide me on what would be the most useful information for debugging, I can try and create a memory profiler log for this problem.

[YannickJadoul]

Nono, you're completely right, @kkawabat: this shouldn't happen. The workaround is mainly meant as a way to get your work done with a broken version of Parselmouth ;-) (Maybe something with subprocess could work to automate this?)

5 to 15 minutes is quite long for debugging, though, but if it crashes in that amount of time, I guess I should be able to see some excessively rising memory consumption after 1 or 2 minutes, already :-) I need to check myself what the best way is of profiling the memory. With these C extensions for Python, things tend to be more complex.

However, if you do want to try something, you could try installing the current development version (which has a newer version of Praat, and a few other minor bug fixes). I'd be curious to know if you see the same problem here. To should be able to find the right version here, download the one that matches your system (should be the one with cp37-cp37m-manylinux2010_x86_64 in the file name for Python 3.7 on 64-bit Linux), and pip install the downloaded file.

[YannickJadoul]

Actually. There's a chance this is fixed in the development version. 2103636 fixed a memory leak (in the parselmouth.praat module), but was not yet included in a release, I think :-|

[kkawabat]

Great! I'll be sure to keep an eye out when that comes out.

[YannickJadoul]

Does it work with the development version, then, for you? :-)

(I'll reopen this issue, to be sure to remind myself I should get the release out)

[YannickJadoul]

Finally a 0.4.0 release! If there's still a leak, do please let me know! :-)