spaCy 1.7 model downloads consume large amounts of memory

[schmod]

When downloading large models such as en_core_web_md, spaCy's built-in downloader appears to consume very large quantities of memory (often greatly exceeding the size of the model itself). On less-powerful systems, this can result in a failure if the process consumes all available memory.

This issue can be replicated by running python -m spacy download en_core_web_md, and inspecting memory usage.

This problem does not exist when installing models via pip or a direct download (as those processes presumably stream the downloaded file directly to disk.

Your Environment

• Operating System: Debian Jessie
• Python Version Used: 2.7
• spaCy Version Used: 1.7.2
• Environment Information: Docker

[honnibal]

Hey,

This is fixed on master --- it's a known problem with pip. The workaround is to specify --no-cache-dir. So if you point pip at the file and set that argument, it should work. I expect we'll have a new point release over the weekend.

Matt

[ines]

Fixed!

[schmod]

Thanks!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.