Breaks JSON package in Python 3.7 on Arch linux

[MichaelGoodale]

No idea why this is the case, but importing parselmouth has caused the JSON library to stop working, I assume due to some sort of reassignment of json.load or something similar.

Minimal case:

import json
import parselmouth
with open("main_survey", "r") as f:
      survey = json.load(f)

crashes, while if I run it without importing parselmouth it works fine.

output

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    survey = json.load(f)
  File "/usr/lib/python3.7/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/lib/python3.7/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4486: ordinal not in range(128)

[YannickJadoul]

Hi @MichaelGoodale, thanks for reporting!

This should not be happening; especially because I am not using or even importing the json standard library in Parselmouth.

However, luckily, I managed to reproduce it (for a file with non-ascii characters), so I'll start investigating which parts of Parselmouth are causing this. I'll get back to you once I know more ;-)

[YannickJadoul]

Updates:

• It's not something specific to the json module; calling open("main_survey", "r").read() also fails
• The same problem also occurs on Python 3.6, so it's not 3.7-specific
• I've traced down the problem to the initialization of Praat. Without that initialization, importing Parselmouth does not change anything.

[YannickJadoul]

@MichaelGoodale A temporary workaround should be adding the encoding='utf-8' keyword argument to your call to open, by the way.

[Thooto]

Hi,

Edit: didn't see you already traced the bug. Good luck with fixing it ! :)

Previous message:

I got a similar problem using tempfile.NamedTemporaryFile. When Parselmouth is imported in my project, calling os.popen(some subprocess).read() with the file opened throws the same error. Setting the encoding to utf-8 doesn't solve the issue ...

The code:

with tempfile.NamedTemporaryFile(mode='w+t', encoding='utf-8') as temp:
    temp.write(args.text)
    temp.seek(0)
    process = os.popen('java -jar utils/languagetool/languagetool-commandline.jar --encoding utf-8 --language fr --json %s' % temp.name)
    analysis = process.read()
    process.close()

Various workarounds don't seem to solve the issue (adding .encode(...), .decode(...), using another method ...).

[YannickJadoul]

@Thooto Thanks for adding this to the issue; I'm working on it for the moment, but it's a bit more complicated than I'd hoped for, involving locales and locale-independent reading/writing of doubles :-(

It's the call to process.read() on the result of os.popen(...) that seems to be the problem - I can reproduce this with os.popen('ls').read(). One temporary workaround, until this gets hopefully fixed soon, is to use the subprocess library where you can either set the encoding or use .decode on the returned string.

One thing to be aware of, is that if Parselmouth breaks this code because of Praat setting the locales, your code can be broken when used in other applications as well.

[Thooto]

Thanks for your quick answer ! I'll try to use the subprocess module instead.

I'm actually using a Flask app running in a docker container, where I had both LC_ALL and LANG "env" set to C.UTF-8 as required by Flask. I'll try loading them later on with the other env vars.

Edit: didn't change anything 🤣 (moving these vars from Dockerfile to .env file)

[YannickJadoul]

Didn't work?? Sigh, I hate locales, by now.

The other temporary workaround hack (that works on my machine; but in this case, that's no guarantee, it seems) is calling import locale and locale.setlocale(locale.LC_ALL, "C.UTF-8") after the first import of Parselmouth. As long as you keep the "C" part, the internal Praat functionality shouldn't break.

[Thooto]

I was a bit unclear, sorry ! I didn't try to use subprocess yet, I just changed where I was setting up the LC_ALL and LANG variables (which were set in the Dockerfile, although I'm loading my private env vars from a .env file) but I think it was a kinda stupid move anyway (I was short on time haha, didn't think that through).

However, the locale workaround seemed to do did the trick for me ! Thanks a lot 😃

[YannickJadoul]

Fixed in version 0.3.3, that I've finally managed to release. Sorry for the delay; it was a horrible job to get rid of those setlocale calls, but it should be working now?

Thank you very much for reporting!