Reproducibility info is lossy due to space joining

[phiresky]

Came across this issue when trying to rerun an older command:

for a cmdline like python plot.py --title "Foo Bar" ... the output is python plot.py --title Foo Bar

which is not distinguishable from having separate arguments. In the above cmdline it's easy to see but in my case the command_lines are longer and I might want to parse the cmdline programmatically after the fact

The related line is in:

typed-argument-parser/tap/tap.py

Line 353 in f30ffbe

'command_line': f'python {" ".join(sys.argv)}',

I'd propose to just directly output sys.argv as an arry instead of joining it.

[swansonk14]

Hi @phiresky,

Thank you for bringing this issue to our attention! We think that it's very convenient to have the command_line as a string so that it can be copied and pasted to reproduce the original command, so we're thinking of keeping it as a string rather than as an array.

However, we agree that the lack of quotes in arguments with whitespace is a problem, so we made a fix in this commit 8bc0861 which automatically adds those quotes. Now commands that include args with spaces can be copied and pasted correctly.

Unfortunately, this solution doesn't work for other types of whitespace such as tabs or newlines. This is because JSON always writes tabs and newlines as \t and \n, which are not parsed correctly on the command line. However, it's relatively straightforward for users to notice this problem and manually correct those escape characters to the tab or newline characters.

Please let us know if you have any ideas for improving this! And thanks again for bringing up the issue!

Best,
Kyle and Jesse

[phiresky]

I think your solution is still kinda wrong because it doesn't escape '. for example ['hello', 'te st\' foobar']
maybe it should use shlex.quote() or shlex.join? https://docs.python.org/3/library/shlex.html#shlex.quote
Then it could be parsed with shlex.parse

[martinjm97]

Hi @phiresky,

We didn't know about shlex.quote. Thanks for pointing it out.

It looks like using shlex.quote still doesn't solve our problem though. The process of writing to json introduces additional escaping that, in the same way as our previous solution, doesn't work on the command line. For example, for

python main.py --arg 'te st" foobar'

the reproducability information in the saved file is:

"command_line": "python main.py --arg 'te st\" foobar'"

We don't think that there's a way to get around the double escaping problem. Do you have any ideas about how to avoid this problem?

Best,
Kyle and Jesse

[cardoso-neto]

When I need to wrap my str instances with quotes, I usually just call their __repr__ which adds a singlequote on either end.
e.g., after word = "foo bar" both f"{word!r}" and f"{repr(word)}" work for outputting 'foo bar'.
Hope this is helpful to you guys as singlequotes won't need special escaping inside .jsons.

[martinjm97]

We believe that this issue is resolved. We're happy to reopen it if we are mistaken.

--Kyle and Jesse

[cardoso-neto]

You are correct. It's fixed.
--test some 'one two' three "four five" perfectly becomes ['some', 'one two', 'three', 'four five'] as expected.
Great work, guys.