Slow treatment of very long lines

[JulienPalard]

Hi,

Did not took the time to fully investigate, but running pycodestyle --exclude=.git --statistics --count . on a clone of github.com/PmagPy/PmagPy looks to never end.

Tested on commit 74a65794 of PmagPy and pycodestyle 2.2.0.

[JulienPalard]

OK I'm trying to diagnose this one now, first hint, the culpit is: https://github.com/PmagPy/PmagPy/blob/master/pmagpy/coefficients.py so probably not an infinite loop, "just" a very long execution time (like more than 8 hours on my laptop if I remember well).

coefficients.py is 6.9MB, 7125673 characters.

I truncated it to 148KB, 141133 characters, and it completes before my patience kill it (So I measured my patience, it's equal to exactly 3mn), runtime for 148K is 0m45.895s to report 9654 E231 missing whitespace after ','

[JulienPalard]

After running cProfile, it looks like 37s out of 42.8s are consumed by noqa.

The "problem" is noqa is called for every token of a line, so for a very long line with a lot of tokens it gets called a lots of time with the whole long line. On a 148KB files I measured more than 3.2GB of strings given to noqa (killed it before the end).

A simple fix may be to use a lru_cache, which sadly isn't available in 2.7 and I don't want to reimplement a lru_cache (errors = more code²) nor copy the cpython one (~100 more lines) I propse here to cache only if lru_cache is available, else just not fix this issue.

On a 148KB file with very long lines, some run times:

• Without lru_cache on noqa: 40.954s, 40.614s, 41.143s
• With lru_cache on noqa: 4.971s, 5.089s, 4.935s