Lint successful, but PyPI rendering failed

[douglasrizzo]

I installed rst-lint to verify my readme file after successive failures to render it successfully in PyPI. rst-lint said my file was clean, however, when I uploaded it to PyPI, it did not render the file again.

In order to make a clean .rst file, I had to install a few packages in my computer, like bibtex-pygments-lexer, so that Pygments could recognize and pygmentize BibTeX snippets. In short, I believe rst-lint is not exactly a linter that validates rst files for PyPI, like its README file says. It just validates rst files generally. PyPI still has its own cryptic way to render rst files.

[twolfson]

We have had discussions in the past about supporting more Sphinx directives (see #11, #14, and #26). Can you provide more details on what was breaking and how we can reproduce so we can better decide if it's something to support or not?

[douglasrizzo]

My problem is that, even though rst-lint says my RST files are clean, they are not rendered correctly on PyPI. I know PyPI uses a different compilation process for rst files (for example, it is said it fails when the readme contains relative links), but I can only know for sure after they are sent to PyPI and I see the result in my package's page.
In order to reproduce my problem, use rst-lint on this file. If you install the necessary packages, it will eventually say the file is clean, but just check the result when it is sent to PyPI.
Maybe if there was an option to validate RST files like PyPI does it and not like one's local machine does it...

[twolfson]

Interesting, thanks for the info. iirc our current architecture is based off of rst2html which I believe is what PyPI uses. The initial revisions were to try something silly like run rst2html on a file and see what happens. I will review that information by the end of the weekend.

If we do find that everything is more/less 1:1 with PyPI except for some flags, it might be saner to either (a) provide overridable defaults that mimic PyPI or (b) document them if they are not easily obtained.

[twolfson]

I have started to research what PyPI does. They are using a library known as readme_renderer for rendering:

https://bitbucket.org/pypa/pypi/src/ef4556223658b5a45e3585f5486b333b37501806/webui.py?at=default&fileviewer=file-view-default#webui.py-50:51

https://bitbucket.org/pypa/pypi/src/ef4556223658b5a45e3585f5486b333b37501806/webui.py?at=default&fileviewer=file-view-default#webui.py-1867:1874

[twolfson]

It turns out I went on a bit of a wild goose chase. The "bad" rst file you linked me to wasn't bad, I had to go back 1 further in its history to find a bad one.

https://github.com/douglasrizzo/catsim/blob/8e07d59442c205be91afa07b630d016754f39201/README.rst

That being said, we now know that PyPI more/less uses the same technique as us:

https://github.com/pypa/readme_renderer/blob/e78d4092026f0e67417109cf40329eadc5e88834/readme_renderer/rst.py

The issue we were seeing in the bad rst can be raised in rst-lint by instaling Pygments (as you noted). When we run the same document via rst2html from docutils we get a warning about needing to install Pygments to handle it. My suggestion is that we bring back propogating that error.

[twolfson]

Actually, it looks like the Pygments error is independent of :linenos: and rst-lint is catching it. Can you provide the case when rst-lint didn't catch the error and PyPI failed to render?

[douglasrizzo]

Alright, I created a gist with a sample rst file that rst-lint said is clean, but when I uploaded it to pypitest (I just can't afford to keep uploading new versions to pypi), it did not render correctly. At first, rst-lint raises a warning, because there is a BibTeX code block, which Pygments cannot pygmentize natively. I installed an additional package and the file was clean, like so:

$ rst-lint README.rst 
WARNING README.rst:61 Cannot analyze code. No Pygments lexer found for "bibtex".
$ sudo pip install bibtex-pygments-lexer
$ rst-lint README.rst 
INFO File README.rst is clean.

Notice that this file has no :linenos: in it. rst-lint raises an error if :linenos: is present, but since you asked for an example that passed rst-lint and failed to be rendered on PyPI, I removed the statement.
It seems to me, then, that when rst2html is able to generate the html documents from the rst files, rst-lint is also able to lint these files and say they're clean. The problem is that this html generation process depends on packages installed locally. If the same packages are not present when PyPI renders our readme files, the process will fail remotely, while still being successful (both html generation and linting) on our machines.

Here is a screenshot from pypitest:

[twolfson]

Wow, thanks for all the help in debugging =) I setup a one-off script to run the PyPI renderer locally and yielded no errors as well

https://github.com/twolfson/restructuredtext-lint/blob/68a060ae05515d3531b408c1df08697b758a959e/main.py

I think you are onto something with respect to local dependencies. Since it's not "solvable" do you think a warning in the README would suffice?

For example: "Any non-standard Pygments lexers will lead to PyPI errors despite local installation"

[douglasrizzo]

Yeah, I don't know if it's the linter's fault when a file is render-able and the linter says it's clean. After all, we can't raise any errors on a perfect rst file!

It would be nice if we could actually know every exception PyPI has though (exceptions we do not have in our computers) and warn people about them. I know GitHub itself has a bunch of them, like not allowing .. include:: file.rst in the readme and stuff like that. I'm pretty sure PyPI doesn't allow those too, it's too dangerous to allow a file to reference another one when it is being processed on your servers. There is also the issue with relative links I mentioned before.

I don't really understand how rst linting works, but it looks like we can't make the linter itself issue a warning, right? Maybe if the application had a flag, like this:

$ rst-lint README.rst
WARNING README.rst:61 Cannot analyze code. No Pygments lexer found for "bibtex".
$ sudo pip install bibtex-pygments-lexer
$ rst-lint README.rst
INFO File README.rst is clean.
$ rst-lint README.rst --pypi
ERROR README.rst:61 Any non-standard Pygments lexers will lead to PyPI errors despite local installation (foreign lexer: "bibtex").

If that is not possible, we should try to warn people about the differences between local and remote rendering, just like you said.

[douglasrizzo]

If that's the case, I can write a few paragraphs explaining what we found out during this issue, add to your README and make a pull request. Just give me a day.

[twolfson]

With respect to linting, we are using docutils to parse the document which is also where the errors originate. As a result, preventing it from automatically loading lexers is not plausible.

It sounds like we are in agreement with using the README as our resolution. With respect to the documentation, while I appreciate the willingness to write it, I'm usually picking about how things are phrased and how concise they are.

I'm going to commandeer and write up some info that covers common PyPI issues.

Thanks again for your quick responses and helpfulness =)

[twolfson]

Alright, the documentation has been updated and released in 0.14.2. Thanks again for all your help =)

https://github.com/twolfson/restructuredtext-lint/tree/0.14.2#pypi-issues

[douglasrizzo]

I'm glad we could further advance the cause of open source.

[bittner]

the cause of open source

Free/libre software, more precisely, I would suppose. 🤓