Split docutils only functionality into separate package

[chrisjsewell]

extracted from #342:

I was surprised to see docutils 0.17 add experimental Markdown support using Recommonmark

ah I completely missed the response on the email thread.

Ah well that's a different proposition: I literally have myst_parser/docutils_renderer.py and myst_parser/sphinx_renderer.py as a sub-class, so that it is certainly possible to use myst-parser with "docutils only" functionality.

If I had known previously about such an intention to ship Markdown in docutils, I would certainly have considered splitting the "docutils only" aspects in to a separate "myst-docutils" package (and have that as a dependency here), which I think would probably be a better solution than just moving dependencies to extras.
Indeed, if there was an agreement in principle with the docutils guys to include something like myst-docutils in docutils I would look into this.

[cpitclaudel]

Ah well that's a different proposition: I literally have myst_parser/docutils_renderer.py and myst_parser/sphinx_renderer.py as a sub-class, so that it is certainly possible to use myst-parser with "docutils only" functionality.

I think this would be great. The thing that's missing right now for convenient integration with docutils, I think (besides separating dependencies), is a version of sphinx_parser for docutils. It would allow this kind of code to work:

from docutils.core import publish_string
from myst_parser.sphinx_parser import MystParser
print(publish_string(source="{math}`e^{i\pi} = -1`", parser=MystParser(), writer_name='html5'))

(and with that integration into a variety of other docutils-based pipelines; my use case is integrating with Alectryon for Coq proofs.)

As currently written, this code crashes:

Traceback (most recent call last):
  File "minimyst.py", line 3, in <module>
    print(publish_string(source="{math}`a^2 + b^2 = c^2`", parser=MystParser(), writer_name='html5'))
  File "/home/clement/.local/lib/python3.8/site-packages/docutils/core.py", line 407, in publish_string
    output, pub = publish_programmatically(
  File "/home/clement/.local/lib/python3.8/site-packages/docutils/core.py", line 665, in publish_programmatically
    output = pub.publish(enable_exit_status=enable_exit_status)
  File "/home/clement/.local/lib/python3.8/site-packages/docutils/core.py", line 217, in publish
    self.document = self.reader.read(self.source, self.parser,
  File "/home/clement/.local/lib/python3.8/site-packages/docutils/readers/__init__.py", line 71, in read
    self.parse()
  File "/home/clement/.local/lib/python3.8/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/home/clement/.local/lib/python3.8/site-packages/myst_parser/sphinx_parser.py", line 55, in parse
    config = document.settings.env.myst_config
AttributeError: 'Values' object has no attribute 'env'

(in fact I see now that it's almost exactly the same example as in https://sourceforge.net/p/docutils/mailman/docutils-users/thread/rkv4nb%24139g%241%40ciao.gmane.io/#msg37118232)

This crash is due to parse taking an additional argument that docutils doesn't pass:

    def parse(self, inputstring: str, document: nodes.document, renderer: str = "sphinx") -> None:
        if renderer == "sphinx":
            config = document.settings.env.myst_config # Here, since docutils will call this function without specifying the renderer
        else:
            config = MdParserConfig()

It works fine if I use a subclass of MystParser that passes a dummy renderer string into super().parse instead of the default "sphinx":

from docutils.core import publish_string
from myst_parser.sphinx_parser import MystParser

class NonSphinxMystParser(MystParser):
    def parse(self, inputstring, document) -> None:
        return super().parse(inputstring, document, r"¯\_(ツ)_/¯")

print(publish_string(source="{math}`e^{i\pi} = -1`", parser=NonSphinxMystParser(), writer_name='html5').decode('utf-8'))

… but it's no clear whether that's the right approach (?) So separating things out in a way that makes this works would be very very nice, especially setting MyST options works from the usual docutils.conf, etc.

Also nice would be some canonical way to register MyST's directives and roles: some function that clients can call to make the appropriate calls to docutils.directives.register_directive etc.

In #342 @astrojuanlu wrote:

It sounds like docutils could potentially use https://github.com/executablebooks/markdown-it-py for its Markdown support then?

and @choldgraf wrote

Yep, i think that is the answer.

But am I correct to think that this requires extra work to also parse MyST's config, recreate its custom roles/directives (?) and make a Reader that's compatible with docutils?

In any case it looks like with very small changes (and independently of integrating with the official docutils package) applications like Alectryon that already depend on docutils but not necessarily sphinx could add support for MyST, which would be great!

[gmilde]

Given the end-of-life for recommonmark, is there a chance for a Docutils-only version of the MyST parser that can be utilised by Docutils?

[chrisjsewell]

Heya, DocutilsRenderer is already a docutils-only renderer:

MyST-Parser/myst_parser/docutils_renderer.py

Line 72 in a28e9b7

class DocutilsRenderer(RendererProtocol):

@cpitclaudel has kindly made a PR to allow for controlling the configuration via docutils: #426, that I'm just trying to find time to circle round to and finalise the tests.

The only sticking point perhaps is myst-parser's pinned dependency on sphinx, creating a cylic dependency. I specifically added this, because changes in docutils/sphinx kept breaking myst-parser for users, so would have to think how this could be achieved

[chrisjsewell]

myst-parser v0.16.0 now introduces docutils-only functionality (https://myst-parser.readthedocs.io/en/latest/docutils.html) and https://pypi.org/project/myst-docutils/ release pipeline, which inlcudes no install dependencies on sphinx/docutils 😄

@gmilde, do you want me to open an issue on docutils, to move from recommonmark to myst-docutils?

[gmilde]

Am 10.12.21 schrieb Chris Sewell:

myst-parser v0.16.0 now introduces docutils-only functionality

This is good news. Thank you. From [MyST with Docutils](https://myst-parser.readthedocs.io/en/latest/docutils.html#myst-with-docutils) I see that 5 front-end tools shall be installed as well (but in the package download I didn't find the corresponding files). IMV, separate front end tools are not necessary when we can get this working with `docutils-cli.py --parser=myst_parser.docutils_`.

@gmilde, do you want me to open an issue on docutils, to move from recommonmark to myst-docutils?

If the above works, Docutils can add `'myst': 'myst_parser.docutils_',` to `docutils.parsers._parser_aliases` to provide a shorter parser name. In future, the default "markdown" and "commonmark" parsers may switch to "myst", but this is an API change which requires planning, advance warnings, and tests. Feel free to open an enhancement ticket or start a thread on ***@***.*** to discuss arising issues, a test strategy and possible closer integration.

[chrisjsewell]

I see that 5 front-end tools shall be installed as well (but in the package download I didn't find the corresponding files).

They don't need separate files, they use python entry points:

MyST-Parser/setup.cfg

Lines 50 to 57 in 11fb239

	[options.entry_points]
	console_scripts =
	myst-anchors = myst_parser.cli:print_anchors
	myst-docutils-html = myst_parser.docutils_:cli_html
	myst-docutils-html5 = myst_parser.docutils_:cli_html5
	myst-docutils-latex = myst_parser.docutils_:cli_latex
	myst-docutils-xml = myst_parser.docutils_:cli_xml
	myst-docutils-pseudoxml = myst_parser.docutils_:cli_pseudoxml

I feel this is the more modern approach for including CLI tools in python package distributions.

IMV, separate front end tools are not necessary

I felt this was easier for end users, and also inline with the separate rst2xxx.py front end tools that docutils already ships with

Feel free to open an enhancement ticket or start a thread

Yep will do then 👍