[RFC] `mdx` stanza rework

[voodoos]

Presentation

This issue documents the design of a rework of the current mdx stanza.
Implementation has already begun in this draft PR#3956, but there is still time to
discuss the design. One important point is the role distribution between mdx
and dune with regard to code generation.

I think the best choice is to do the code generation on the mdx side because
the generated code uses the public mdx libs and if these change the generator
should also change and it will be easier to keep these in sync if they are
packaged together. However relying on dune to generate the code might be more
performant.

Current status of MDX / Dune :

MDX = multiple binaries:

• ocaml-mdx a native entry point which invoke mdx-test
• mdx-test the actual test binary, compiled to bytecode because it interacts
  with the toplevel

The Stanza

(mdx
 (files <glob>)         ; files to test
 (packages <packages>)  ; local dune pkgs that the doc blocks depend on
 (preludes <files>))    ; code to run before the test

No generic deps field and only public libraries (in packages) can be exposed
to MDX.

For each file to test:

• Dune calls ocaml-mdx deps thefile.md to obtain the dependencies of the
  tested file (when code blocks include code from other files)
• Dune call ocaml-mdx test [--prelude ...] -o thefile.corrected thefile.md,
  which call the mdx-test binary. This binary basically calls a function in
  MDX's oublic library: Mdx_test.run-exn.
• Diff the result with the expected output and allow the promotion of these
  results.

Limitations

1. MDX cannot link to private libs without knowing their CMI paths.
2. mdx-test must be compiled to bytecode
3. ocaml-mdx invokes mdx-test at runtime, but this dependency cannot be
   expressed easily in dune. In RWO, workaround is to add an explicit dependency
   on mdx package.
4. No full "deps" field
5. The future opam monorepo version of RWO will not compatible with the
   (package mdx) solution because (package) works only locally.

The new MDX stanza

(mdx
 (files <glob>)
 (deps <deps>)
 (libraries <library-dependencies>) ; Libraries that will be statically linked in the test executable
 (preludes <files>))

The stanza's syntax is similar but the execution differs a little

Once for each mdx stanza:

• Dune would call once a ocaml-mdx dune-gen subcommand (not for end-user use)
  that would generate the source of an executable MDX as a library.
• This test-running program will use the newly accessible Mdx_test.run_exn function and
  is linked with the public and private libraries that the tests use.
• Arguments for this generator would include:
  • paths to cmi files of the deps, including the ones of private libraries
  • the preludes defined in the stanza
• Then dune builds a -- bytecode -- test executable from the generated source.

For each file to test:

• Dune calls ocaml-mdx deps thefile.md to obtain the dependencies of the
  tested file (when code blocks include code from other files)
• Dune call the newly generated executable with the file as argument
• Diff the result with the expected output and allow the promotion of these
  results.

This should solve limitations 1, 3, 4 and 5. However bytecode compilation will remain
necessary as long as no native "JIT" toplevel is available.

Ping @NathanReb, @emillon

[NathanReb]

I want to add one precision.

The current limitation regarding private libraries is tied to the fact that the only way you can use a library in a MDX toplevel block is to load it using #require ocamlfind directives. For this to work properly, the library has to be public and opam installed or to be part of a local dune package. In the second case, users need to add the (packages ...) (or in the new version a (deps (package ...) ...)) for the package that ships that library. That way the library will be built and added to the _build/install folder which will be in the path when ocamlfind tries to load it.

To clarify, here are a couple of examples. If you need to use an opam installed library somelib you would write the following dune and test.md files:

(mdx
  (files test.md))

```ocaml
# #require "somelib";;
# Somelib.do_something ();;
...
```

If you need to use a local public library, here's what you need:

(mdx
  (files test.md)
  (packages somelib))

```ocaml
# #require "somelib";;
# Somelib.do_something ();;
...
```

We want to preserve that behaviour as in some cases we do prefer to explicitly load the libraries using ocamlfind so that the toplevel blocks are similar to what a toplevel user would type and their content can actually be copy/pasted into a toplevel session.

Loading libraries through ocamlfind has some limitations though, it has to be public libraries and as of today, you can't write an MDX stanza which would work both when the library is from a local package and when it is opam installed as (packages a b) is just a shortcut for (deps (package a) (package b)) which don't work for opam installed packages and lead to errors in that case.

This is why we want to provide another way to make a library available in the toplevel blocks, that's by linking when compiling the generated executable that will run the tests and therefore the toplevel sessions. To do this we need an extra field to the stanza. Here's an example of what it should look like:

(mdx
  (files test.md)
  (libraries somelib))

```ocaml
# Somelib.do_something ();;
...
```

Here that means that somelib is linked when compiling the generated executable. To be able to properly use it in the toplevel sessions, dune need to also pass MDX the path to the corresponding .cmi files. This should work whether the library is public or private, locally defined or opam installed.

[bobot]

In the description of the issue there is no libraries field, but I prefer if there is one as the previous comment explain.

Possible points:

• Instead of relying on the #require of ocamlfind mdx could (perhaps in the dune mode) use the (experimental) library dune-site.plugins for implementing it. It changes nothing except no dependency on ocamlfind.
• If you intercept the #require it would be nice to not load the library but raise an error if it is not already loaded (i.e linked) which would mean missing from the library field.

Even more, it is perhaps too late for this iteration (I don't want to delay things that is ready):

• dune-site.plugins is extended to be able to load private libraries.
• mdx knows #require and #require_private using dune-site.plugins.
• mdx gives the dependencies to dune (private and not private libraries)

(Possibly utop could use the same convention).

[NathanReb]

To make sure we're on the same page I want to emphasize that at the moment mdx does not intercepts #require directives and will still depend on ocamlfind's Topfind to do so. We want to preserve the possibility to use #require directives and we do want them to behave the way they do now. I consider it a feature for as long as toplevel users will rely on ocamlfind to load libraries, which is still the case for a lot of people, especially newcomers, students etc...
If we start interpreting them differently than toplevel users would, I'm afraid we'll end up in a situation where in some context mdx code blocks are properly evaluated but trying to copy/paste their content into a toplevel won't work or the other way around.

If in the future ocamlfind isn't required to load libraries in toplevels and a dune based solution has been widely adopted, I'd be more than happy to switch to it as it would likely make the maintenance easier but as I mentioned above, I think for now it's important that mdx supports this.

For this iteration I would much rather stick to the current solution which despite some of its limitations, has been battle-tested in RWO. I think this new proposal will lift some of those limitation by allowing the use of libraries without relying on #require directives but I think it's still too soon to completely drop ocamlfind here unfortunately.

That being said I'd be happy to further discuss dune-site.plugins as part of future plans for mdx!

Also it's important to say that we aim at making the dune mode the only mode, or to be more specific the end goal is to delete the main mdx executable and to only have the described mode of execution, i.e. generate .ml file -> compile it -> execute it on input files. Eventually we could write integrations with other build system if needed but in practice it's likely to be the only mode.

[bobot]

If we start interpreting them differently than toplevel users would, I'm afraid we'll end up in a situation where in some context mdx code blocks are properly evaluated but trying to copy/paste their content into a toplevel won't work or the other way around.

I agree we need to keep the same semantic. The semantic is for me to just load the library cma, adding the -I, and loading the ppx. Is there any other thing?

For this iteration I would much rather stick to the current solution which despite some of its limitations, has been battle-tested in RWO.
That being said I'd be happy to further discuss dune-site.plugins as part of future plans for mdx!

No problem let that proposition aside for now, we can discuss all of that later. dune-site.plugins needs some little work for getting the ppx information for example.

More on point with the proposition:

• I didn't understand what happen if someone forget library foo in the mdx stanza but use #require "foo";;.
• If someone add a #require "fo" for a non-existant library.

[emillon]

I think that we can close that one now that stanza 0.2 exists?

[emillon]

Closed by #3956