Feedback for upstream mdbook

[Michael-F-Bryan]

I recently became the main maintainer of mdbook and was wondering what I can do to make the process of working on TRPL easier? I know The Book is one of our biggest users, so it probably makes sense for there to be a bit more communication/collaboration between the maintainers of mdbook and TRPL.

For example, one of the big things I've been working on lately is alternate backends. There's an EPUB renderer in the works, as well as a backend for running tests through rust-skeptic (mdbook-test). Especially given there are programs like @killercup's trpl-ebook, would a more bespoke solution like this be useful?

Are there any other things we can do which could really help out TRPL? Your README mentions translations are currently blocked on upstream support. There's currently an issue for this (rust-lang/mdBook#5) but it looks like discussion has stalled and it's not apparent from the comments what exactly is desired. We've recently cleaned up a lot of the internals, so it may be a good time to revisit multilingual support.

[carols10cents]

Hi, thanks for checking in and congrats on your new maintainership duties! I'd love a way to generate an EPUB or PDF with mdbook!

And you've discovered correctly that the biggest item on our wishlist would be translation support-- I'd love to be able to incorporate translations in a way that makes sense for people doing the translations and for the main repo. I don't know exactly what that would look like, but I'm up for trying things!

[Michael-F-Bryan]

I'd love to be able to incorporate translations in a way that makes sense for people doing the translations and for the main repo. I don't know exactly what that would look like, but I'm up for trying things!

Awesome! To get started it might be a good idea to ask what exactly does maintaining multiple translations require, that way we can figure out a set of requirements, deal breakers, etc. For example, what's stopping you from just having multiple separate books which are maintained in parallel?

Feel free to CC in anyone who might have an opinion on translations, or just mdbook in general! I know I've spammed @steveklabnik a lot in the past regarding mdbook issues (sorry!), but it'd be good to get different opinions and ideas from people.

[carols10cents]

@sebasmagri will definitely have more informed thoughts than I will-- Sebastián, I'm ccing you on this issue which is the new maintainer of mdbook asking for feedback on what the rust book could use from mdbook, including translation support. We'd love your thoughts/ideas if they've changed at all since this comment of yours! rust-lang/mdBook#5 (comment)

---

My thoughts: I see a few stakeholders here, only some of which I can really speak for...

Main repo maintainers/deployment

For example, what's stopping you from just having multiple separate books which are maintained in parallel?

Nothing, that's what we're doing right now-- but we're just linking to them from an appendix which isn't very discoverable. I'd also like to recognize the translations as being "official" (have them in a rust-lang repo, have them available at doc.rust-lang.org), although not managing all the pull requests/issues for all the translations in this repo is a plus of the current system...

We're also not shipping the translations with Rust for offline reading. Currently the way that works is that rust-lang/rust has a git submodule of this repo and the rust-lang/rust build process runs mdbook. We could add more submodules and run mdbook more times, but that means we'd have to send a PR to rust-lang/rust to modify the build system for every new translation-- it'd be nicer if we could just add a translation to this repo and have it included the next time the git submodule is updated. I suppose there's probably a hacky solution we could do where this repo has git submodules as well???

Readers

As far as the reader experience goes, I would imagine this from rust-lang/mdBook#5 would be ideal:

1 to 1 mapping from pages from one language to another, would allow changing the language of the page directly from a menu button

But I don't actually know! Do people switch between the English and a translation and read both together? Or do people just switch to a translation and stay with the translation?

Translators

This comment is interesting, about tools to help manage translation efforts: rust-lang/mdBook#5 (comment) I have no idea the level of effort of integrating with something like this or if it's desirable!

[steveklabnik]

It's basically

1. translations
2. search
3. ... i had something, but forgot it :)

[Michael-F-Bryan]

2. search

There's an open PR (rust-lang/mdBook#472) which adds search functionality to the book. The author also included a proof-of-concept of TRPL with search enabled which looks really, really nice!

Unfortunately that PR has developed a couple merge conflicts as bits and pieces of the code base have been refactored, but hopefully they shouldn't be too difficult to resolve.

---

But I don't actually know! Do people switch between the English and a translation and read both together? Or do people just switch to a translation and stay with the translation?

I feel like you'd want to choose a translation in the language you're most comfortable with and stick to that. Chopping and changing between language means you're constantly context switching, I remember it used to take about 5-10 minutes for my brain to properly change from English to French although that could just be me.

One possible idea is you have multiple separate books which share a book.toml, but can internally have their own structure (SUMMARY.md), to make maintaining easier. You could then have some sort of table on the side that says which translated chapters are equivalent, with the table being used to allow changing the language directly from a menu button/drop down.

• /
  • book.toml
  • equivalents.csv
  • en/
    • SUMMARY.md
    • first.md
    • second.md
    • ...
  • fr/
    • SUMMARY.md
    • premier.md
    • deuxieme.md
    • ...

With the equivalent pages table looking something like this:

en/first.md, fr/premier.md
en/second.md, fr/deuxieme.md

(actual format subject to bikeshedding)

The root index.html can then just be a redirect to the default translation, as specified in book.toml.

[sebasmagri]

Hi! Thanks for the mention @carols10cents!

I agree in general terms with the mapping proposal, which has also been discussed in the existing issues. Generally, at least in my experience, people switches languages and stay in that language, so the mapping should also affect links and ToCs.

Other than some bikeshedding feedback like having the default language without a directory prefix, having a directory prefix for all translations like in translations/{fr,es,ru,etc} and having all translated files with the same filename as the originals, which will make tracking translation status easier, I believe the proposal is pretty good.

I remember I asked @carols10cents about how would a workflow to integrate translations into the TRPL builds work based on this. An idea I had was that the translation directories could be git submodules, but we'd need some sort of acceptance criteria. So if mdBook could provide some sort of coverage reporting facilities for translations that'd be awesome too, leaving quality tracking to translation maintainers.

[Michael-F-Bryan]

Generally, at least in my experience, people switches languages and stay in that language, so the mapping should also affect links and ToCs.

Other than some bikeshedding feedback like having the default language without a directory prefix, having a directory prefix for all translations like in translations/{fr,es,ru,etc} and having all translated files with the same filename as the originals, which will make tracking translation status easier, I believe the proposal is pretty good.

I was thinking we could take a leaf out of cargo's book and adopt a set of "sane" conventions which can then be overridden using the configuration file. Possibly something like this:

regarding directory structure

• Use the current single-language structure by default
• If a [translations] table is present in book.toml switch to "multilingual" mode, where each directory within src/ is interpreted as a unique translation with its own SUMMARY.md
• If you want to use your own directory structure, you provide this by adding extra keys to the [translations] table (e.g. en = "translations/trpl-en")

File linking in "multilingual" mode

• By default, if files in two translations have the same name (e.g. src/en/first.md and src/fr/first.md), they're considered equivalent and you can switch between them via a drop-down
• If the translations table contains a translation-links, we just use that to figure out which translations are equivalent and should be linked

[translations]
en = "translations/trpl-en"
fr = "translations/trpl-fr"
ru = "translations/trpl-ru"
translation-links = "equivalents.csv"

... Or is that going to over-complicate things in the long run?

---

So if mdBook could provide some sort of coverage reporting facilities for translations that'd be awesome too, leaving quality tracking to translation maintainers.

#507 introduces the ability to implement your own backend, which is essentially just some program that's given a representation of the loaded book so the backend can do something with it.

There's nothing stopping you from implementing your own TRPL-specific "backend" to do things like spell and link checking, coverage reporting, or whatever you want. A backend is just some command which is invokable (e.g. python link-check.py) and passed a JSON representation of the book via stdin.

So even though we may not provide a coverage reporting tool out of the box (at first anyway), we're still giving people everything they need to craft the tools needed for their specific use case... Would that work?

Sorry for the wall of words!

[Michael-F-Bryan]

Just a little update on how things are going... The alternate backends PR landed on master, so you can now make your own backends and tooling with mdbook 🎉

I've continued working on the epub renderer and, other than poor styling because I haven't really tweaked the embedded stylesheet, it looks really really good!

Here's a sample when I run mdbook-epub --stand-alone over the second edition of the book:

The Rust Programming Language.zip

[Michael-F-Bryan]

FYI I created a mdbook-linkcheck backend for checking links. I've tried it out on several repositories so far and it's been fairly robust.

It managed to find a couple broken links in Rust By Example (rust-lang/rust-by-example#990), but I wasn't able to find anything when I ran it over both the first and second versions of TRPL other than false positives like "../../std/prelude/index.html" doesn't exist... So either there are no broken links in TRPL because your link checking script is doing its job properly, or I've missed things 😜

EDIT: There's now a tutorial on how you can create your own tools on top of mdbook.

[Michael-F-Bryan]

I just updated the "add a spellchecker" issue for mdbook (rust-lang/mdBook#357 (comment)). Now alternate backends are a thing and mdbook 0.1.0 has been released, we can create a mdbook-ispell tool which everyone can use for spellchecking their documents. That way we can avoid needing hand-rolled solutions like spellcheck.sh.

If you know anyone who may be keen to take on the job, please let them know 🙂

[steveklabnik]

We'll continue to send you upstream issues, but I don't think tracking them here makes sense. Thanks for all of your work <3