Localization/i18n RFC

[MrTact]

Localization/i18n proposal

Localizing a project of any size is deceptively complex. Localization touches an application in many places. Not only that, it demands building processes to maintain translations and building communities of users responsible for doing that maintenance. I’ve taken a stab at identifying all the major considerations. I have clear thoughts on some of these, but others are essentially placeholders for “remember to think about this.”

I wanted to pull this together and make it available for comment before starting any of the work, as I know there will be lots of interested stakeholders, many of whom come from regions outside the US and therefore bring valuable perspective to the problem.

I don’t think we need to answer literally every question in here before we can start on the work. There are a handful of critical decisions that have to be made; the rest can be planned out while that work is underway.

Overarching goals

• Create infrastructure for
  • Identifying locale
  • Overriding locale via settings
  • Loading appropriate string resources
  • Rendering translations in the UI where we currently only display English
• Get existing UI text translated
  • This is a community-building exercise
    • We should probably have translation leads for each language
  • Initial target languages?
• Craft a workflow for adding new translations/integrating proposed revisions
  • Make it as easy as possible for community members to propose new translations
  • Minimize the amount of technical skill required

Dependencies

fluent

I recommend that we integrate the Fluent library for localization

• Designed to handle very complex use cases (e.g. gender, pluralization, and text casing)
• Personal bias -- it's the one I've used previously
• Was created by Mozilla for use in Firefox, so it's got roots in the Rust community
• More DLs and more packages using it than i18n, the next contender
• Playground tool for experimenting with translations before submitting them https://projectfluent.org

fluent-templates

Provides a useful abstraction for loading translations. (Also includes templating, but we won't need that.)

unic-locale

An API for managing locale identifiers.

Translation macro

I haven't found anything for fluent as nice for performing translation operations as the t! macro from rust-i18n. We will probably want to steal and adapt that to work with fluent.

Architecture

Loc resource storage

There are two schools of thought on this -- embedding strings directly in the binary, and loading them at startup. I think embedding is a non-starter; we want to permit volunteers to target new locales without having to think about the impact to startup time (or, for that matter, even to be required to build Zed themselves).

I recommend starting out with whatever string file format is supported by the localization crate we choose & loading the entire string table at startup. On top of this, add a metric that tracks the cost of string table load on startup so we can monitor for regressions. In the event this gets to the point where the startup cost is untenable, we can explore more complex but also more efficient approaches (memory-mapping string files, lazy loading, caching etc. etc.)

Initialization

Locale resolution

TBD, but consider something like https://docs.rs/locale-match/0.2.4/locale_match/, at least as a starting position.

String table loading

Here’s what I want to do. It might be overkill.

• We have a command to import translations. This can be just a random file on the user’s hard drive. This allows translators to just pull the current state of translations into their Downloads directory and import them with this command.
  • Importing translations merges translation files into the string files that are installed with the application binary. Merges should be straightforward, as we assume the new translation overrides any prior definition.
  • Merged translations are cached in a specific directory
  • We will want translation linting, to catch cases where (for example) the number of tokens in a string changes. It can be disconcerting (to say the least) when you update translations and the build breaks.
  • Do we build a merged runtime string table specific to the user or do we load multiple string tables? (This relates to the question of whether we need to support complex fallback chains under the “Settings” header below.)
• On startup, we load the preferred string table from the build cache.
• We also need a “reset translations” command to purge the cache and repopulate it with just the shipped strings.
  • Maybe also a “rebuild translations” command if we support complex fallback chains and bake those out per-user, just in case the cache gets corrupted.

Rendering strategy

Make the caller do lookup

In this case, wherever you instantiate a component, you would have to call (probably!) a macro to translate your string in place, passing it an identifier.

menu.entry(
	t!(remove-from-project), // <-- HERE
	None,
	cx.handler_for(&this, move |this, cx| {
		this.project.update(cx, |project, cx| {
			project.remove_worktree(worktree_id, cx)
		});
	}),
)

Pros:

• Efficient. Macros embed string lookup code at compile time.
• Approach is the same for tokenized strings
• Doesn’t require much (if any) changes to components
• Incremental PRs can translate whatever, and we see those translations as soon as the PRs are merged.

Cons:

• You HAVE TO create the string entry before the string can be displayed. This means adding it in two places: add the key/value to the baseline (en_US) string table, then add the key lookup macro in the code.
  • We could mitigate this by finding (or creating, if necessary) tooling to automate string insertion

Make components locale-aware

With this approach, we update the components to take the strings that are passed to them and “hash” them (this could be as simple as converting them to lower-kebab-case). These hashes get used as the string ids.

Pros:

• Requires minimal changes to development of UI code the way it is done today
  • Because the identifier is the swizzled English string, we can have fallback code that simply displays the string as entered. Ergo when adding a new UI element, you can just... add it
  • String tables can be extracted mechanically
  • Tokenized strings will have to be updated
• IMHO makes the code more readable
• Don’t have to retrofit existing instantiation calls.
• If we find ourselves needing highly customized rendering code for particular languages, the component is where that would have to happen anyway.

Cons:

• More work up front on setting up the infrastructure.
• Components don’t support translations until the individual component itself is updated.
• Performance could be a concern -- running the hash every lookup could be expensive. We can amortize this by caching, but that costs memory. Could probably be mitigated by the macro?
• May have to figure out how to handle hash collisions (in the event that you have the same English string in multiple places that would not cleanly translate to the same string in other languages. Yes, I realize that’s an extreme edge case. Maybe it’s not worth worrying about until it happens and we can better see the shape of the problem.)
• Arguably more brittle when doing translation updates
• Requires components add a separate initialization function that takes a raw (pre-translated) string

A mixture of both

We don’t make components locale-aware, so they can receive untranslated strings. However, we use the hashing-the-English-string-as-key approach from the locale-aware component proposal.

Pros:

• Kind of the best of both worlds? Still allows untranslated strings & incremental translation,
• Doesn’t require a lot of re-engineering of UI components.

Cons:

• We still don’t see translations until we add the macro call… but this is pretty straightforward with AI and/or mechanical code updating. We could probably just do one big up-front PR do all these at one go.

I favor this approach, mainly because it significantly reduces friction on UI developers, automating away the more laborious parts of the process, but doesn’t require updating every single component in the app. Having to go through and add a macro call to every string is laborious, but mostly grunt work and can probably be automated.

Token handling

We need to ensure tokens for things like numbers, dates, times are passed to the localization macro as rich types & not pre-rendered to strings and embedded. This will require some code searching.

We will need to then figure out which crates to use for rendering these elements in place and incorporate them into our rendering macro.

We will also need to establish conventions on how to pass tokens to the rendering system.

Settings

• Force preferred locale
  • Is there any complexity we need to handle here? E.g. “force pt_BR but fall back to es_BR?” Any non-US folks want to weigh in on whether this is a thing?
• TBD: How to handle this via the existing settings code. These details could be relegated to a separate ticket.

Error handling

• We need to gracefully handle translation errors, notifying users without being TOO invasive.
• This is (probably? Hopefully?) mostly useful for translators debugging their own work.

Translation tooling

Requirements

• Translation web site, allowing for non-technical users (here “defined as people who use Zed, but don’t build it from scratch”) to update translations
  • Ideally, this lives on zed.dev.
  • Ideally: Integration with the fluent playground, so you can test translations with tokens etc. Maybe we deploy our own version of the playground?
• Simple process for downloading translations into your local running build

Build and Release

• Deploying current string tables as part of a Zed update.
  • Consider releasing string tables as extensions, if string tables get to be large enough that we are concerned about the amount of disk space we occupy. (How large is that? ¯\_(ツ)_/¯ )
• Translation linting as part of the CI process. This should check both translations AND code (i.e. scrape for usages of the t! macro and run some rudimentary tests, such as that the expected number of tokens are correctly supplied in code).
  • For string tables shipped with a release, errors here should be a build failure.

Documentation

Future consideration

Component rework for specific locales

• Some components might have to be adjusted in how they render themselves in order to support certain languages
  • E.g. German (very long strings), RTL languages

How do we populate the OS's UX elements?

• This is a WHOLE SEPARATE USE CASE from our UI, which will need some love and attention
  • E.g dropdown menus on MacOS
  • System tray?
  • Notifications?

[imnotlxy]

Localization involves translating the app itself, as well as its plugins. In your design, how do plugin authors translate their works? If Zed releases language packs as plugins, is it possible to modify/notify all of them when a string is changed?

[MrTact]

This is a really good question, and something I completely missed in the initial proposal. I think the short answer is to enable plugins to supply their own string tables as part of the plugin, which will be translated by the same mechanisms as the main application. However, this demands more thought, so I'll spend some time pondering this, and maybe prototyping it, to find a way that works well for both plugin authors and the user community.

[zbraniecki]

Your intuition matches what generally plugin-enabled apps do (see webextensions etc.)

[imnotlxy]

Your intuition matches what generally plugin-enabled apps do (see webextensions etc.)

Thank you for teaching me this! I looked at MDN (https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Internationalization) to learn about how web extensions deal with translations. The points are:

1. The browser provides i18n API;
2. An extension brings its own message strings.

Exactly what I think. It is clear that extensions supply their own strings.

My question is whether translations of Zed itself should be out-of-tree plugins like VS Code, or part of the tree like what most "native" apps do.

[MrTact]

I think for the core Zed UI, translations need to be part of the app. That doesn't mean that the string tables need to be included in the install bundle, but the appropriate mechanisms should be in place to detect the locale and fetch translations as needed.

[zbraniecki]

Hi, comments on your proposal:

1. For negotiation you can also use https://github.com/projectfluent/fluent-langneg-rs
2. Depending on the timeline for this work, I'd also like to transition fluent-rs to use ICU4X. I'm wrapping up icu4x negotiation and just landed https://github.com/unicode-org/icu4x/tree/main/utils/host_info - together those two will handle tying the runtime to the host locale preferences. (so, effectively you can start with unic-locale + fluent-langneg, and then we'll swap to icu4x/locale, icu4x/locale-negotiate and icu4x/host_info).
3. For management of fluent locales you can deploy Pontoon - open source translation management system. You can see an example deployment used by Mozilla here - https://pontoon.mozilla.org/projects/firefox/
4. Once MF2 matures and ICU4X support lands, and Mozilla is ready to start migrating Fluent to MF2, you will be able to reuse the same tooling, and Pontoon will support MF2 on par with Fluent, so your migration will be easy.

Early vs late localization

One difference between your proposal and what I'd suggest is the interaction between UI construction and l10n resolution.
Your proposal maintains the early resolution model where the component construction is isolated from localization. Example:

menu.entry(
	t!(remove-from-project), // <-- HERE
	None,
	cx.handler_for(&this, move |this, cx| {
		this.project.update(cx, |project, cx| {
			project.remove_worktree(worktree_id, cx)
		});
	}),
)

In this example the entry function takes a string and t! macro provides it. The UI tree does not know what locale the entry is in.
If you want to update translation, you need to rerun this imperative code, reconstructing the menu entry set with new string result from t!.

I believe this approach is deceptively bad for GUI applications, where the UI tree should be treated as a "source of truth" and allow for reactiveness - a'ka - partial or full retranslations based on the UI tree and env information.

I documented this philosophy here raphlinus/crochet#7 and here unicode-org/message-format-wg#118

In that model, I'd recommend the above example to do:

menu.entry(
	"remove-from-project" // bind l10n id to the entry, l10n resolution is a step in layout construction later
	None,
	cx.handler_for(&this, move |this, cx| {
		this.project.update(cx, |project, cx| {
			project.remove_worktree(worktree_id, cx)
		});
	}),
)

This binds the L10n Element to UI Element, rather than the resolved string to an UI element and allows for all sorts of reactiveness:

• dynamic pseudotranslations
• dynamic locale changes / updates (Try it - change locales in Firefox Settings!)
• locale-independent caching
• identity preserved locale updates (you don't need to destroy the entry to retranslate it)
• responsive UI

This is also in line with DOM L10n proposal - https://github.com/mozilla/explainers/blob/main/dom-localization.md

The easiest analogy that I would use is that it's a difference like between passing resolved CSS values to an element constructor, vs CSS class binding. Managing themes is much easier with the latter.

[MrTact]

Thanks so much for your thoughtful feedback! It's clear you have a lot of experience in this area, so I'm inclined to take your guidance to heart. (Also, thanks for your hard work on Fluent, which I greatly enjoyed working with on a previous project.)

I definitely see the value you describe in "late" translation. The upside of components being aware of what locale they are in probably outweighs the downsides I cited in my original proposal (mainly, scope of work), so I'm on my way to being swayed that way. It's also likely that some of the "future considerations" I cited may be nearly impossible without locale-awareness at a component level (e.g. handling languages with long strings like German, RTL languages etc.)

Regarding work being done with ICU4X and MF2, would you say it would be better to wait until those migrations land, or are the transitions likely to be low-to-moderate impact?

Finally, Pontoon was definitely my main contender for a translation platform, given that most of the other prominent choices don't support .ftl out of the box (and since Pontoon was MADE for Fluent, I know that it should support all the flexibility that Fluent gives us).

[zbraniecki]

Regarding work being done with ICU4X and MF2, would you say it would be better to wait until those migrations land, or are the transitions likely to be low-to-moderate impact?

I recommend going with Fluent now.

@eemeli is validating MF2 design via a lossless transpiler from Fluent to MF2 and you'll be able to just reuse it.

[eemeli]

MF2 provides a superset of the message-level capabilities that Fluent has; @messageformat/fluent is a relatively stable JS package that provides bidirectional convertibility between the two. Fluent's .ftl files provide more structure (like message attributes) that's beyond the scope of MF2 itself; for that, work on message resources is continuing under the W3C i18n WG.

The JS package's message resource representation is therefore not completely locked down, but it is kept in sync with e.g. @messageformat/xliff (not yet released, still a bit of a work in progress) which provides bidirectional convertibility with XLIFF2, which does potentially allow you to use Fluent in a product, but have the translation/localization work happen based on XLIFF2. That won't give you all the capabilities of Fluent (e.g. allowing translators to fine-tune pluralization or create their own term attributes), for that you kinda have to go with Pontoon atm, but it'll do most of the heavy lifting for you (like supporting plurals with locale-appropriate categories).

[larkzhang]

When is the internationalization (i18n) feature expected to be launched? Many users are looking forward to localized UI support. Is there an official roadmap or an estimated release window?

[MrTact]

I'm not a Zed dev, but as someone who wants to see this happen, I'm prepared to move it to the top of my personal projects stack. I think the next step is either to attract the attention of a dev, who can sponsor the work (be primary reviewer & merger, and help with getting a translation tool hosted somewhere official, for example) OR just throw caution to the wind, start doing work in a fork, and hope that once the feature has enough momentum that the Zed team is willing to merge it. I think the former is preferable, if for no reason other than that we don't want to be stomping on any internal efforts.

[lcretan]

Hi Team,

As you know, we did similar struggling in NCSA Mosaic.

When we made it and other components such as servers, we consisted on the I18N, not L10N.

In this approach, you may seem the initial investment too much, however, the scalability and offloading were promised.

However, the full wrapping of other system is vullnable from two aspects:

1. The risk of the unconscious usages of the commercialized / capitalized words and the characteristic words usage.

2. The loop of the wrappings would the streamlining of the system vulnerability.

Indeed, 1. would need the intellectual property professionals, however, 2 makes us the urgent awareness of the possible severity and we may need the 3rd party inspector.

We need to start the split the programming language and the human natural language.

After these refactors, including the languages written from right to left, the ones using many different sizes of words in the same context,

Anyway, as you split your words from your usual usage of languages to the prompts of LLMs, we need the solid and flexible based consensus and running codes.

The programing code is the computer language and we can use the different format checkers,
we can deploy many LLMs for the basics of the natural expressions.

Can you write a converter from English to others in UTF8 by your own effort?
No, GenAI do almost of them.

After these step, finally, the minimum cost of human blush up.

[zsyo]

Hi everyone,

Regarding the implementation of i18n in Zed, I would like to suggest using fluent-bundle. This approach offers several key advantages for a high-performance editor:

• Externalized Resources: By keeping translation files outside the binary, we can support dynamic language switching at runtime without requiring a restart or re-indexing.

• Decoupling Code from Content: Adding a new language would simply involve dropping a new .ftl file into the designated localization directory. There is no need to modify the source code or trigger a recompile.

• Lowering the Contribution Barrier: This "zero-code" approach allows non-developers and community translators to contribute new languages easily. They can focus entirely on the translation files without needing to understand the underlying codebase or the Rust build pipeline.

I believe this would make Zed much more accessible to a global audience while maintaining the project's commitment to efficiency and flexibility. What are your thoughts on this direction?

[leic4u]

Zed v1.0 is released, so is there any progress about localization by official?