Specify UTF-8 as the character set

[sunfishcode]

Continuing the discussion from bytecodealliance/wasmtime#86:

WASI currently doesn't document the character sets used for filesystem paths, command-line arguments or environment variables.

Two high-level strategies have been proposed:

• Just use UTF-8, and say that WASI can't directly interact with non-UTF-8-encodable strings from the outside world. Where needed, provide escape-hatch features in the API (eg., you can't open a file with an unencodable name by name, but you can get to it by iterating through a directory).
• Use uninterpreted byte seqeunces, and then additional functions for translating to and from UTF-8, as described here.

[kg]

My suggestion for dealing with per-platform/per-filesystem normalization constraints would be to expose an explicit API call for normalizing paths, so that an application can do that once and compare the strings to identify whether they were changed and avoid paying the cost of normalization on every IO call.

IIRC at least one platform requires this now? I think Apple's APFS?

I strongly vote for UTF-8. I have never seen a filesystem path non-representable by UTF-8 in practice.

[npmccallum]

I also strongly agree with UTF-8 everywhere.

[Hywan]

Agree with UTF-8.

[sunfishcode]

I propose we go with UTF-8, which is overwhelmingly supported by commenters here and in other feedback I've received, and has a lot of obvious advantages. To my knowledge, it's what most implementations have started doing as well.

Concerning efficiency, string-oriented APIs already don't play to WASI's strengths -- if there are APIs which involve a lot of string passing, we should look for ways to let users to create and pass around handles instead, as that has several advantages, character encoding being just one.

API design that de-emphasises strings

As an example, consider iterating over the files in a large directory. With fd_readdir as it currently exists, this involves transcoding all the filenames into UTF-8, even if the names aren't needed. If we wish to avoid this overhead, we could change fd_readdir to return directory-entry handles instead, from which filenames could be requested via additional API calls, or which could be used to open files without naming them.

Accessing unnameable files

One other issue is the ability to open files which don't have encodable names. This is very uncommon, but it should be possible to do from applications willing to do a little extra effort. My idea here is, to encode an arbitrary byte sequence in UTF-8:

• replace unencodable bytes with U+FFFD, in the manner of other relevant standards which do such things
• for strings which lose information that way, append a NUL followed by a hash of the first part of the string along with the lost information -- specific syntax TBD, but all encoded within UTF-8 bytes. NUL is not a valid filename character in Windows or Unix, so this won't collide with actual files. The hash allows us to detect if the string is modified by code that doesn't understand the encoding scheme, so we can report errors instead of doing bogus things.
• change WASI's APIs to use ptr+length strings rather than NUL-terminated strings (most already do, but currently the command-line and environment-variable APIs don't)

This way, applications that can use ptr+length could treat these strings as black boxes and retain the ability to open any file. Applications that use C-style strings would work on all encodable names, but see lossy strings in unencodable cases. For many use cases, this will be fine, because unencodable filenames are very rare. And when they do happen, the U+FFFDs should at least make it fairly obvious what's happening.

[Serentty]

Both Windows and Linux have their quirks in this regard. Windows allows pretty much any string of 16-bit numbers as a filename, including unpaired UTF-16 surrogates, whereas Linux leaves the issue of actually interpreting filenames up to applications, so files don't have to be named in the same encoding throughout a drive, and can be nearly any bag of bytes.

Interestingly, I see a lot of software bend over backwards to support unpaired surrogates in filenames for the sake of Windows, despite them not appearing often at all in practice, and yet I never see the same attempt being made for Linux.

I think the approach mentioned above of using UTF-8 with null-based hash escape sequences for unrepresentable characters is definitely worth considering. It allows WASI to give very strong guarantees to software about the encoding of filenames, which is something that most platforms can't do.

[indolering]

I think the approach mentioned above of using UTF-8 with null-based hash escape sequences for unrepresentable characters is definitely worth considering. It allows WASI to give very strong guarantees to software about the encoding of filenames, which is something that most platforms can't do.

Prohibiting control characters would prevent some attacks as well. DWheeler's Fixing Unix/Linux/POSIX Filenames documents a lot of existing prohibitions at the application level. His desire to prohibit leading dashes - is understandable, but probably too restrictive.

[sunfishcode]

That's a great essay. I agree that it'd be good to see if we can prohibit control characters in filenames while we're here.

I also agree that prohibiting leading - feels like it might be too restrictive. Filenames are end-user facing, so a possible guideline we could use here is: how would a user who only uses computers through GUIs perceive this restriction? For control characters, it's difficult to see how they would even be able to observe the restriction. For leading -, such a user wouldn't expect filenames like "---> hello <---" or "-40 degrees in Whitefish" to be invalid, even if they are uncommon.

[indolering]

For leading -, such a user wouldn't expect filenames like "---> hello <---" or "-40 degrees in Whitefish" to be invalid, even if they are uncommon.

I'm very into the Lang Sec philosophy, but where does it end? The only time you would insert a control character on purpose is as an attack on some input sanitizer. But quotes, unicode literals, and others just as dangerous as a leading dash, just in a different context.

If we are going to scrub filenames, then for the sake of performance we could parameterize the function to allow developer to specify things like a blocklist, a specific Unicode normalization, or even WTF-8. That way we can provide a sane default, but allow developers to tweak it as needed.

[indolering]

Interestingly, I see a lot of software bend over backwards to support unpaired surrogates in filenames for the sake of Windows, despite them not appearing often at all in practice

Can we quantify this somehow?