I'm using Neon to build a VS Code extension, and I am listening for editing events on open text editors. While doing so, I realized that JavaScript strings are represented as UTF-16.
Usually that's not an issue, because Neon uses the NAPI APIs that auto-convert to UTF-8 and the string can be used as usual in Rust.
Now these editing events include line and character positions, and sadly those character positions are not actual char positions but UTF-16 byte positions. So I'll have to adjust them to UTF-8 lengths to work with them in Rust, which I currently do in JavaScript/TypeScript.
I saw that the NAPI has napi_get_value_string_utf16, so it would be possible to get the UTF-16 variant of the string, and I could do all these conversions directly in Rust.
The standard library doesn't have any specific UTF-16 string type, but the widestring crate seems very popular and has what would be needed for this. Probably preferred to hide this behind a feature flag (although, the crate has zero deps).
It would be great if we can add APIs for the UTF-16 string variants of this. Maybe not even necessary to add widestring, and could possibly just expose a to_string_utf16() on the JsString type that returns a Vec<u16> 🤔. The widestring variant would then return a Utf16String.
What do you think about this idea? I could prepare a PR for this, if there is interest to extend the APIs a bit.
I'm using Neon to build a VS Code extension, and I am listening for editing events on open text editors. While doing so, I realized that JavaScript strings are represented as UTF-16.
Usually that's not an issue, because Neon uses the NAPI APIs that auto-convert to UTF-8 and the string can be used as usual in Rust.
Now these editing events include line and character positions, and sadly those character positions are not actual char positions but UTF-16 byte positions. So I'll have to adjust them to UTF-8 lengths to work with them in Rust, which I currently do in JavaScript/TypeScript.
I saw that the NAPI has
napi_get_value_string_utf16, so it would be possible to get the UTF-16 variant of the string, and I could do all these conversions directly in Rust.The standard library doesn't have any specific UTF-16 string type, but the
widestringcrate seems very popular and has what would be needed for this. Probably preferred to hide this behind a feature flag (although, the crate has zero deps).It would be great if we can add APIs for the UTF-16 string variants of this. Maybe not even necessary to add
widestring, and could possibly just expose ato_string_utf16()on theJsStringtype that returns aVec<u16>🤔. Thewidestringvariant would then return aUtf16String.What do you think about this idea? I could prepare a PR for this, if there is interest to extend the APIs a bit.