Feature: conversion of values for application/x-www-form-urlencoded

[ThisIsMissEm]

In many cases it's necessary to send data with application/x-www-form-urlencoded content type, and it turns out encoding to this content type is a bit of a mess. Firstly, when it was originally defined in HTML 4.01 in 1999, it wasn't registered as an IANA media type. It has retroactively been added to the registry, but resulted in additional definitions such as this one in OAuth 2.0 which is used for the token endpoint auth method known as "client_secret_basic". It was also redefined in the WHATWG specs: https://url.spec.whatwg.org/#concept-urlencoded-serializer

It's also maybe used for Authorization: Basic <...> headers — though the OAuth 2.0 spec may be deviating from RFC7617 here as RFC6749 (OAuth 2.0) is older.

I thought initially that it should be String.addingPercentEncoding(withAllowedCharacters: .urlQueryAllowed), however that didn't seem to be correct as it doesn't encode + to %2B. I think that may be for the entirety of a query string, not the individual parameter names and values.

In projects like AlamoFire, they have a thousand-ish line encoder for URLEncodedForm: https://github.com/Alamofire/Alamofire/blob/master/Source/Features/URLEncodedFormEncoder.swift

I'm not sure if it would make sense to go that far, but maybe we just need a CharacterSet that is correct for this encoding?

This is also the default content type for POST request bodies.

[ThisIsMissEm]

I think the correct CharacterSet is:

extension CharacterSet {
    /// Creates a CharacterSet from RFC 3986 allowed characters.
    ///
    /// RFC 3986 states that the following characters are "reserved" characters.
    ///
    /// - General Delimiters: ":", "#", "[", "]", "@", "?", "/"
    /// - Sub-Delimiters: "!", "$", "&", "'", "(", ")", "*", "+", ",", ";", "="
    ///
    /// In RFC 3986 - Section 3.4, it states that the "?" and "/" characters should not be escaped to allow
    /// query strings to include a URL. Therefore, all "reserved" characters with the exception of "?" and "/"
    /// should be percent-escaped in the query string.
    public static let urlFormEncodedAllowed: CharacterSet = {
        let generalDelimitersToEncode = ":#[]@" // does not include "?" or "/" due to RFC 3986 - Section 3.4
        let subDelimitersToEncode = "!$&'()*+,;="
        let encodableDelimiters = CharacterSet(charactersIn: "\(generalDelimitersToEncode)\(subDelimitersToEncode)")

        return CharacterSet.urlQueryAllowed.subtracting(encodableDelimiters)
    }()
}

RFC 3986 updates RFC 1738 which is referenced by HTML 4.01, though I'm not sure if this is actually the same character set as WHATWG has, which is the IANA registry spec reference for application/x-www-form-urlencoded (instead of HTML 4.01 which is what OAuth 2.0 references)

[ThisIsMissEm]

WHATWG's URL spec also has a bunch of test vectors defined: https://github.com/web-platform-tests/wpt/blob/master/url/urlencoded-parser.any.js

[guoye-zhang]

Sounds like a good feature to have. @jrflat Maybe this should be in Swift Foundation?

[ThisIsMissEm]

@guoye-zhang @jrflat okay, so I attempted to do a parser with removingPercentEncoding and this ends up failing the really weird test cases in the Web Platform Tests. The test cases of:

[
  "b=%2sf%2a",
  "b=%2%2af%2a",
  "b=%%2a",
]

are expected to output:

[
  [['b', '%2sf*']],
  [['b', '%2*f*']],
  [['b', '%*']]
]

respectively, but removingPercentEncoding on these returns nil, so if you're doing str.removingPercentEncoding ?? str then you get the original value input. That is "%2sf%2a".removingPercentEncoding returns nil instead of `"%2sf*"

Likewise, for addingPercentEncoding for "%2sf*" returns "%252sf*" not the expected "%2sf%2a"

So I don't think this can actually be implemented with addingPercentEncoding / removingPercentEncoding since the output of these methods doesn't match the output expected by the Web Platform Tests, which is testing the implementations of this algorithm: https://url.spec.whatwg.org/#urlencoded-parsing

Maybe there's code in Safari for this?

[jrflat]

Yeah, this WHATWG behavior of leaving invalid % escapes during decoding isn't exactly compatible with the current behavior of removingPercentEncoding, which is more strict and will return nil like you mention. We would likely need a new opt-in API to match the WHATWG behavior if desired.

However, I'll also mention that addingPercentEncoding(withAllowedCharacters:) can perform component-specific behavior based on the set you pass in. For example, .urlHostAllowed does not encode leading and trailing [ ] that denote an IP-literal, even though those characters aren't allowed elsewhere in the host. If there's a need for special encoding behavior, I think it's reasonable to implement it within the current addingPercentEncoding function by matching on the .urlFormEncodedAllowed set, specifically.

[ThisIsMissEm]

For now we're just moving forwards with using addingPercentEncoding even though the encoding and decoding is technically WHATWG compliant. It's "good enough", but it would be really good to see in Swift a native "FormData" type which can do both multipart/form and application/x-www-form-urlencoded since these are so common in HTTP.