Accept A-labels

[jsha]

In letsencrypt/boulder#2278 we have an issue where TLDs that are IDNs are not recognized by Boulder as ending in a public suffix. Boulder currently calls publicsuffix.DefaultList.Find with the A-label form of the domain.

We have a few options:

1. Boulder could convert to U-label for querying publicsuffix
2. publicsuffix-go could switch to looking up by A-label by default
3. publicsuffix-go could offer a different set of methods for looking up by A-label.

I think (2) is my ideal solution. What do you think?

[pfigel]

FWIW, Go's x/net/publicsuffix accepts both U- and A-labels, but does not work accurately for suffixes with multiple U-labels:

EffectiveTLDPlusOne("foo.example.xn--o1ach.xn--90a3ac") == "example.xn--o1ach.xn--90a3ac"
EffectiveTLDPlusOne("foo.example.упр.срб") == "упр.срб"

I don't think that's quite intentional, probably just a side-effect of how it's been implemented (and should probably be made more consistent, or at least documented).

Anyway, if compatibility with x/net/publicsuffix is a goal, the easiest way to replicate this behaviour would probably be to run idna.ToASCII when generating the rules (which is pretty much what x/net/publicsuffix does).

[weppos]

Thanks for the ticket @jsha @PatF

Historically, the PSL never included Punycode entries. Instead, only Unicode values are stored (althought I frankly don't recall by heard the reason of the decision, assuming there was one).

So far, I've refrained from adding U/A conversion in most of my libraries, primarily because of the extra-overhead (and lack of support for good IDNA libraries).

However, Go provides a pretty good IDNA library and I may reconsider providing native support for both A/U directly in the library.

If I understood correctly, Let's Encrypt is currently storing and handling names as A labels. Is it correct?

@jsha I'm not thrilled by 2), mostly because it's exactly the opposite behavior of what the PSL is doing. 3) is a potentially solution, although I see a value in (potentially) not having to determine whether the input is A or U (and leave it up to the library). Of course, assuming the overhead is reasonable.

On other words, I'm inclined to consider having:

• a default version that works with U as today
• a potential helper methods that performs the conversion and/or
• a single convenient method (in addition to Find) that will automagically determine if A/U and deal with that

What do you think?

[weppos]

Anyway, if compatibility with x/net/publicsuffix is a goal, the easiest way to replicate this behaviour would probably be to run idna.ToASCII when generating the rules (which is pretty much what x/net/publicsuffix does).

It is, as long as it's not a bug. That seems more a bug than a feature to me. I have to investigate it, but if that's the case I will likely not support the buggy version.

In fact, the x/net/publicsuffix comes essentially as an handy transition mechanism (and I bet most of the tools transitioning are not fully supporting IDN).

[jsha]

Historically, the PSL never included Punycode entries. Instead, only Unicode values are stored (althought I frankly don't recall by heard the reason of the decision, assuming there was one).

I was aware of this, but thinking about it some more it's surprising to me. The IDN RFCs indicate that U-labels are intended for presentation, while A-labels should be used for under-the-hood processing. To me, this feels like a historic accident. But perhaps it was for a good reason!

If I understood correctly, Let's Encrypt is currently storing and handling names as A labels. Is it correct?

Correct.

I'm inclined to consider having:
a default version that works with U as today
a potential helper methods that performs the conversion and/or
a single convenient method (in addition to Find) that will automagically determine if A/U and deal with that

I'm in favor of (1) and (2). (3) seems unnecessarily risky. Callers should know whether they are handling A or U, and choose the appropriate function.

[weppos]

I was aware of this, but thinking about it some more it's surprising to me. The IDN RFCs indicate that U-labels are intended for presentation, while A-labels should be used for under-the-hood processing. To me, this feels like a historic accident. But perhaps it was for a good reason!

That's a reasonable feedback, and in fact I must say that we do the same at DNSimple and in pretty much most of the environments where I worked with names.

In the light of that, switching to A name by default may be logical, although the list is Unicode. It's also true that the internal list representation is irrelevant, as I pre-process it in any case.

I need to take a couple of days to experiment a little bit on which internal representation may be more appropriate. I'd also have a quick check with @sleevi and @gerv (which have an extensive experience on how Chromium/Firefox browsers handles the A/U transformation) to see if they have some specific hint (in addition to a possible explanation of why we store Unicode values).

@jsha I see the issue is currently causing you a bug in the rate-limiting. Is it if I take a few days to research the issue, or do you need a super quick fix? In case, you can merge letsencrypt/boulder#2278 and I'll happily provide a patch once publicsuffix-go is updated. Does it work for you?

[jsha]

I'm willing to a wait a few days while you research the issue. The rate limiting issue makes things too strict rather than too lax, and I haven't heard from people who are having problems with it. Thanks!

[gerv]

Why we chose U-label as the default (the A-label is in a comment) is lost in the mists of history, but perhaps it was because we thought that was the "proper" version of the name, and the A-label is an implementation detail. Anyway, as there are lossless conversions between the two, it doesn't seem like a big deal.

The two namespaces also don't overlap - it's not permitted to have a U-label of any sort, even an ASCII-only one, with hyphens in the third and fourth position. So it would be entirely reasonable (I'd say) for a smart API to take either, work out which, do the lookup, and return the results in the form submitted.

But if you don't want to do that, any of these other fixes are fine. But I don't think we'll be changing the PSL.

Gerv

[sleevi]

On Wednesday, October 26, 2016, Gervase Markham notifications@github.com
wrote:

Why we chose U-label as the default (the A-label is in a comment) is lost
in the mists of history, but perhaps it was because we thought that was the
"proper" version of the name, and the A-label is an implementation detail.
Anyway, as there are lossless conversions between the two, it doesn't seem
like a big deal.

IDNA2003 vs IDNA2008 was the concern at the time, and the fact that a given
name can, in theory, have two separate representations. At the time,
Mozilla was trying to keep its flexibility - at least based on past code I
can tell from around that time.

I haven't heard any substantial movement to IDNA2008 for quite some time.
Perhaps it's not happening? Or did it already happen and I missed it?

The two namespaces also don't overlap - it's not permitted to have a
U-label of any sort, even an ASCII-only one, with hyphens in the third and
fourth position. So it would be entirely reasonable (I'd say) for a smart
API to take either, work out which, do the lookup, and return the results
in the form submitted.

But if you don't want to do that, any of these other fixes are fine. But I
don't think we'll be changing the PSL.

Gerv

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#31 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABayJ6W9p1v4YdBIJYmYUJ2pd-u4DIzBks5q32cfgaJpZM4KfIXf
.

[gerv]

We switched 8 months ago, in Firefox 45:
https://bugzilla.mozilla.org/show_bug.cgi?id=1218179

Feel free to join us :-)
https://bugs.chromium.org/p/chromium/issues/detail?id=505262

[jsha]

FWIW, the current ACME spec requires IDNA2008: https://ietf-wg-acme.github.io/acme/#rfc.section.6.1.4.

[sleevi]

@gerv I'll poke jshin on that, but I think it would at least highlight a 'possibility' where the U-label form is valuable - to allow consistent decomposition to the A-label in both IDNA2003 and IDNA2008 apps. Of course, I wish we'd just go with IDNA2008.

Perhaps an interim solution is convert to A-label across the board (which will certainly break Chrome's PSL implementation, and presumably Firefox, but we can fix both of those easily) in IDNA2008, and then perhaps add a consistency check as part of the PSL's tests to make sure the IDNA2003 and 2008 representations are consistent, until the PSL full throws IDNA2003 under the bus?

[gerv]

Converting to A-label would presumably break everyone's implementation? What would be the advantages of doing that?

[sleevi]

@gerv - As jsha@ mentioned, it would allow applications to deal only with A-Labels when dealing with the PSL, without having to worry about the IDNA transformation. I certainly know that some of our mobile applications that don't support U-Labels don't want to carry the weight of an IDNA translation, to keep download size small.

[weppos]

@sleevi @gerv I agree that changing the list now may potentially break implementation. In fact, we can even consider providing 2 alternate sources if we decide to go with the transition, and automatically convert one into another.

It's also worth to mention that consumers can perform the translation on their own, pre-processing the file.

In fact, my initial research goal is to understand which representation is better for the specific case of this library. If it happens to be A-labels, I'll be happy to add an extra step in the pre-processing that converts it into A-label strings.
https://github.com/weppos/publicsuffix-go/blob/master/cmd/gen/gen.go#L60

I can check directly, but you may save me some time. Given that Chrome/Firefox are pre-processing the list, do you know if the data is stored in A-labels or U-labels?