pyyaml does not support literals in unicode over codepoint 0xffff

[kitterma]

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=806826

the yaml spec says that

“The allowed character range explicitly excludes the surrogate
block #xD800-#xDFFF, DEL #x7F, the C0 control block #x0-#x1F
(except for #x9, #xA, and #xD), the C1 control block #x80-#x9F,
#xFFFE, and #xFFFF.”

however pyyaml has chosen to negate that check and apply it to only
plane 0. This means that any yaml document that contains unicode
literals in higher planes will fail to parse (and, on output, use the
rather unfriendly \Uxxxxxxxx format).

The attached patch fixes this in a minimally intrusive way, by
extending the checks to cover the additional codepoints where
appropriate. A better fix would be to use the check as the spec
specifies it, but that would be a bigger change.

Index: pyyaml-3.11/lib/yaml/emitter.py

--- pyyaml-3.11.orig/lib/yaml/emitter.py
+++ pyyaml-3.11/lib/yaml/emitter.py
@@ -8,9 +8,13 @@

all = ['Emitter', 'EmitterError']

+import sys
+
from error import YAMLError
from events import *

+has_ucs4 = sys.maxunicode > 0xffff
+
class EmitterError(YAMLError):
pass

@@ -701,7 +705,8 @@ class Emitter(object):
line_breaks = True
if not (ch == u'\n' or u'\x20' <= ch <= u'\x7E'):
if (ch == u'\x85' or u'\xA0' <= ch <= u'\uD7FF'

•                    or u'\uE000' <= ch <= u'\uFFFD') and ch != u'\uFEFF':
  

•                    or u'\uE000' <= ch <= u'\uFFFD'
  

•                    or ((not has_ucs4) or (u'\U00010000' <= ch < u'\U0010ffff'))) and ch != u'\uFEFF':
               unicode_characters = True
               if not self.allow_unicode:
                   special_characters = True
  

  Index: pyyaml-3.11/lib/yaml/reader.py

  --- pyyaml-3.11.orig/lib/yaml/reader.py
  +++ pyyaml-3.11/lib/yaml/reader.py
  @@ -19,7 +19,9 @@ all = ['Reader', 'ReaderError']

  from error import YAMLError, Mark

-import codecs, re
+import codecs, re, sys
+
+has_ucs4 = sys.maxunicode > 0xffff

class ReaderError(YAMLError):

@@ -134,7 +136,10 @@ class Reader(object):
self.encoding = 'utf-8'
self.update(1)

• NON_PRINTABLE = re.compile(u'[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]')

• if has_ucs4:

•    NON_PRINTABLE = re.compile(u'[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD\U00010000-\U0010ffff]')
  

• else:

•    NON_PRINTABLE = re.compile(u'[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]')
  

  def check_printable(self, data):
  match = self.NON_PRINTABLE.search(data)
  if match:
  Index: pyyaml-3.11/lib3/yaml/emitter.py

  --- pyyaml-3.11.orig/lib3/yaml/emitter.py
  +++ pyyaml-3.11/lib3/yaml/emitter.py
  @@ -698,7 +698,8 @@ class Emitter:
  line_breaks = True
  if not (ch == '\n' or '\x20' <= ch <= '\x7E'):
  if (ch == '\x85' or '\xA0' <= ch <= '\uD7FF'

•                    or '\uE000' <= ch <= '\uFFFD') and ch != '\uFEFF':
  

•                    or '\uE000' <= ch <= '\uFFFD'
  

•                    or '\U00010000' <= ch < '\U0010ffff') and ch != '\uFEFF':
               unicode_characters = True
               if not self.allow_unicode:
                   special_characters = True
  

  Index: pyyaml-3.11/lib3/yaml/reader.py

  --- pyyaml-3.11.orig/lib3/yaml/reader.py
  +++ pyyaml-3.11/lib3/yaml/reader.py
  @@ -134,7 +134,7 @@ class Reader(object):
  self.encoding = 'utf-8'
  self.update(1)

• NON_PRINTABLE = re.compile('[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD]')

• NON_PRINTABLE = re.compile('[^\x09\x0A\x0D\x20-\x7E\x85\xA0-\uD7FF\uE000-\uFFFD\U00010000-\U0010ffff]')
  def check_printable(self, data):
  match = self.NON_PRINTABLE.search(data)
  if match:

[mortoray]

Any updates on this? It's a significant limitation for loading Unicode YAML files (since all emoji are > 0xFFFF and commonly used now)

[adamchainz]

+1, had a crash on encountering an emoji someone wrote in a YAML config file

[samdmarshall]

Going to add my voice to this as wanting to see this addressed as well.

[peterkmurphy]

All, Okay. I want to write the code to address this defect. I guess I need to write unit tests as well. Could anyone give me some heads up on how to get Travis working? Thanks and best regards, Peter

…

On Sat, May 6, 2017 at 9:18 PM, Samantha Marshall ***@***.***> wrote: Going to add my voice to this as wanting to see this addressed as well. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#25 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-AtWWVYC7oUON5AgNvhvwRUkfl_4khks5r3FcagaJpZM4HGN9Z> .

-- Email: peterkmurphy@gmail.com WWW: http://www.pkmurphy.com.au/

[samdmarshall]

@peterkmurphy is the patch in the original post (abit poorly formatted for display on github) not sufficient for submitting a PR?

[peterkmurphy]

Samantha, It looks like the code has been done for me! Well, that's good. But I want to do some testing. What I've done is make a fork of the code base at: https://github.com/peterkmurphy/pyyaml I seem to have got Travis working as well (to my surprise - it was as easy as pie!). So if the code works with all the environments in store (Python 2.6, 2.7, 3.3 to 3.5 and PyPy), then we can say that the patch should be good to go. Give me a day or so. I hope that helps. Best regards, Peter

…

On Sat, May 6, 2017 at 10:09 PM, Samantha Marshall ***@***.*** > wrote: @peterkmurphy <https://github.com/peterkmurphy> is the patch in the original post (abit poorly formatted for display on github) not sufficient for submitting a PR? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-AtWgxRC9WswNchXw4PA3EzB47haBUks5r3GMLgaJpZM4HGN9Z> .

-- Email: peterkmurphy@gmail.com WWW: http://www.pkmurphy.com.au/

[peterkmurphy]

All, A heads up for everybody. According to Travis, the code passes the _existing_ tests (all 1264 of them) in Python 2.x, 3.x and PyPy. But since none of the tests were written with extended Unicode in mind, I think I better write some, shouldn't I? That shouldn't stop people trying my fork with "interesting" text - one containing emojis or Cuneiform or Tengwar. But I wouldn't consider it completely tested yet. Best regards, Peter

…

On Sun, May 7, 2017 at 9:05 AM, Peter Murphy ***@***.***> wrote: Samantha, It looks like the code has been done for me! Well, that's good. But I want to do some testing. What I've done is make a fork of the code base at: https://github.com/peterkmurphy/pyyaml I seem to have got Travis working as well (to my surprise - it was as easy as pie!). So if the code works with all the environments in store (Python 2.6, 2.7, 3.3 to 3.5 and PyPy), then we can say that the patch should be good to go. Give me a day or so. I hope that helps. Best regards, Peter On Sat, May 6, 2017 at 10:09 PM, Samantha Marshall < ***@***.***> wrote: > @peterkmurphy <https://github.com/peterkmurphy> is the patch in the > original post (abit poorly formatted for display on github) not sufficient > for submitting a PR? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#25 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AA-AtWgxRC9WswNchXw4PA3EzB47haBUks5r3GMLgaJpZM4HGN9Z> > . > -- Email: ***@***.*** WWW: http://www.pkmurphy.com.au/

-- Email: peterkmurphy@gmail.com WWW: http://www.pkmurphy.com.au/

[adamchainz]

@peterkmurphy yes more tests!

[adamchainz]

yaml.load('👍')

[peterkmurphy]

All, A heads up on the issue. The patch seems to work with some tests I've run. For example, if I have the following file: 😀😁😂😃😄😅😆😇 😈😉😊😋😌😍😎😏 😐😑😒😓😔😕😖😗 😘😙😚😛😜😝😞😟 😠😡😢😣😤😥😦😧 😨😩😪😫😬😭😮😯 😰😱😲😳😴😵😶😷 😸😹😺😻😼😽😾😿 🙀🙁🙂🙃🙄🙅🙆🙇 🙈🙉🙊🙋🙌🙍🙎🙏 Then I can use PyYAML to load it and unload it. The problem I am having is with the testing code, which will need a lot of unwrangling. There are some frankly ... "bizarre" assumptions in what won't work in parseable code, except it sometimes will be parseable by accident. For example: def test_unicode_input_errors(unicode_filename, verbose=False): data = open(unicode_filename, 'rb').read().decode('utf-8') for input in [data.encode('latin1', 'ignore'), # <--- Look at this! data.encode('utf-16-be'), data.encode('utf-16-le'), codecs.BOM_UTF8+data.encode('utf-16-be'), codecs.BOM_UTF16_BE+data.encode('utf-16-le'), codecs.BOM_UTF16_LE+data.encode('utf-8')+'!']: try: yaml.load(input) except yaml.YAMLError, exc: if verbose: print exc else: raise AssertionError("expected an exception") The idea: let's cause some bizarre combinations of byte sequences, attempt to parse it, and if it _doesn't_ throw a YAMLError, raise an exception. Except that when one does data.encode('latin1', 'ignore') on data, one results in ten line breaks, which is happily parseable as YAML. So no exception raised, so AssertionError. What should I do in this case - remove test_unicode_input_errors from the PyYaml testing code? Yes, the number of tests will go down, which is generally not a good thing, but if the tests are based on dodgy assumptions... Peter

…

On Mon, May 8, 2017 at 5:12 PM, Adam Johnson ***@***.***> wrote: yaml.load('👍') — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-AtenATgNrySHLiA_wZsO-OLVgeMKeks5r3sBNgaJpZM4HGN9Z> .

-- Email: peterkmurphy@gmail.com WWW: http://www.pkmurphy.com.au/

[adamchainz]

Can you open a PR so it's easier to review?