UnicodeEncodeError: 'ascii' codec can't encode character '\u2019' in position 1884: ordinal not in range(128)

[josch]

Hi,

this is with alot from git on Debian stable (Trixie) with this email (i gzipped it):

spam.gz

The following traceback appears:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    from alot.__main__ import main; main()
                                    ~~~~^^
  File "/home/josch/git/alot/alot/__main__.py", line 142, in main
    UI(dbman, cmdstring, EVENT_LOOP)
    ~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josch/git/alot/alot/ui.py", line 145, in __init__
    self.mainloop.run()
    ~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/urwid/event_loop/main_loop.py", line 337, in run
    self._run()
    ~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/urwid/event_loop/main_loop.py", line 439, in _run
    self.event_loop.run()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/urwid/event_loop/twisted_loop.py", line 241, in run
    raise exc.with_traceback(exc.__traceback__)
  File "/usr/lib/python3/dist-packages/urwid/event_loop/twisted_loop.py", line 257, in wrapper
    rval = f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/urwid/event_loop/main_loop.py", line 218, in cb
    callback(self, user_data)
    ~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/josch/git/alot/alot/ui.py", line 626, in clear
    self.clear_notify(msgs)
    ~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/josch/git/alot/alot/ui.py", line 523, in clear_notify
    self.update()
    ~~~~~~~~~~~^^
  File "/home/josch/git/alot/alot/ui.py", line 667, in update
    self.mainloop.draw_screen()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/urwid/event_loop/main_loop.py", line 667, in draw_screen
    canvas = self._topmost_widget.render(self.screen_size, focus=True)
  File "/usr/lib/python3/dist-packages/urwid/widget/widget.py", line 112, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/widget/attr_map.py", line 158, in render
    canv = self._original_widget.render(size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/widget/widget.py", line 112, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/widget/frame.py", line 481, in render
    body = self.body.render((maxcol, maxrow - ftrim - htrim), focus and self.focus_part == "body")
  File "/home/josch/git/alot/alot/buffers/buffer.py", line 19, in render
    return self.body.render(size, focus)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urwid/widget/widget.py", line 112, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/lib/python3/dist-packages/urwid/widget/listbox.py", line 712, in render
    middle, top, bottom = self.calculate_visible((maxcol, maxrow), focus=focus)
                          ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urwid/widget/listbox.py", line 547, in calculate_visible
    next_pos, pos = self._body.get_next(pos)
                    ~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/josch/git/alot/alot/walker.py", line 46, in get_next
    return self._get_at_pos(start_from + self.direction)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josch/git/alot/alot/walker.py", line 72, in _get_at_pos
    widget = self._get_next_item()
  File "/home/josch/git/alot/alot/walker.py", line 85, in _get_next_item
    next_widget = self.containerclass(next_obj, **self.kwargs)
  File "/home/josch/git/alot/alot/widgets/search.py", line 26, in __init__
    self.rebuild()
    ~~~~~~~~~~~~^^
  File "/home/josch/git/alot/alot/widgets/search.py", line 60, in rebuild
    width, part = build_text_part(partname, self.thread,
                  ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
                                  self.structure[partname])
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josch/git/alot/alot/widgets/search.py", line 145, in build_text_part
    content = prepare_string(name, thread, maxw)
  File "/home/josch/git/alot/alot/widgets/search.py", line 212, in prepare_string
    s = content(thread)
  File "/home/josch/git/alot/alot/widgets/search.py", line 187, in prepare_content_string
    lastcontent = ' '.join(m.get_body_text() for m in msgs)
  File "/home/josch/git/alot/alot/widgets/search.py", line 187, in <genexpr>
    lastcontent = ' '.join(m.get_body_text() for m in msgs)
                           ~~~~~~~~~~~~~~~^^
  File "/home/josch/git/alot/alot/db/message.py", line 287, in get_body_text
    return extract_body_part(self.get_mime_part())
  File "/home/josch/git/alot/alot/db/utils.py", line 499, in extract_body_part
    rendered_payload = render_part(
        body_part,
        **{'field_key': 'view'} if body_part.get_content_type() == 'text/plain'
        else {})
  File "/home/josch/git/alot/alot/db/utils.py", line 345, in render_part
    raw_payload = remove_cte(part)
  File "/home/josch/git/alot/alot/db/utils.py", line 436, in remove_cte
    bp = quopri.decodestring(payload.encode('ascii'))
                             ~~~~~~~~~~~~~~^^^^^^^^^
UnicodeEncodeError: 'ascii' codec can't encode character '\u2019' in position 1884: ordinal not in range(128)

I'd like to submit a fix but I'm unsure why a conversion to ascii is necessary here in the first place?

[lucc]

Can you try #1698?

[josch]

Interestingly I'm unable to reproduce the full crash from my initial post. It only now says at the bottom "'ascii' codec can't encode character '\u2019' in position 1884: ordinal not in range(128)" but does not completely crash anymore. No idea what changed.

But yes, that message is gone with the changes from #1698.