fix: Preserve Unicode whitespace in inline formatting tags

[jhogstrom]

Problem

Unicode whitespace characters (like   / \xa0) inside inline formatting tags are being stripped during HTML-to-Markdown conversion, causing words to run together in the output.

Example

Confluence HTML:

property<em>&nbsp;JungerRoot</em> .

Expected markdown:

property *JungerRoot* .

Actual markdown (before fix):

property*JungerRoot* .

❌ Missing space - words run together!

Impact

• 2,600+ instances in one medium-sized Confluence space alone (642 leading nbsp, 1991 trailing nbsp)
• Affects all inline formatting tags: <em>, <strong>, <code>, <i>, <b>
• Makes exported documentation harder to read
• Confluence commonly uses   for spacing control in formatted text

Root Cause

1. BeautifulSoup correctly converts HTML entities:   → \xa0 (non-breaking space Unicode character)
2. markdownify's chomp() function handles leading/trailing whitespace:
   • Regular space (' '): Moved to prefix/suffix ✅
   • Unicode whitespace (\xa0, \u2000-\u200a, etc.): Stripped entirely ❌
3. Result: Space is completely lost

Evidence:

from markdownify import chomp

# Regular space - preserved in prefix
chomp(' text')  # → (' ', '', 'text')  ✅

# Non-breaking space - lost completely
chomp('\xa0text')  # → ('', '', 'text')  ❌

Solution

Normalize Unicode whitespace to regular ASCII spaces before passing text to parent converter methods. This allows markdownify's chomp() to handle the spaces correctly.

Implementation

1. Added helper method _normalize_unicode_whitespace(text: str) -> str

   • Converts all Unicode whitespace to regular spaces
   • Preserves semantic whitespace (\n, \r, \t)
   • Handles: \xa0 (nbsp), \u2000-\u200a (various spaces), \u2028 (line separator), etc.

2. Overrode 5 converter methods to normalize text parameter:

   • convert_em()
   • convert_strong()
   • convert_code()
   • convert_i()
   • convert_b()

3. Comprehensive unit tests: 17 test cases covering:

   • Leading/trailing nbsp in all tag types
   • Multiple nbsp in sequence
   • Other Unicode whitespace (EM SPACE, THIN SPACE)
   • Preserves newlines and tabs
   • Real-world Confluence examples

Code Changes

File: confluence_markdown_exporter/confluence.py

def _normalize_unicode_whitespace(self, text: str) -> str:
    """Normalize Unicode whitespace to regular spaces."""
    normalized = text
    for char in text:
        if char.isspace() and char not in " \n\r\t":
            normalized = normalized.replace(char, " ")
    return normalized

def convert_em(self, el: BeautifulSoup, text: str, parent_tags: list[str]) -> str:
    """Convert <em> tags, preserving spaces from Unicode whitespace entities."""
    text = self._normalize_unicode_whitespace(text)
    return super().convert_em(el, text, parent_tags)

# Similar overrides for convert_strong, convert_code, convert_i, convert_b

Testing

✅ All 17 unit tests passing

pytest tests/unit/test_nbsp_fix.py -v
# 17 passed in 0.06s

Test coverage:

• Leading nbsp: <em>&nbsp;text</em> → *text* (space preserved in context)
• Trailing nbsp: <em>text&nbsp;</em> → *text* (space preserved in context)
• Both sides: <em>&nbsp;text&nbsp;</em> → *text* (both preserved)
• All tag types: <em>, <strong>, <code>, <i>, <b>
• Real-world: property<em>&nbsp;JungerRoot</em> . → property *JungerRoot* . ✅
• Multiple nbsp: <em>&nbsp;&nbsp;text</em> → * text* (all spaces preserved)
• Unicode spaces: EM SPACE (\u2003), THIN SPACE (\u2009) normalized correctly
• Preserves newlines/tabs: <em>text\nmore</em> → unchanged
• No modification when no Unicode whitespace present

Performance Impact

None - Simple character replacement, O(n) on text content within tags only.

Backward Compatibility

• ✅ No API changes
• ✅ No configuration changes
• ✅ Output format improved (spaces now preserved)
• ✅ All existing functionality preserved
• ✅ Only affects previously broken output

Files Changed

• confluence_markdown_exporter/confluence.py (+57 lines)
• tests/unit/test_nbsp_fix.py (+177 lines, new file)

Total: +234 lines

Alternative Approaches Considered

1. Preprocess HTML globally - Would normalize all text, not just inline tags
2. Post-process markdown - Would need complex regex to avoid false positives
3. Override process_tag - Higher-level interception but more complex

Chosen approach (normalize text parameter in convert methods):

• ✅ Minimal, surgical fix
• ✅ Only affects inline formatting tags where issue occurs
• ✅ Delegates to parent class (follows existing pattern)
• ✅ Easy to test and maintain

References

• Confluence uses   extensively for formatting control
• markdownify chomp() source: https://github.com/matthewwithanm/python-markdownify/blob/develop/markdownify/__init__.py#L60-L70
• Unicode whitespace characters: https://en.wikipedia.org/wiki/Whitespace_character#Unicode

[Spenhouet]

@jhogstrom Thanks for your work!