Implement wagtail-newsletter #519

mgax · 2025-04-16T19:07:37Z

Fixes #481. This PR supersedes #492.

I was able to replicate a recent newsletter edition with very small spacing differences. I'm pretty happy with the editing experience and have somewhat tested this using Litmus. The main issue with testing is that images are served locally, so Litmus can't load them; we'd need to either deploy this to staging, or use a tunnelling tool like ngrok.

Things that I'm not so happy about:

Links in the accent block aren't styled as white text in some Outlook versions.
The vertical spacing is a bit fiddly, since I mostly had to work with padding, not margins (margins can overlap, padding can't).
~~There's no code to migrate the archive from Mailchimp just yet. I'm hoping that an AI might make writing that code not too painful.~~ Import works.
The detail view of a newsletter will load the email-formatted HTML from a variable into the shadow DOM of a<div>, so that its styling doesn't break the header and footer. I'm not sure how well this plays with search engines. The content is now rendered using the Declarative Shadow DOM syntax.
~~The import_newsletter_archive command doesn't work for TWiW issues 138 through 172.~~ Fixed.

mgax · 2025-04-28T15:13:28Z

I've added a management command to import a newsletter edition:

./manage.py import_newsletter https://mailchi.mp/wagtail/twiw-11034221 "Issue #180"

It will skip over the date and the footer, detect headings, copy images, and preserve links in the text. It doesn't copy the "Packages" section using the right block type (AccentRichTextBlock) but I'm wondering if that might be best cleaned up manually.

vossisboss · 2025-04-30T20:57:06Z

FYI Alex. I got all the packages installed and such to test your branch but I'm running into a migration error that reads:

cannot ALTER TABLE "newsletter_newsletterpage" because it has pending trigger events

Baptiste suggested that we split one of the migration files in two. Apparently Postgres doesn't like it when you try to combine certain actions together. I'm going to attempt that and see if it fixes the issue. If I have no luck, I may need you to take another look at this or help me through the fiddly migration parts.

mgax · 2025-05-01T13:41:50Z

I couldn't replicate the migration error, but I've encountered such errors in the past, and I think it's best to avoid them. I've split the migration into two steps.

laymonage · 2025-05-01T13:47:12Z

wagtailio/newsletter/migrations/0005_wagtail_newsletter_data.py

+        migrations.RemoveField(
+            model_name="newsletterpage",
+            name="body",
+        ),
+        migrations.RemoveField(
+            model_name="newsletterpage",
+            name="intro",
+        ),
+        migrations.RenameField(
+            model_name="newsletterpage",
+            old_name="content",
+            new_name="body",
+        ),


I'd recommend splitting the RemoveField operations to another migration. The problem tends to happen if you have both a data migration operation (in this case RunPython(convert_body_to_streamfield)) and a schema migration operation in a single migration. Migrations are run in a single transaction (if the backend supports DDL transactions), and the database may not allow data manipulation in a DDL transaction.

Fair point. I've also noticed that images were not getting preserved in the data migration so I've fixed that too.

Stormheg · 2025-05-09T11:58:10Z

@mgax I ran into issues with the migration script not parsing the images correctly. I've pushed a fixup commit for that, please review.

Stormheg

@mgax couple minors comments.

Impressed by the migration of the old issues (Issues <= 138). Though this is not how they originally looked, though I don't think it is super important to keep their original look. They look clean now.

The detail view of a newsletter will load the email-formatted HTML from a variable into the shadow DOM of a
, so that its styling doesn't break the header and footer. I'm not sure how well this plays with search engines.

Have you considered loading the email HTML in an iframe? That should accomplish mostly the same thing. Iframes have been around for so long it is more likely they behave nicely in the context of SEO - if that is even important.

Stormheg · 2025-05-09T12:01:39Z

wagtailio/newsletter/models.py

+
+@register_setting
+class NewsletterSettings(BaseGenericSetting):
+    footer = StreamField(NewsletterContentBlock(), blank=True, use_json_field=True)


@mgax this and the StreamField below don't require setting use_json_field anymore, see https://docs.wagtail.org/en/stable/releases/6.0.html#streamfield-no-longer-requires-use-json-field-true

Suggested change

footer = StreamField(NewsletterContentBlock(), blank=True, use_json_field=True)

footer = StreamField(NewsletterContentBlock(), blank=True)

Fixed in d00d9b0.

Stormheg · 2025-05-09T12:01:50Z

wagtailio/newsletter/models.py

+class NewsletterPage(NewsletterPageMixin, Page):
+    date = models.DateField()
+    preview = models.TextField(blank=True)
+    body = StreamField(NewsletterContentBlock(), blank=True, use_json_field=True)


Suggested change

body = StreamField(NewsletterContentBlock(), blank=True, use_json_field=True)

body = StreamField(NewsletterContentBlock(), blank=True)

Fixed in d00d9b0.

Stormheg · 2025-05-09T12:02:11Z

wagtailio/newsletter/models.py

    search_fields = Page.search_fields + [
        index.SearchField("intro"),
        index.SearchField("body"),
+        index.SearchField("body"),


Accidental double?

Suggested change

index.SearchField("body"),

Stormheg · 2025-05-09T12:10:00Z

wagtailio/newsletter/templates/newsletter/newsletter_mjml.html

+                        <p>
+                            Want to change how you receive these emails? You can
+                            <a href="*|UPDATE_PROFILE|*">update your preferences</a> or
+                            <a href="*|UNSUB|*">unsubscribe from this list</a>.
+                        </p>


Suggestion: it looks like these end up on the web page version. Shouldn't these links be hidden? They don't go anywhere.

Fixed in bec1ad0.

Stormheg · 2025-05-09T12:10:25Z

wagtailio/newsletter/templates/newsletter/newsletter_mjml.html

+                    <mj-text align="center" css-class="rich-text">
+                        <a href="{{ page.full_url }}">View this email online</a>
+                    </mj-text>


Same as below - shouldn't this be hidden on the web version of the page?

Fixed in bec1ad0.

Stormheg · 2025-05-09T12:22:16Z

wagtailio/newsletter/templates/newsletter/newsletter_page.html

+    <script>
+        (function() {
+            const container = document.getElementById("newsletter-container");
+            const shadow = container.attachShadow({mode: "open"});
+            const wrapper = document.createElement("div");
+            wrapper.innerHTML = JSON.parse(document.getElementById("email-html-data").textContent);
+            shadow.appendChild(wrapper);
+        })();
+    </script>


Suggestion: maybe there is an alternative for the shadow dom?

Issue: otherwise this should be converted to a dedicated JS script that is loaded on this page and creates the shadow DOM.

Adding inline JS is not acceptable as we're trying to add a Content Security Policy (see #468) in the future, which does not allow for inline JS

Fixed in c65e242. It turns out you can inline the shadow dom content.

mgax · 2025-05-26T14:22:28Z

Have you considered loading the email HTML in an iframe?

Getting the iframe height is notoriously difficult. Also, I suspect the search engines might index the content at the iframe's src url, which is not great.

But it turns out that you can inline the content for the shadow dom, and it seems search engine will index it: https://dev.to/stuffbreaker/seo-and-web-components-2023-edition-3l6i

mgax · 2025-05-26T14:28:56Z

I also made a import_newsletter_archive management command, which includes a hardcoded list of all the TWiW issues from when we moved to Mailchimp until now. It turns out that the import script only works from issue 173 onwards; the markup of the older issues is different. I've commented out the ones that fail to import.

Also @vossisboss, did we skip issue 163? It doesn't seem to be in my personal email archive either.

mgax · 2025-05-26T15:48:47Z

It turns out that the import script only works from issue 173 onwards; the markup of the older issues is different. I've commented out the ones that fail to import.

The importing is fixed now, I've checked all of the back issues by opening them in the browser and having a quick glance. This PR is ready for another review.

And I almost forgot: thank you @Stormheg for the thorough review and the fix to the data migration code 🙌

vossisboss · 2025-05-28T21:25:19Z

@mgax We may have skipped an issue. I wouldn't be surprised. It will just be a part of Wagtail lore now. I am testing your code right now. I pushed the Wagtail 7.0 upgrade today, so would you mind rebasing this to the latest version?

I don't have a lot more time left in my day to test this so I might pick this up again in the morning and send you my notes. I can confirm that the database issue was solved though and I got the development server to start.

mgax · 2025-06-03T14:07:22Z

I pushed the Wagtail 7.0 upgrade today, so would you mind rebasing this to the latest version?

@vossisboss I did a merge with main, since it looks like we're squashing PRs anyway. I can rebase and force-push if that makes more sense.

mgax · 2025-09-04T15:22:33Z

I had some time today and I've polished this PR. I think it's just about ready for prime time.

@vossisboss I've implemented the short list of things that you've flagged:

Buttons can now reference a page, not just a URL
Paragraphs in the "Accent Rich Text" block have no margin
New social buttons for Mastodon and Bluesky. Since Mailchimp doesn't seem to have a Mastodon icon on their CDN, I've reused the SVG icons from the wagtail.org footer (and rendered them as 80x80 PNGs with no transparency).
I have not hardcoded the footer text because I'd already implemented a way to edit it. It's under "Settings > Newsletter settings". I've now added a 10px spacing between the body and the footer.
I've checked that it's safe to duplicate pages – the newsletter Mailchimp state is not copied over.

Also, I've bumped wagtail-newsletter to the newly released v0.2.3, and updated the list of existing newsletters to import. @vossisboss please check that I've got the numbering right!

mgax added 7 commits April 15, 2025 12:48

Remove temporary containers

9b705c9

Fix delete_local_renditions

e56ecfa

Fix newsletter index template

44f9aa5

Add wagtail-newsletter dependency

6ba9de1

Set up NewsletterPage with NewsletterPageMixin

287329a

New page fields

c6d7cae

Remove old newsletter page template

3f06a77

mgax mentioned this pull request Apr 16, 2025

Feature/481 wagtail newsletter #492

Closed

6 tasks

mgax added 3 commits April 28, 2025 14:58

Migrate the existing pages and delete old fields

b61d610

Newsletter MJML

d345e4f

Import archival newsletter editions

d8f8523

mgax force-pushed the newsletter branch from 206ccfa to d8f8523 Compare April 28, 2025 15:05

mgax marked this pull request as ready for review April 28, 2025 15:14

Split migration into two files

5cea0cb

laymonage reviewed May 1, 2025

View reviewed changes

mgax added 2 commits May 1, 2025 17:00

Split off another migration

dcd056d

Fix migration of images in the body

f541765

fixup! Fix migration of images in the body

e4b19a3

Stormheg force-pushed the newsletter branch from 5894122 to e4b19a3 Compare May 9, 2025 11:59

Stormheg reviewed May 9, 2025

View reviewed changes

MR fixes

d00d9b0

mgax mentioned this pull request May 26, 2025

get_newsletter_html should accept extra context wagtail/wagtail-newsletter#80

Closed

mgax added 3 commits May 26, 2025 16:42

Include content inline with shadowrootmode=open

c65e242

Hide irrelevant content when rendering for web

bec1ad0

Single command to import newer editions

c1f1577

Fix import script to work for all issues

c96ea2d

mgax requested a review from Stormheg May 26, 2025 15:48

Merge branch 'main' into newsletter

d924532

mgax added 8 commits September 4, 2025 16:55

Merge branch 'main' into newsletter

efb3377

Bump wagtail-newsletter to version 0.2.3

7b3548a

Change button to select a page or enter a URL

8e4fd65

No space around paragraphs in accented text block

90293af

Add Mastodon and Bluesky social icons

5cdb68e

Add some spacing before the footer

53e2404

Add links to recent TWiW editions

4c670fb

Remove workaround for injecting context

adf364d

Add 2025-09-05 TWiW edition

ebc8471

	footer = StreamField(NewsletterContentBlock(), blank=True, use_json_field=True)
	footer = StreamField(NewsletterContentBlock(), blank=True)

	body = StreamField(NewsletterContentBlock(), blank=True, use_json_field=True)
	body = StreamField(NewsletterContentBlock(), blank=True)

Implement wagtail-newsletter #519

Are you sure you want to change the base?

Implement wagtail-newsletter #519

Uh oh!

Conversation

mgax commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgax commented Apr 28, 2025

Uh oh!

vossisboss commented Apr 30, 2025

Uh oh!

mgax commented May 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Stormheg commented May 9, 2025

Uh oh!

Stormheg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgax commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgax commented May 26, 2025

Uh oh!

mgax commented May 26, 2025

Uh oh!

vossisboss commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgax commented Jun 3, 2025

Uh oh!

mgax commented Sep 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mgax commented Apr 16, 2025 •

edited

Loading

mgax commented May 26, 2025 •

edited

Loading

vossisboss commented May 28, 2025 •

edited

Loading