[WEB-4447] Add MDX to Markdown transpilation with content negotiation #3000
+570
−6
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing touches🧪 Generate unit tests (beta)
Comment |
kennethkalmer
temporarily deployed
to
ably-docs-web-4447-tran-bjkybb
kennethkalmer
had a problem deploying
to
ably-docs-web-4447-tran-mqes5f
kennethkalmer
temporarily deployed
to
ably-docs-web-4447-tran-mqes5f
kennethkalmer
temporarily deployed
to
ably-docs-web-4447-tran-mqes5f
Implements automatic transpilation of MDX documentation files to Markdown format during the Gatsby build process. The transpiled Markdown files are generated in the public/docs/ directory alongside HTML output, making documentation accessible to LLMs and other text-based tools.
Key features:
- Transpiles all MDX files under src/pages/docs/ to .md format
- Removes frontmatter except title (converted to # heading)
- Removes import/export statements and script tags
- Replaces template variables ({{API_KEY}}, {{RANDOM_CHANNEL_NAME}})
- Preserves JSX components and code blocks as-is
- Smart path mapping: index.mdx → parent.md, file.mdx → file.md
Successfully transpiles 211 documentation pages in ~5 seconds during onPostBuild hook.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
Implements HTTP content negotiation in nginx to serve markdown versions of documentation pages based on the Accept header. This allows clients (like LLMs and text-based tools) to request markdown format while browsers continue to receive HTML by default. Key features: - Serves markdown for Accept: text/markdown, application/markdown, or text/plain - Maintains backward compatibility (HTML is default) - Works with existing authentication system - Supports both index and non-index file paths - No performance impact (uses nginx map blocks) Content negotiation behavior: - /docs/channels with Accept: text/markdown → serves docs/channels.md - /docs/channels with Accept: text/html → serves docs/channels/index.html - /docs/channels (browser default) → serves docs/channels/index.html - /docs/channels.md (direct access) → serves docs/channels.md Implementation: - Added text/markdown MIME type to config/mime.types - Added text/markdown to gzip_types for compression - Created map blocks to detect Accept header preferences - Updated location blocks to use content-negotiated file paths - Fallback to HTML when markdown doesn't exist All 211 markdown documentation files are now accessible via content negotiation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Adds comprehensive CI test suite to verify content negotiation works correctly for all Accept header scenarios. The test suite validates that nginx serves markdown or HTML based on the Accept header, with proper fallback behavior. Test coverage (16 test cases): - Basic content negotiation (6 tests): text/markdown, application/markdown, text/plain, text/html, */* - Browser behavior (2 tests): Complex Accept headers, HTML priority when listed first - Direct access (2 tests): .md and .html file access - Path variations (3 tests): Index paths, non-index paths, nested paths - Edge cases (3 tests): 404 handling, fallback behavior, non-docs paths Also fixes nginx map priority order to ensure anchored patterns (^text/html, ^text/plain) are evaluated before wildcard patterns. This ensures "text/html, text/markdown" correctly serves HTML instead of markdown. Changes: - Created bin/assert-content-negotiation.sh with run_test() helper function - Integrated test into CircleCI test-nginx job - Reordered nginx map patterns for correct priority matching - All 16 tests passing locally 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
kennethkalmer
force-pushed
the
WEB-4447-transpile-mdx-to-md
branch
from
de0ba4d to
f227aa4
Compare
m-hulbert
approved these changes
Dec 4, 2025
Contributor
m-hulbert
left a comment
m-hulbert
left a comment
There was a problem hiding this comment.
This LGTM (under the assumption that links contain ably-dev because it's a review app. There's a few really minor changes we could address, but I think that's more source-based than the .md generation. 🚀
Member
Author
|
@m-hulbert thanks! let me get the code ready for team review then :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements automatic MDX-to-Markdown transpilation during the Gatsby build process and adds nginx content negotiation to serve markdown documentation to LLMs and text-based tools while maintaining HTML for browsers.
Related to ticket: WEB-4447
What's Changed
1. MDX to Markdown Transpilation
src/pages/docs/to.mdformat during buildonPostBuildhook# Titleheading){{API_KEY}}→your-api-key,{{RANDOM_CHANNEL_NAME}}→your-channel-name)channels/index.mdx→public/docs/channels.md2. Nginx Content Negotiation
text/markdownMIME type toconfig/mime.typesAccept: text/markdown,application/markdown, ortext/plain3. Comprehensive Test Suite
bin/assert-content-negotiation.shwith 16 test casestest-nginxjobContent Negotiation Behavior
/docs/channels(default)/docs/channels+Accept: text/markdown/docs/channels+Accept: application/markdown/docs/channels+Accept: text/plain/docs/channels+ browser Accept header/docs/channels.md(direct)Files Changed
Created:
data/onPostBuild/transpileMdxToMarkdown.ts(246 lines) - Transpilation logicbin/assert-content-negotiation.sh(138 lines) - Test suiteModified:
data/onPostBuild/index.ts- Add transpilation to build pipelineconfig/mime.types- Add text/markdown MIME typeconfig/nginx.conf.erb- Content negotiation maps and location blocks.circleci/config.yml- Add test to CI pipelineTest Plan
public/docs/with correct paths.mdaccess works🤖 Generated with Claude Code