Use bot user agent for openGraph queries

We found that some websites return opengraph information based
on the user agent.

In order to address this, using (bot) in the user agent string
may help the website behave correctly. When testing Twitter, we
found that the correct metadata is returned with the (bot) user
agent while it isn't for the default user agent.

This change additionally updates providers.json, as with the
user agent fix twitter will now return the expected openGraph
metadata.

Signed-off-by: Andrew Ryan <[email protected]>

Author: AndrewRyanChama

changelog.d/11985.misc

@@ -0,0 +1 @@
+Use bot user agent for openGraph queries.

synapse/res/providers.json

@@ -5,13 +5,11 @@
         "endpoints": [
             {
                 "schemes": [
-                    "https://twitter.com/*/status/*",
-                    "https://*.twitter.com/*/status/*",
                     "https://twitter.com/*/moments/*",
                     "https://*.twitter.com/*/moments/*"
                 ],
                 "url": "https://publish.twitter.com/oembed"
             }
         ]
     }
-]
+]

synapse/rest/media/v1/preview_url_resource.py

@@ -402,7 +402,12 @@ async def _download_url(self, url: str, output_stream: BinaryIO) -> DownloadResu
                 url,
                 output_stream=output_stream,
                 max_size=self.max_spider_size,
-                headers={"Accept-Language": self.url_preview_accept_language},
+                headers={
+                    b"Accept-Language": self.url_preview_accept_language,
+                    b"User-Agent": [
+                        "Synapse (bot; +https://github.com/matrix-org/synapse)"
+                    ],
+                },
                 is_allowed_content_type=_is_previewable,
             )
         except SynapseError: