[gatsby-plugin-sitemap] Multiple/Split Sitemap Files

[youmustfight]

Summary

When submitting sitemaps to Google Search Console, an error is thrown for sitemaps with more than 50,000 urls. It'd be great if there was an option to have multiple sitemap files generate with a 50,000 url limit per file.

This could be done with a number limit or maybe an include option (rather than just exclude)?

Basic example

Expecting sitemap-1.xml, sitemap-2.xml, etc.

  {
    resolve: `gatsby-plugin-sitemap`,
    options: {
       split: 50000,
     }
  }

or manually defined

  {
    resolve: `gatsby-plugin-sitemap`,
    options: {
       output: `/sitemap.xml`,
       exclude: ['/learn/*'],
     }
  },
  {
    resolve: `gatsby-plugin-sitemap`,
    options: {
       output: `/sitemap-learn.xml`,
       include: ['/learn/*'],
     }
  },

Motivation

For a larger website, this is a critical way to make sure all pages are crawled and indexed!

[freiksenet]

That would be a great feature indeed! Would you like to try to tackle it and create a PR?

[t-jindai]

@freiksenet @youmustfight It's interesting for me too 👀 Can I tackle it ? I'll send PR for it later.

[youmustfight]

Go for it @tomoyukikashiro! 😄 I can get to it tomorrow if you aren't able to later today

[t-jindai]

@youmustfight I'm still working it now 😅 in this branch. plz wait one moment. 👍

[t-jindai]

@youmustfight the PR was merged and shipped in gatsby@2.1.30 👍

[ghashi]

According to this article "Google: No Technical Advantage To Splitting Up Sitemaps But Useful For Debugging" it would be nice to split the sitemap by "category" (instead of sitemapSize).

Do you guys know if it is possible to split it in this way?

[Baartooo]

@ghashi it is possibility to split sitemap by category, language or anything you want using this plugin. You just have to put gatsby-plugin-sitemap config multiple times in your gatsby-config.js file and serialize each sitemap by yourself. I don't know if it's a best or even good solution, but it works. Below I paste fragment of my gatsby-config.js file. In my case I create each sitemap for each language separately. In query I filter all pages by gatsby-plugin-intl locale code to get all in specific language.

{
      resolve: `gatsby-plugin-sitemap`,
      options: {
        output: '/sitemap_en.xml',
        query: `
        {
          site {
            siteMetadata {
              siteUrl
            }
          }

          allSitePage(filter: {context: {language: {eq: "en"}}}) {
            edges {
              node {
                path
                context {
                  intl {
                    originalPath
                  }
                }
              }
            }
          }
      }`,
        serialize: ({ site, allSitePage }) => {
          const pages = [];

          allSitePage.edges.forEach(edge => {
            const { node: { path: pathWithLang, context: { intl: { originalPath } } } } = edge;

            if (!EXCLUDED_FROM_SITEMAP.includes(originalPath)) {
              console.log(`added "${pathWithLang}" to sitemap `);
              const path = originalPath === '/' ? '' : originalPath;
              const page = {
                url: `${site.siteMetadata.siteUrl}${path}/`,
                changefreq: 'daily',
                priority: 0.7,
                links: [
                  { lang: 'en', url: `${site.siteMetadata.siteUrl}${path}/` },
                  { lang: 'pl', url: `${site.siteMetadata.siteUrl}/pl${path}/` },
                  { lang: 'de', url: `${site.siteMetadata.siteUrl}/de${path}/` },
                  { lang: 'fr', url: `${site.siteMetadata.siteUrl}/fr${path}/` },
                  { lang: 'es', url: `${site.siteMetadata.siteUrl}/es${path}/` },
                  { lang: 'cs-CZ', url: `${site.siteMetadata.siteUrl}/cz${path}/` },

                  // The default in case page for user's language is not localized.
                  { lang: 'x-default', url: `${site.siteMetadata.siteUrl}${path}/` },
                ],
              };
              pages.push(page);
            }
          });
          return pages;
        },
      },
    },
    {
      resolve: `gatsby-plugin-sitemap`,
      options: {
        output: '/sitemap_pl.xml',
        query: `
        {
          site {
            siteMetadata {
              siteUrl
            }
          }
          allSitePage(filter: {context: {language: {eq: "pl"}}}) {
            edges {
              node {
                path
                context {
                  intl {
                    originalPath
                  }
                }
              }
            }
          }
      }`,
        serialize: ({ site, allSitePage }) => {
          const pages = [];

          allSitePage.edges.map(edge => {
            const { node: { path: pathWithLang, context: { intl: { originalPath } } } } = edge;

            if (!EXCLUDED_FROM_SITEMAP.includes(originalPath)) {
              console.log(`added "${pathWithLang}" to sitemap `);
              const path = originalPath === '/' ? '' : originalPath;
              const page = {
                url: `${site.siteMetadata.siteUrl}/pl${path}/`,
                changefreq: 'daily',
                priority: 0.7,
                links: [
                  { lang: 'en', url: `${site.siteMetadata.siteUrl}${path}/` },
                  { lang: 'pl', url: `${site.siteMetadata.siteUrl}/pl${path}/` },
                  { lang: 'de', url: `${site.siteMetadata.siteUrl}/de${path}/` },
                  { lang: 'fr', url: `${site.siteMetadata.siteUrl}/fr${path}/` },
                  { lang: 'es', url: `${site.siteMetadata.siteUrl}/es${path}/` },
                  { lang: 'cs-CZ', url: `${site.siteMetadata.siteUrl}/cz${path}/` },

                  // The default in case page for user's language is not localized.
                  { lang: 'x-default', url: `${site.siteMetadata.siteUrl}${path}/` },
                ],
              };
              pages.push(page);
            }
          });
          return pages;
        },
      },
    },
    {
      resolve: `gatsby-plugin-sitemap`,
      options: {
        output: '/sitemap_de.xml',
        query: `
        {
          site {
            siteMetadata {
              siteUrl
            }
          }
          allSitePage(filter: {context: {language: {eq: "de"}}}) {
            edges {
              node {
                path
                context {
                  intl {
                    originalPath
                  }
                }
              }
            }
          }
      }`,
        serialize: ({ site, allSitePage }) => {
          const pages = [];

          allSitePage.edges.map(edge => {
            const { node: { path: pathWithLang, context: { intl: { originalPath } } } } = edge;

            if (!EXCLUDED_FROM_SITEMAP.includes(originalPath)) {
              console.log(`added "${pathWithLang}" to sitemap `);
              const path = originalPath === '/' ? '' : originalPath;
              const page = {
                url: `${site.siteMetadata.siteUrl}/de${path}/`,
                changefreq: 'daily',
                priority: 0.7,
                links: [
                  { lang: 'en', url: `${site.siteMetadata.siteUrl}${path}/` },
                  { lang: 'pl', url: `${site.siteMetadata.siteUrl}/pl${path}/` },
                  { lang: 'de', url: `${site.siteMetadata.siteUrl}/de${path}/` },
                  { lang: 'fr', url: `${site.siteMetadata.siteUrl}/fr${path}/` },
                  { lang: 'es', url: `${site.siteMetadata.siteUrl}/es${path}/` },
                  { lang: 'cs-CZ', url: `${site.siteMetadata.siteUrl}/cz${path}/` },

                  // The default in case page for user's language is not localized.
                  { lang: 'x-default', url: `${site.siteMetadata.siteUrl}${path}/` },
                ],
              };
              pages.push(page);
            }
          });
          return pages;
        },
      },
    },