gatsby with graphql won't load new data

[Necromant1k]

I have an issue - gatsby with graphql won't update data from database ( firebase ).

1. Server started ( npm run develop ) in my gatsby-node.js data fetched and dynamic pages created.
2. I'm going to add some rows into my database - added
3. Reloading page (where i should see new items ) => I see there only old items (no new appear). Same even if i will go to localhost:8000/___graphql There is only old data
4. Going to close connection and start server again
5. New data appear
   And i have to do it always to see new data.

Any solutions on this?
Thanks!

gatsby-node.js

const _ = require(`lodash`);
const Promise = require(`bluebird`);
const path = require(`path`);
const slug = require(`slug`);
const slash = require(`slash`);

exports.createPages = ({ boundActionCreators, graphql }) => {
  const { createPage } = boundActionCreators

  return new Promise((resolve, reject) => {
    resolve(
        graphql(
            `
            {
              allImports {
                edges {
                  node {
                    id,
                    opened,
                    unsubscribed,
                    uid,
                    totalCsvLength,
                    clicked,
                    complained,
                    delivered,
                    duplicate,
                    failed,
                    importStartedAt
                  }
                }
              }
            }
        `
        ).then(result => {
          if (result.errors) {
            reject(result.errors)
          }
          const importTemplate = path.resolve(`src/templates/import-detail.js`);
          result.data.allImports.edges.map(edge => {
            createPage({
              path: `/${slug(edge.node.id)}/`,
              component: slash(importTemplate),
              context: {
                id: edge.node.id,
              },
            })
          })
        })
    )
  })
};

gatsby-config.js

module.exports = {
  siteMetadata: {
    title: `example`,
  },
  plugins: [
      `gatsby-plugin-react-helmet`,
    {
      resolve: `gatsby-source-firebase`,
      options: {
        // point to the firebase private key downloaded
        credential: require("./config/firebase.json"),

        // your firebase database root url
        databaseURL: "https://url.firebaseio.com",

        // you can have multiple "types" that point to different paths
        types: [
          {
            type: "Imports",
            path: "imports",
          }
          ]
      }
    }
  ],
}

[dardub]

While in development mode, the server will watch for changes to files and rebuild/update accordingly.

In production, this is a static site and content is only updated when you do a fresh build and deploy to your host.

There is no way for Gatsby to know that you made changes to your database. Firebase would need to trigger an event to your CI server that you need a new build.

Another approach is to not use Firebase as a source and have it update content dynamically in your app. But then you are losing some of advantages of having static compiled content.

[dardub]

Just to add, I would make sure that using Firebase is necessary. Can you not achieve the same outcome using markdown or yaml? This would make things simple and easy to maintain.

[jtschulz]

What if there was an optional flag when restarting the server that allows you to add a query parameter that refetches from your data source? Perhaps, the server could accept post requests to trigger rebuilding the schema and pulling data, so data sources could use webhooks. Use case: our content editors want to preview their entries before publishing.

I would be interested in working on something like this, but could use a little guidance through the code, since I'm new to the project.

[jtschulz]

Any thoughts on this proposed solution? Would gatsby be amenable to one of these or some variation thereof?

[KyleAMathews]

My preferred solution is the last idea here #2271 (comment)

We'd expose a way to run commands while gatsby develop one of which would be to refresh data from sources.

[KyleAMathews]

Perhaps, the server could accept post requests to trigger rebuilding the schema and pulling data, so data sources could use webhooks

You want to run gatsby develop on a web server somewhere that's accessible to many people? Hmmm interesting 🤔

[jtschulz]

Admittedly not the greatest option, but it would be behind an oauth proxy so only users in our organization can access the site. How are other people serving up previews of their site?

[jtschulz]

I like the version of using a source plugin to add new nodes. Was not aware that was a supported use case. Perhaps I could add this as an option for gatsby-source-contentful

[bjrn]

Being able to run a gatsby develop on a web server for site previews, with data coming in from gatsby-source-contentful matches my use case exactly, @PepperTeasdale — can I help out making it happen in any way?

[jtschulz]

Sending you an email @bjrn

[tlvenn]

Being able to run a gatsby develop on a web server for site previews, with data coming in from gatsby-source-xxxx

This is also my use case with prismic instead of contentful. I dont really understand why we could not start gatsby develop with an additional flag that would force gatsby to refetch its data for the requested page.

Pushing that concern for this particular use case (preview mode) to each individual source plugin does not seem like the right approach.

You want to run gatsby develop on a web server somewhere that's accessible to many people? Hmmm interesting

I dont really see any other way to let authors preview their contents before it's getting published. And for many working with a headless CMS, this is really mandatory for adoption.

There is definitely a legitimate use case to run gatsby on a server in order to provide previews imho and it seems gatsby develop already provide 95% of what is needed.

It could probably be a good idea to introduce gatsby preview rather than passing an additional flag to gatsby develop.

[jtschulz]

I agree with @tlvenn , a gatsby preview server sounds like the ideal solution. It could offer a basic version to allow re-bootstraping before handling requests, taking the responsibility off the plugins. Even better would be to expose an API to receive webhooks from data sources and allow source plugins to update accordingly.

I took a look at the filesystem source plugin as @KyleAMathews recommended and noticed it was using chokidar to watch filesystem changes. At first I thought I could hook into onClientEntry and use EventEmitter to trigger a re-running of sourceNodes, but then realized that browser api and node api wouldn't work together that way. Any suggestions on which parts of the API I need to be looking at to make this work?

[jtschulz]

I guess ignore the latter part of the above comment, I came up with some options. @tlvenn , maybe have a look at #3054 if you'd like to join that conversation