gatsby-source-wordpress fails to build site

[scruffydan]

I am trying to build my first gatsby powered site, using my current WordPress site (1000 posts and almost 3000 pictures) as a source for gatsby and I cannot get it to build. So far I have just the basic site generated by gatsby new and the gatsby-source-wordpress plugin. My gatsby site is published here https://github.com/scruffydan/MindofDan

Description

gatsby develop doesn't complete

This is what I get:

=END PLUGIN=====================================: 33804.250ms
error Plugin gatsby-source-wordpress returned an error


  TypeError: Cannot read property 'id' of undefined

  - normalize.js:275
    [MindofDan]/[gatsby-source-wordpress]/normalize.js:275:9

  - Array.map

  - normalize.js:270 Object.exports.mapElementsToParent
    [MindofDan]/[gatsby-source-wordpress]/normalize.js:270:19

  - gatsby-node.js:142 _callee$
    [MindofDan]/[gatsby-source-wordpress]/gatsby-node.js:142:34

  - next_tick.js:61 process._tickCallback
    internal/process/next_tick.js:61:11


⠠ source and transform nodes

I am using verboseOutput: true and concurrentRequests: 10.

It just hangs at source and transform nodes and never completes. I have left it running for 12 hours.

However I don't always get the TypeError: Cannot read property 'id' of undefined error, sometimes it just hangs at source and transform nodes.

Steps to reproduce

Clone the repo https://github.com/scruffydan/MindofDan
run npm install
run gatsby develop

Expected result

The site should build

Actual result

gatsby hangs and doesn'tcomplete

Environment

• Gatsby version (npm list gatsby): gatsby@1.9.259
• gatsby-cli version (gatsby --version): 1.1.52
• Node.js version: 10.1.0
• Operating System: macOS 10.13.4

File contents (if changed)

gatsby-config.js:

module.exports = {
  siteMetadata: {
    title: 'Gatsby Default Starter',
  },
  plugins: [
	'gatsby-plugin-react-helmet',
  	{
		resolve: 'gatsby-source-wordpress',
		options: {
			baseUrl: 'mind.ofdan.ca',
			protocol: 'https',
			concurrentRequests: 10,
			hostingWPCOM: false,
			useACF: false,
			verboseOutput: true
		}
	}
  ]
}

package.json:

{
  "name": "mind.of.dan",
  "description": "Mind of Dan built with Gatsby",
  "version": "0.1.0",
  "author": "Dan Moutal",
  "dependencies": {
    "gatsby": "^1.9.259",
    "gatsby-link": "^1.6.44",
    "gatsby-plugin-react-helmet": "^2.0.11",
    "gatsby-source-wordpress": "^2.0.84",
    "react-helmet": "^5.2.0"
  },
  "keywords": [
    "gatsby",
    "WordPress"
  ],
  "license": "MIT",
  "scripts": {
    "build": "gatsby build",
    "develop": "gatsby develop",
    "format": "prettier --write 'src/**/*.js'",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "devDependencies": {
    "prettier": "^1.12.1"
  }
}

gatsby-node.js: N/A
gatsby-browser.js: N/A
gatsby-ssr.js: N/A

Thanks in advance

[KyleAMathews]

Can you try excluding some of the WP API routes?

[pieh]

After adding some debug output:

Parent element not found: wordpress__wp_comments id: 6688 parent: 6681
Parent element not found: wordpress__wp_comments id: 6687 parent: 6681
Parent element not found: wordpress__wp_comments id: 6686 parent: 6681
Parent element not found: wordpress__wp_comments id: 6685 parent: 6681
Parent element not found: wordpress__wp_comments id: 6684 parent: 6681
Parent element not found: wordpress__wp_comments id: 6683 parent: 6681
Parent element not found: wordpress__wp_comments id: 6682 parent: 6681

I will add handling of that soonish.

Getting stuck without output would almost certainly be happening while pulling images, some kind of error that we don't handle (or handle correctly), will need to investigate more

[scruffydan]

Thanks for looking into my issues

What is the best/easiest way to try out the changes in these new commits?

[pieh]

You'll have to wait for next npm publish - so probably few days. I'll notify here when it's done, so you could update your packages

Alternatively You could try to apply changes to your local node_modules packages, but be prepared for hardly readable transpiled scripts there.

[scruffydan]

I'll wait.

Thanks!

[scruffydan]

Not sure of a new version has been published but I was able to run gatsby develop by excluding the comments end point. So the problem is likely there.

I also looked at my WordPress database and confirmed that there is a comment with an ID of 6681 Not sure how to square that with the debug output, but comment 6681 is a ping back so maybe thats it

[scruffydan]

As test I removed the comment type field for comment 6681 so Wordpress would no longer thinks its a pingback and now even with the comments API endpoint re-enabled it works.

Looks like Gatsby doesn't like pingbacks as parents. Not sure if the issue lies with the WordPress API or gatsby, but the end result isn't a pleasant experience

Regardless it works now. Time for me to figure out how to build something nice:)

[maxbaun]

I was able to fix this issue by adjusting the amount of concurrent items were in the queue in the gatsby-source-filesystem plugin:

gatsby/packages/gatsby-source-filesystem/src/create-remote-file-node.js

Line 90 in 15cdf2c

concurrent: 200,

It would be nice to have an option in the gatsby-source-filesystem to adjust the amount of concurrent requests that are in the queue. That way on a lesser powerful machine, so the node process doesn't lock up.

Changing it between 1-50 concurrent requests fixed this for me.

[scruffydan]

I thought adding concurrentRequests: 10 to the gatsby-source-wordpress config limited the concurrent requests?

[pieh]

@scruffydan there couple request queues running under the hood - wordpress plugin has queue for REST endpoints (controlled by concurrentRequests) and filesystem has separate queue for file downloading (which is currently hardcoded to 200)