Gatsby develop and build hang on source and transform nodes stage on large CSV file with extremely high ram usage

[DanielPBliss]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

When using gatsby build or gatsby develop with the plugin gatsby-transformer-csv with a largeish csv file ~16'000 rows it hangs on the 'source and transform nodes' stage seemingly indefinitely. Also the nodejs process is using approximately 50Gb out of my 64gb of memory.
Ive tried this with a new starter of gatsby 4 and created a new csv file with the same length to make sure it wasnt an issue the one project or the file was somehow corrupted/invalid.

I know a similar issue was mentioned here : #11839 (comment) but this one doesnt mention high memory usage so may have a different cause

Reproduction Link

https://github.com/DanielPBliss/GatsbySourceAndTransformIssue

Steps to Reproduce

1. Clone linked repo
2. Run 'gatsby build' or 'gatsby develop'

...

Expected Result

Build to complete and Ram usage to stay within normal range

Actual Result

Build hangs on 'source and transform nodes' stage seemingly indefinitely and ram usage quickly accumulates to about 50gb out of my 64gb.

Environment

System:
    OS: Windows 10 10.0.19042
    CPU: (8) x64 Intel(R) Core(TM) i7-7820HK CPU @ 2.90GHz 
  Binaries:
    Node: 14.15.5 - C:\Program Files\nodejs\node.EXE       
    Yarn: 1.22.5 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 6.14.11 - C:\Program Files\nodejs\npm.CMD
  Languages:
    Python: 3.8.3 - C:\Python38\python.EXE
  Browsers:
    Edge: Spartan (44.19041.1266.0), Chromium (95.0.1020.40)
  npmPackages:
    gatsby: ^4.1.0 => 4.1.0
    gatsby-plugin-gatsby-cloud: ^4.1.0 => 4.1.0
    gatsby-plugin-image: ^2.1.0 => 2.1.0
    gatsby-plugin-manifest: ^4.1.0 => 4.1.0
    gatsby-plugin-offline: ^5.1.0 => 5.1.0
    gatsby-plugin-react-helmet: ^5.1.0 => 5.1.0
    gatsby-plugin-sharp: ^4.1.0 => 4.1.0
    gatsby-source-filesystem: ^4.1.0 => 4.1.0
    gatsby-transformer-csv: ^4.1.0 => 4.1.0
    gatsby-transformer-sharp: ^4.1.0 => 4.1.0
  npmGlobalPackages:
    gatsby-cli: 3.14.2

Config Flags

No response

[nickamckenna]

"gatsby build" -> It uses 105Gb in about 30 seconds on my computer! Crazy!

[wardpeet]

My guess is that we should do csv streaming instead of loading the whole file. It's not something we will prioritize anytime soon but feel free to open a PR.

[pragmaticpat]

So @wardpeet this looks like some inefficiency within gastby-transformer-csv specifically - is that right?

[joernroeder]

I'm also experiencing very high usage of RAM on my local machine on my gatsby 4 test branch on the same build step.
Also gatsby-could (obviously) fails with Your Gatsby build's memory consumption exceeded the limits allowed in your plan. For more details, see https://gatsby.dev/memory.

I don't use the csv source plugin, but maybe the following list of yarn list | grep gatsby- can help to resolve that.

├─ custom-gatsby-plugin@1.0.0
│  ├─ gatsby-source-apiserver@^2.1.8
│  └─ gatsby-source-filesystem@^4.1.3
├─ custom-gatsby-utils@1.0.0
│  └─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-background-image@^1.3.1
│  ├─ gatsby-plugin-feed@^4.1.1
│  ├─ gatsby-plugin-force-trailing-slashes@^1.0.4
│  ├─ gatsby-plugin-gatsby-cloud@^4.1.3
│  ├─ gatsby-plugin-image@^2.1.3
│  ├─ gatsby-plugin-loadable-components-ssr@^4.1.0
│  ├─ gatsby-plugin-netlify-cms@^6.1.0
│  ├─ gatsby-plugin-react-helmet-async@^1.1.1
│  ├─ gatsby-plugin-react-svg@^3.1.0
│  ├─ gatsby-plugin-robots-txt@^1.6.14
│  ├─ gatsby-plugin-sharp@^4.1.4
│  ├─ gatsby-plugin-sitemap@^5.1.1
│  ├─ gatsby-plugin-theme-ui@^0.12.0
│  ├─ gatsby-plugin-twitter@^4.1.1
│  ├─ gatsby-plugin-webpack-bundle-analyser-v2@^1.1.25
│  ├─ gatsby-remark-autolink-headers@^5.1.1
│  ├─ gatsby-remark-embedder@^5.0.0
│  ├─ gatsby-remark-responsive-iframe@^5.1.0
│  ├─ gatsby-source-datocms@^3.0.10
│  ├─ gatsby-source-filesystem@^4.1.3
│  ├─ gatsby-source-wordpress@^6.1.3
│  ├─ gatsby-transformer-cloudinary@^2.2.3
│  ├─ gatsby-transformer-json@^4.1.0
│  ├─ gatsby-transformer-remark@^5.1.4
│  ├─ gatsby-transformer-sharp@^4.1.0
│  ├─ gatsby-transformer-yaml@^4.1.0
│  └─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-core-utils@^3.1.3
│  └─ gatsby-legacy-polyfills@^2.1.0
├─ gatsby-background-image@1.5.3
├─ gatsby-cli@4.1.4
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-recipes@^1.1.3
│  ├─ gatsby-telemetry@^3.1.3
├─ gatsby-core-utils@3.1.3
├─ gatsby-graphiql-explorer@2.1.0
├─ gatsby-legacy-polyfills@2.1.0
├─ gatsby-link@4.1.0
├─ gatsby-page-utils@2.1.3
│  ├─ gatsby-core-utils@^3.1.3
├─ gatsby-plugin-catch-links@4.1.0
├─ gatsby-plugin-feed@4.1.1
│  ├─ gatsby-plugin-utils@^2.1.1
├─ gatsby-plugin-force-trailing-slashes@1.0.5
├─ gatsby-plugin-gatsby-cloud@4.1.3
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-telemetry@^3.1.3
├─ gatsby-plugin-image@2.1.3
│  ├─ gatsby-core-utils@^3.1.3
├─ gatsby-plugin-loadable-components-ssr@4.1.0
├─ gatsby-plugin-netlify-cms@6.1.0
├─ gatsby-plugin-page-creator@4.1.4
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-page-utils@^2.1.3
│  ├─ gatsby-plugin-utils@^2.1.1
│  ├─ gatsby-telemetry@^3.1.3
├─ gatsby-plugin-react-helmet-async@1.2.0
├─ gatsby-plugin-react-svg@3.1.0
├─ gatsby-plugin-robots-txt@1.6.14
├─ gatsby-plugin-sharp@4.1.4
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-plugin-utils@^2.1.1
│  ├─ gatsby-telemetry@^3.1.3
├─ gatsby-plugin-sitemap@5.1.1
├─ gatsby-plugin-theme-ui@0.12.0
├─ gatsby-plugin-twitter@4.1.1
├─ gatsby-plugin-typescript@4.1.3
├─ gatsby-plugin-utils@2.1.1
├─ gatsby-plugin-webpack-bundle-analyser-v2@1.1.25
├─ gatsby-react-router-scroll@5.1.0
├─ gatsby-recipes@1.1.3
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-telemetry@^3.1.3
├─ gatsby-remark-autolink-headers@5.1.1
├─ gatsby-remark-embedder@5.0.0
├─ gatsby-remark-responsive-iframe@5.1.0
├─ gatsby-source-apiserver@2.1.8
├─ gatsby-source-datocms@3.0.10
│  ├─ gatsby-core-utils@^3.1.0
│  ├─ gatsby-core-utils@3.1.0
│  ├─ gatsby-plugin-utils@^2.0.0
│  ├─ gatsby-plugin-utils@2.0.0
├─ gatsby-source-filesystem@4.1.3
│  ├─ gatsby-core-utils@^3.1.3
├─ gatsby-source-wordpress@6.1.3
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-plugin-catch-links@^4.1.0
│  ├─ gatsby-source-filesystem@^4.1.3
├─ gatsby-telemetry@3.1.3
│  ├─ gatsby-core-utils@^3.1.3
├─ gatsby-transformer-cloudinary@2.2.4
├─ gatsby-transformer-json@4.1.0
├─ gatsby-transformer-remark@5.1.4
│  ├─ gatsby-core-utils@^3.1.3
├─ gatsby-transformer-sharp@4.1.0
├─ gatsby-transformer-yaml@4.1.0
├─ gatsby-worker@1.1.0
│  ├─ gatsby-cli@^4.1.4
│  ├─ gatsby-core-utils@^3.1.3
│  ├─ gatsby-graphiql-explorer@^2.1.0
│  ├─ gatsby-legacy-polyfills@^2.1.0
│  ├─ gatsby-link@^4.1.0
│  ├─ gatsby-plugin-page-creator@^4.1.4
│  ├─ gatsby-plugin-typescript@^4.1.3
│  ├─ gatsby-plugin-utils@^2.1.1
│  ├─ gatsby-react-router-scroll@^5.1.0
│  ├─ gatsby-telemetry@^3.1.3
│  ├─ gatsby-worker@^1.1.0

[joernroeder]

shot in the dark, but it could be the same issue #34081