createRemoteFileNode failing on contentLength check, but file might be not incomplete at all

[laurenskling]

Description

I am using the createRemoteFileNode as explained in https://www.gatsbyjs.com/docs/reference/graphql-data-layer/schema-customization/#feeding-remote-images-into-gatsby-image, it takes in my image field, which can be a png, jpg, [...] or svg, sourcing from Cloudinary.

Now with the latest gatsby releases my code starts crashing to Failed to download ... after 3 attempts (the INCOMPLETE_RETRY_LIMIT one) and I seem to have narrowed it down to svgs. And even more so: https://github.com/gatsbyjs/gatsby/pull/28547/files#diff-ed27ffe776b2ef7b45a29f65fc4b9b01922de1c85ec42e9a92722b3949110a82R188 failing to match content lengths makes the code crash the download. Now I debugged Cloudinary shipping my svgs gzipped. I'm no expert on content length, but my (not so) wild guess right now is that zipped svg's are indeed not the same content length as written bytes?

Steps to reproduce

adding a createRemoteFileNode to a remote url. my code (fetching remote images on Strapi uploads via gatsby-source-graphql):

createResolvers({
    STRAPI_UploadFile: {
      imageFile: {
        type: 'File',
        resolve: async (source) => {
          if (!source.url) {
            return null;
          }
          let res;
          try {
            res = await createRemoteFileNode({
              url: source.url,
              store,
              cache,
              createNode,
              createNodeId,
              reporter,
            });
          } catch (err) {
            console.error(`[createResolvers] error in fetching remote image ${source.url} (${source.name}):`, err);
            throw err;
          }
          return res;
        },
      },

Expected result

A file can be checked on incompleteness, including SVGs and zipped files

Actual result

An error is thrown because the file seems to have an incorrect content length. (but the file is downloaded correctly)

Environment

System:
OS: macOS 10.15.7
CPU: (8) x64 Intel(R) Core(TM) i7-1060NG7 CPU @ 1.20GHz
Shell: 5.7.1 - /bin/zsh
Binaries:
Node: 14.15.1 - /var/folders/gh/yg99pwqj2rngz5wysjcgl8ph0000gn/T/yarn--1610380123259-0.7945398836762099/node
Yarn: 1.22.4 - /var/folders/gh/yg99pwqj2rngz5wysjcgl8ph0000gn/T/yarn--1610380123259-0.7945398836762099/yarn
npm: 6.14.8 - /usr/local/bin/npm
Languages:
Python: 2.7.16 - /usr/bin/python
Browsers:
Chrome: 87.0.4280.88
Safari: 14.0.2

[wardpeet]

Hey, thanks for pointing it out. We have a fix in #28913 that will solve this. We'll publish it in the next patch release. For now you can try the @next tag.

[laurenskling]

Thanks so much <3

[rroslaniec]

@wardpeet
Problem still occurs for me. I was debugging that two evenings and finally I tracked down the issue. My story:

After upgrade to gatsby v3 and migrating all images to gatsby-plugin-image from my Strapi CMS I faced similar error like @laurenskling got. gatsby-source-strapi got some issues and some times it is needed to write down custom createResolvers function (again like @laurenskling shown here). First I was sure that I did not migrate my gatsby-node.js correctly - nope. Problem only occurs with images created by createRemoteFileNode therefore I directed my suspicions to "download" functions.

I started with https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-core-utils/src/fetch-remote-file.ts#L144 - easy. Checked response status code: 200, url: ok (curl'ed url, valid), gzip: nope. haveAllBytesBeenWritten was false. question why? IDK, so I dive into https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-core-utils/src/fetch-remote-file.ts#L99 function, simple console.log - nothing (that code is not executed). Maybe error handlers got answer? Nope - 0 errors - fsStream and got stream. I checked also dependencies and what I found? https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/package.json#L91 (got v8.x)

And guess what? I just override got version by type magic yarn add got (before install other deps) to override local version (v11.x). And problem disappeared. I think problem is somewhere in got version.

TL;DR Gatsby Team need to just bump got version cc @wardpeet.

[Ifletcher668]

I also have this issue. It appears to be related to the code above and gatsby-source-graphql Are there any updates on this problem?

P.S. This only happens in Netlify builds. If I run gatsby build locally, everything compiles just fine

[ahmadkhalaf1]

@wardpeet
hi ,
seems like the fix from the version 2.9.1 is not pushed forward to new versions ,
can you please confirm ?

[ahmadkhalaf1]

@wardpeet Hi again :)

can you please help with this issue? gzip files are still not working correctly , sometimes the file is downloaded fine and sometimes it show failed to download after 3 retry ... i am now using latest version 4 for gatsby-source-filesystem