[gatsby-plugin-sharp] Generating (all) images fails

[LekoArts]

Description

Since the middle of this month (I don't know which version change) the image generation fails if I try to generate all images. I've got images all over my site and Gatsby tells me that it will generate ~ 600 images (since I'm using gatsby-image and responsiveSizes).

Last year everything was fine. During the image generation there were messages (I don't recall them in detail):
MaxEventListener ...
And after that it started in a new line and went on. All images were generated.

Environment

Gatsby version: 1.9.172
"gatsby-plugin-sharp": "^1.6.27",
"gatsby-transformer-sharp": "^1.6.17",
"gatsby-remark-images": "^1.5.41",
Node.js version: 8.1.4
Operating System: Win 10 64bit

Actual result

Now it hangs during the image generation at no specific point.
Generating image thumbnails [==================-------] 402/603 44.3 secs
Sometimes it's at the half, sometimes before or after the middle.

If I remove content and let it build incrementally I can finish the image generation :/

Expected behavior

The image generation should throw an error (it just hangs and doesn't do anything) or work as intended.

Steps to reproduce

I can't provide a sample repo but a Gatsby project with a lot of images + responsiveSizes should be sufficient.

[KyleAMathews]

I believe I've seen this. I won't have time to look into this right away but it'd be great if someone could trace where things break.

@LekoArts in the meantime — could you try setting up your environment so you're caching the generated images in the public directory? That way even if generating all images fail, you can restart it with far fewer images to generate the second time. Then finally, you'll only be generating new thumbnails for images that have changed or when you change your queries. Obviously not the long-term fix but should get you unstuck for now.

[LekoArts]

@KyleAMathews I normally don't delete the public folder but in this case I tried some things hence I needed to generate all images again.

That way even if generating all images fail, you can restart it with far fewer images to generate the second time.

No, sadly this doesn't work. If I restart the process it just skips the remaining images.

Then finally, you'll only be generating new thumbnails for images that have changed or when you change your queries.

Yeah, that's what I'm doing right now :)

[jaggad]

@KyleAMathews @LekoArts I have also recently encountered this problem. It seems to get stuck at a random percetange completion. If I cancel out, and start again it won't pick up from where it left off and will leave a bunch of broken images.

I've had some seemingly random success if I delete the .cache folder and then rebuild but it's not always.

I also have to do add in posts incrementally to try and get some success. Even then a bunch still don't generate. This is making Gatsby near unusable for me, I've spent over 4 hours trying to get this to work.

Any help or updates is appreciated.

[LekoArts]

Yes, you‘re having the exact same issue as I have. Once I generated all images and only add them one by one in the process of writing a new blog post everything works fine. But when I want to pull and build the repo on another machine it won‘t work.

[jaggad]

I've resolved this issue. I modified the gatsby-plugin-sharp code to log the files as it processed it. Turns out the code hangs whenever it encounters an unfamiliar file type. In my case in the old blog posts, there was a single .tif image. Removing this image fixed the problem.

This is likely why it's seen with a large number of images, as existing blogs are more likely to have sneaky strange file types.

@KyleAMathews I propose that this should not fail silently as I wasted a great deal of time. Instead, the gatsby-plugin-sharp code should add a check when processing each image to ensure it is a valid file type and should throw a descriptive error pointing to the source file or add support for converting more filetypes?

[pieh]

@jackedgson this is interesting - transformer sharp actually does check extensions ( https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-transformer-sharp/src/on-node-create.js#L6 ) but .tif is in list of allowed types and it's backed by sharp package description - "High performance Node.js image processing, the fastest module to resize JPEG, PNG, WebP and TIFF images. " ( https://github.com/lovell/sharp ). Perhaps Your particular .tif file is something sharp package chokes on. Could You share that .tif file that sharp can't process?

[pieh]

But we probably could/should add some error checking in gatsby-plugin-sharp to not wait indefinitely for file that failed to be processed

[jaggad]

Here's the file. Agreed that at the very least there should be something that handles an image taking too long or perhaps you could narrow down with that file to what specifically is wrong and we could just add a check for that. Wrong headers maybe?

Stitches.tif.zip

[pieh]

Actually - this might not be that this particular .tiff file can't be handled by sharp. Now when I checked gatsby-plugin-sharp code it seems we don't actually process .tiff files at all ( https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-plugin-sharp/src/index.js#L98 ). I will work on that later today.

[LekoArts]

I don‘t have any .tif file in my folders. Just jpg and png. It always stopped if I processed a lot of images.

But last year everything went fine until an update removed the „maxEventListener“ thingy.