fix(gatsby-source-filesystem): Retry stalled remote file downloads - #20843
Merged
Conversation
ascorbic
force-pushed
the
retry-stalled-downloads
branch
from
January 24, 2020 17:35
df34bb6 to
d673e9a
Compare
pieh
reviewed
Jan 24, 2020
pieh
reviewed
Jan 24, 2020
ascorbic
marked this pull request as ready for review
January 27, 2020 17:08
ascorbic
force-pushed
the
retry-stalled-downloads
branch
from
January 29, 2020 15:22
837654d to
5304fc9
Compare
sidharthachatterjee
approved these changes
Jan 31, 2020
sidharthachatterjee
left a comment
sidharthachatterjee
left a comment
Contributor
There was a problem hiding this comment.
Thank you! 🥇
sidharthachatterjee
suggested changes
Jan 31, 2020
sidharthachatterjee
left a comment
sidharthachatterjee
left a comment
Contributor
There was a problem hiding this comment.
Left a small comment!
ascorbic
force-pushed
the
retry-stalled-downloads
branch
from
January 31, 2020 12:02
56d11bd to
f02a80c
Compare
sidharthachatterjee
approved these changes
Jan 31, 2020
sidharthachatterjee
added
the
bot: merge on green
muescha
reviewed
Feb 15, 2020
| }) | ||
| const fsWriteStream = fs.createWriteStream(tmpFilename) | ||
| responseStream.pipe(fsWriteStream) | ||
| responseStream.on(`downloadProgress`, pro => console.log(pro)) |
Contributor
There was a problem hiding this comment.
the console.log is removed accidentally in resetTimeout?
Contributor
|
I think this is breaking my build (at least locally with Don't have time to grok the code, but what I think is happening: I'm using Ultimately, what I'm sure of is that increasing the timeout to 20 min fixes my issues. |
mwfrost
pushed a commit
to mwfrost/gatsby
that referenced
this pull request
Apr 20, 2023
…atsbyjs#20843) * fix(gatsby-source-filesystem): Retry stalled remote file downloads * Switch to got@8 * Await the recursive callback * Add tests for loading fix * Don't reset on data. Start timout after `response` * Add got timeout * Don't await the recursive call * Use constants for connection timeouts
raffishquartan
pushed a commit
to raffishquartan/gatsby
that referenced
this pull request
Apr 28, 2026
…atsbyjs#20843) * fix(gatsby-source-filesystem): Retry stalled remote file downloads * Switch to got@8 * Await the recursive callback * Add tests for loading fix * Don't reset on data. Start timout after `response` * Add got timeout * Don't await the recursive call * Use constants for connection timeouts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
gatsby-source-filesystem has a problem where downloads would stall and never timeout or error. Eventually all the concurrent downloads were stalled, preventing any more starting. This was causing builds to stall indefinitely.
This PR adds a timeout in
createRemoteFileNode. If nodataevents are received in the download stream for 30 seconds then the download is cancelled and retried up to three times. In tests this unblocks the downloads on the first attempt.WIP while I'm writing unit tests.
Documentation
Related Issues