Skip to content

Conversation

@KyleAMathews
Copy link
Contributor

Don't bundle any route code by default — only when requested.

This can cut down the initial bundling time by 50%+

@KyleAMathews KyleAMathews requested a review from a team November 9, 2020 18:50
@gatsbot gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Nov 9, 2020
@KyleAMathews KyleAMathews requested review from a team November 9, 2020 19:22
@KyleAMathews
Copy link
Contributor Author

I'm going to close this and re-open it when #27432 & #27884 are in

@LekoArts LekoArts added topic: core Relates to Gatsby's core (e.g. page loading, reporter, state machine) and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Nov 10, 2020
KyleAMathews added a commit that referenced this pull request Nov 17, 2020
* Prototype SSR wip

* Make it work

* Fix issue with running two instances of webpack

* Add a test suite for SSR

* Linting

* Run tests in CI

* Lint

* Show activity for HTML rendering + renable socket.io so server doesn't crash

* Add error page when templates don't render correctly

* Rebuild dev ssr bundle when source files change

* Fix some lint errors

* Fix building html

* use gatsby colors for syntax highlighting

* Add test script to compare dev & prod output

* Add return types

* Don't respond to dev server page loads until any sourcing/transforming is done

* typings

* fix types... maybe

* maybe typescript happy

* Add missing globals to tests & update snapshots

* moer merry type work

* Remove outdated typography.js test

* Start migrating route handler to spawned service

* back out moving dev html route into state machine

* cleanups

* more cleanups

* yet moer cleanups

* Add test for error parsing & codeframe creation

* add return type

* Feature flag functionality behind env variable

* Lighten how much the dev ssr html webpack instance is watching

* cleanup

* track usage while we're in experimental stage

* update snapshots

* cleanup

* Restore support for dev 404 page

* fix test

* Catch reading errors

* Make static queries work in dev-ssr

* Keep the renderer around

* Send 'loading' page if webpack is busy

* Show more error so can debug CI

* lazily build dev 404 page so it happens after webpack/queries are run

* just use /

* Add ansi-html as depdnency

* Run test w/ experimental flag

* meaningless change to run tests again

* use older version compatable with CI

* Update packages/gatsby/cache-dir/develop-static-entry.js

Co-authored-by: Peter van der Zee <[email protected]>

* remove unnecessary changes

* Consistently use imports

* fix

* Conditionally generate body str

* remove unneeded change

* Add find-page-by-path util from @pieh

* make typescript happy

* Switch to use @pieh's page finder util

* fix find-page-by-path tests

* Enable dev ssr for tests

* Add build:types again

* Do not await a flush

* Only delete the render-page.js module cache when it changes

* Try to reduce memory retention

* Fix recreating dev 404 page on every request + cache requires

* Add return

* this wasn't necessary

* Remove unused var

* fix return type

* Share cache across develop/develop-html instances of webpack

* This caused a lot of runtime tests to fail

* Use the webpack hash

* This didn't work

* fix lint error

* Meaningless change to try tests again

* SSR pages in jest-worker so memory doesn't accumulate in main process

* fix lint

* make typescript happy too

* fix test import

* Automatically fork the dev ssr renderer so it's ready to go when the user requests a page

* Add structured logging on dev ssr failure

* Need require.resolve I think

* Add filepath + line/column to terminal error

* try try again

* Fixes hopefully

* typescript 😱

* lint

* Try tweaking jest settings

* Debuggin

* use default reporter

* explicitly init dev html worker pool so it doesn't start during tests

* restore original ci settings

* sup

* Update packages/gatsby-cli/src/structured-errors/error-map.ts

Co-authored-by: Lennart <[email protected]>

* console.logs seem to break jest-worker on CI

* Increase pageLoadTimeout

* try taskTimeout

* This might be confusing cypress

* Don't re-spawn the worker process on every change as that's very expensive. Just delete the module cache for 25 edits before re-spawning

* cleanups

* Update packages/gatsby/cache-dir/develop-static-entry.js

Co-authored-by: Ward Peeters <[email protected]>

* Cleanups suggested by @wardpeet

* fix lint

* Lazily compile page components

This makes the initial creation of the dev ssr bundle ~85% faster.

* fix typescript

* mock /lazy-sync-requires

* The lazy bundling created a race condition where two pages could be simultaneously requested but both would think they're done as soon as the first to arrive finishes — 'suspend' rendering until the pageComponent is found to avoid this

* Add more pages to make sure we're going to hit the race condition

* Check file directly that the page component has been added

This is a lot simpler & more reliable

* for some reason this lets log warnings from React not break jest-worker

* fix test & comment

* We can't use the gatsby reporter inside a child as it uses process.send for console.* which breaks jest-worker

* Move writing lazyComponents to requires-writer & still use old develop-static-entry if no flag

* update tests

* Don't render body of pages w/ matchPath

* use core util joinPath so works on windows

* try again

* try try again

* remove mistakenly added file

* fix pnp test

* Move lazy bundling changes to #27932

* More cleanup

* more cleanups

* Test both old & new develop-static-entry

* fix lint

* more lint fixes

* Use execa

* fix tests

* Update names

* Fail more gracefully when we source-maps don't work

* fix test & remove testing code

* Update snapshot

* fix dependency check

* remove unused import

* Update packages/gatsby/src/utils/dev-ssr/develop-html-route.ts

Co-authored-by: Ward Peeters <[email protected]>

* revert unnecessary change

* restore more old behavior + move all new requires behind the flag

* fix

* fix problem w/ merge w/ master

Co-authored-by: Sidhartha Chatterjee <[email protected]>
Co-authored-by: gatsbybot <[email protected]>
Co-authored-by: Peter van der Zee <[email protected]>
Co-authored-by: Michal Piechowiak <[email protected]>
Co-authored-by: Lennart <[email protected]>
Co-authored-by: Ward Peeters <[email protected]>
gatsbybot added a commit to gatsbyjs/gatsby-starter-mdx-basic that referenced this pull request Nov 17, 2020
* Prototype SSR wip

* Make it work

* Fix issue with running two instances of webpack

* Add a test suite for SSR

* Linting

* Run tests in CI

* Lint

* Show activity for HTML rendering + renable socket.io so server doesn't crash

* Add error page when templates don't render correctly

* Rebuild dev ssr bundle when source files change

* Fix some lint errors

* Fix building html

* use gatsby colors for syntax highlighting

* Add test script to compare dev & prod output

* Add return types

* Don't respond to dev server page loads until any sourcing/transforming is done

* typings

* fix types... maybe

* maybe typescript happy

* Add missing globals to tests & update snapshots

* moer merry type work

* Remove outdated typography.js test

* Start migrating route handler to spawned service

* back out moving dev html route into state machine

* cleanups

* more cleanups

* yet moer cleanups

* Add test for error parsing & codeframe creation

* add return type

* Feature flag functionality behind env variable

* Lighten how much the dev ssr html webpack instance is watching

* cleanup

* track usage while we're in experimental stage

* update snapshots

* cleanup

* Restore support for dev 404 page

* fix test

* Catch reading errors

* Make static queries work in dev-ssr

* Keep the renderer around

* Send 'loading' page if webpack is busy

* Show more error so can debug CI

* lazily build dev 404 page so it happens after webpack/queries are run

* just use /

* Add ansi-html as depdnency

* Run test w/ experimental flag

* meaningless change to run tests again

* use older version compatable with CI

* Update packages/gatsby/cache-dir/develop-static-entry.js

Co-authored-by: Peter van der Zee <[email protected]>

* remove unnecessary changes

* Consistently use imports

* fix

* Conditionally generate body str

* remove unneeded change

* Add find-page-by-path util from @pieh

* make typescript happy

* Switch to use @pieh's page finder util

* fix find-page-by-path tests

* Enable dev ssr for tests

* Add build:types again

* Do not await a flush

* Only delete the render-page.js module cache when it changes

* Try to reduce memory retention

* Fix recreating dev 404 page on every request + cache requires

* Add return

* this wasn't necessary

* Remove unused var

* fix return type

* Share cache across develop/develop-html instances of webpack

* This caused a lot of runtime tests to fail

* Use the webpack hash

* This didn't work

* fix lint error

* Meaningless change to try tests again

* SSR pages in jest-worker so memory doesn't accumulate in main process

* fix lint

* make typescript happy too

* fix test import

* Automatically fork the dev ssr renderer so it's ready to go when the user requests a page

* Add structured logging on dev ssr failure

* Need require.resolve I think

* Add filepath + line/column to terminal error

* try try again

* Fixes hopefully

* typescript 😱

* lint

* Try tweaking jest settings

* Debuggin

* use default reporter

* explicitly init dev html worker pool so it doesn't start during tests

* restore original ci settings

* sup

* Update packages/gatsby-cli/src/structured-errors/error-map.ts

Co-authored-by: Lennart <[email protected]>

* console.logs seem to break jest-worker on CI

* Increase pageLoadTimeout

* try taskTimeout

* This might be confusing cypress

* Don't re-spawn the worker process on every change as that's very expensive. Just delete the module cache for 25 edits before re-spawning

* cleanups

* Update packages/gatsby/cache-dir/develop-static-entry.js

Co-authored-by: Ward Peeters <[email protected]>

* Cleanups suggested by @wardpeet

* fix lint

* Lazily compile page components

This makes the initial creation of the dev ssr bundle ~85% faster.

* fix typescript

* mock /lazy-sync-requires

* The lazy bundling created a race condition where two pages could be simultaneously requested but both would think they're done as soon as the first to arrive finishes — 'suspend' rendering until the pageComponent is found to avoid this

* Add more pages to make sure we're going to hit the race condition

* Check file directly that the page component has been added

This is a lot simpler & more reliable

* for some reason this lets log warnings from React not break jest-worker

* fix test & comment

* We can't use the gatsby reporter inside a child as it uses process.send for console.* which breaks jest-worker

* Move writing lazyComponents to requires-writer & still use old develop-static-entry if no flag

* update tests

* Don't render body of pages w/ matchPath

* use core util joinPath so works on windows

* try again

* try try again

* remove mistakenly added file

* fix pnp test

* Move lazy bundling changes to gatsbyjs/gatsby#27932

* More cleanup

* more cleanups

* Test both old & new develop-static-entry

* fix lint

* more lint fixes

* Use execa

* fix tests

* Update names

* Fail more gracefully when we source-maps don't work

* fix test & remove testing code

* Update snapshot

* fix dependency check

* remove unused import

* Update packages/gatsby/src/utils/dev-ssr/develop-html-route.ts

Co-authored-by: Ward Peeters <[email protected]>

* revert unnecessary change

* restore more old behavior + move all new requires behind the flag

* fix

* fix problem w/ merge w/ master

Co-authored-by: Sidhartha Chatterjee <[email protected]>
Co-authored-by: gatsbybot <[email protected]>
Co-authored-by: Peter van der Zee <[email protected]>
Co-authored-by: Michal Piechowiak <[email protected]>
Co-authored-by: Lennart <[email protected]>
Co-authored-by: Ward Peeters <[email protected]>
gatsbybot added a commit to gatsbyjs/gatsby-starter-minimal that referenced this pull request Nov 17, 2020
* Prototype SSR wip

* Make it work

* Fix issue with running two instances of webpack

* Add a test suite for SSR

* Linting

* Run tests in CI

* Lint

* Show activity for HTML rendering + renable socket.io so server doesn't crash

* Add error page when templates don't render correctly

* Rebuild dev ssr bundle when source files change

* Fix some lint errors

* Fix building html

* use gatsby colors for syntax highlighting

* Add test script to compare dev & prod output

* Add return types

* Don't respond to dev server page loads until any sourcing/transforming is done

* typings

* fix types... maybe

* maybe typescript happy

* Add missing globals to tests & update snapshots

* moer merry type work

* Remove outdated typography.js test

* Start migrating route handler to spawned service

* back out moving dev html route into state machine

* cleanups

* more cleanups

* yet moer cleanups

* Add test for error parsing & codeframe creation

* add return type

* Feature flag functionality behind env variable

* Lighten how much the dev ssr html webpack instance is watching

* cleanup

* track usage while we're in experimental stage

* update snapshots

* cleanup

* Restore support for dev 404 page

* fix test

* Catch reading errors

* Make static queries work in dev-ssr

* Keep the renderer around

* Send 'loading' page if webpack is busy

* Show more error so can debug CI

* lazily build dev 404 page so it happens after webpack/queries are run

* just use /

* Add ansi-html as depdnency

* Run test w/ experimental flag

* meaningless change to run tests again

* use older version compatable with CI

* Update packages/gatsby/cache-dir/develop-static-entry.js

Co-authored-by: Peter van der Zee <[email protected]>

* remove unnecessary changes

* Consistently use imports

* fix

* Conditionally generate body str

* remove unneeded change

* Add find-page-by-path util from @pieh

* make typescript happy

* Switch to use @pieh's page finder util

* fix find-page-by-path tests

* Enable dev ssr for tests

* Add build:types again

* Do not await a flush

* Only delete the render-page.js module cache when it changes

* Try to reduce memory retention

* Fix recreating dev 404 page on every request + cache requires

* Add return

* this wasn't necessary

* Remove unused var

* fix return type

* Share cache across develop/develop-html instances of webpack

* This caused a lot of runtime tests to fail

* Use the webpack hash

* This didn't work

* fix lint error

* Meaningless change to try tests again

* SSR pages in jest-worker so memory doesn't accumulate in main process

* fix lint

* make typescript happy too

* fix test import

* Automatically fork the dev ssr renderer so it's ready to go when the user requests a page

* Add structured logging on dev ssr failure

* Need require.resolve I think

* Add filepath + line/column to terminal error

* try try again

* Fixes hopefully

* typescript 😱

* lint

* Try tweaking jest settings

* Debuggin

* use default reporter

* explicitly init dev html worker pool so it doesn't start during tests

* restore original ci settings

* sup

* Update packages/gatsby-cli/src/structured-errors/error-map.ts

Co-authored-by: Lennart <[email protected]>

* console.logs seem to break jest-worker on CI

* Increase pageLoadTimeout

* try taskTimeout

* This might be confusing cypress

* Don't re-spawn the worker process on every change as that's very expensive. Just delete the module cache for 25 edits before re-spawning

* cleanups

* Update packages/gatsby/cache-dir/develop-static-entry.js

Co-authored-by: Ward Peeters <[email protected]>

* Cleanups suggested by @wardpeet

* fix lint

* Lazily compile page components

This makes the initial creation of the dev ssr bundle ~85% faster.

* fix typescript

* mock /lazy-sync-requires

* The lazy bundling created a race condition where two pages could be simultaneously requested but both would think they're done as soon as the first to arrive finishes — 'suspend' rendering until the pageComponent is found to avoid this

* Add more pages to make sure we're going to hit the race condition

* Check file directly that the page component has been added

This is a lot simpler & more reliable

* for some reason this lets log warnings from React not break jest-worker

* fix test & comment

* We can't use the gatsby reporter inside a child as it uses process.send for console.* which breaks jest-worker

* Move writing lazyComponents to requires-writer & still use old develop-static-entry if no flag

* update tests

* Don't render body of pages w/ matchPath

* use core util joinPath so works on windows

* try again

* try try again

* remove mistakenly added file

* fix pnp test

* Move lazy bundling changes to gatsbyjs/gatsby#27932

* More cleanup

* more cleanups

* Test both old & new develop-static-entry

* fix lint

* more lint fixes

* Use execa

* fix tests

* Update names

* Fail more gracefully when we source-maps don't work

* fix test & remove testing code

* Update snapshot

* fix dependency check

* remove unused import

* Update packages/gatsby/src/utils/dev-ssr/develop-html-route.ts

Co-authored-by: Ward Peeters <[email protected]>

* revert unnecessary change

* restore more old behavior + move all new requires behind the flag

* fix

* fix problem w/ merge w/ master

Co-authored-by: Sidhartha Chatterjee <[email protected]>
Co-authored-by: gatsbybot <[email protected]>
Co-authored-by: Peter van der Zee <[email protected]>
Co-authored-by: Michal Piechowiak <[email protected]>
Co-authored-by: Lennart <[email protected]>
Co-authored-by: Ward Peeters <[email protected]>
gatsbybot added a commit to gatsbyjs/gatsby-starter-notes-theme that referenced this pull request Nov 17, 2020
* Prototype SSR wip

* Make it work

* Fix issue with running two instances of webpack

* Add a test suite for SSR

* Linting

* Run tests in CI

* Lint

* Show activity for HTML rendering + renable socket.io so server doesn't crash

* Add error page when templates don't render correctly

* Rebuild dev ssr bundle when source files change

* Fix some lint errors

* Fix building html

* use gatsby colors for syntax highlighting

* Add test script to compare dev & prod output

* Add return types

* Don't respond to dev server page loads until any sourcing/transforming is done

* typings

* fix types... maybe

* maybe typescript happy

* Add missing globals to tests & update snapshots

* moer merry type work

* Remove outdated typography.js test

* Start migrating route handler to spawned service

* back out moving dev html route into state machine

* cleanups

* more cleanups

* yet moer cleanups

* Add test for error parsing & codeframe creation

* add return type

* Feature flag functionality behind env variable

* Lighten how much the dev ssr html webpack instance is watching

* cleanup

* track usage while we're in experimental stage

* update snapshots

* cleanup

* Restore support for dev 404 page

* fix test

* Catch reading errors

* Make static queries work in dev-ssr

* Keep the renderer around

* Send 'loading' page if webpack is busy

* Show more error so can debug CI

* lazily build dev 404 page so it happens after webpack/queries are run

* just use /

* Add ansi-html as depdnency

* Run test w/ experimental flag

* meaningless change to run tests again

* use older version compatable with CI

* Update packages/gatsby/cache-dir/develop-static-entry.js

Co-authored-by: Peter van der Zee <[email protected]>

* remove unnecessary changes

* Consistently use imports

* fix

* Conditionally generate body str

* remove unneeded change

* Add find-page-by-path util from @pieh

* make typescript happy

* Switch to use @pieh's page finder util

* fix find-page-by-path tests

* Enable dev ssr for tests

* Add build:types again

* Do not await a flush

* Only delete the render-page.js module cache when it changes

* Try to reduce memory retention

* Fix recreating dev 404 page on every request + cache requires

* Add return

* this wasn't necessary

* Remove unused var

* fix return type

* Share cache across develop/develop-html instances of webpack

* This caused a lot of runtime tests to fail

* Use the webpack hash

* This didn't work

* fix lint error

* Meaningless change to try tests again

* SSR pages in jest-worker so memory doesn't accumulate in main process

* fix lint

* make typescript happy too

* fix test import

* Automatically fork the dev ssr renderer so it's ready to go when the user requests a page

* Add structured logging on dev ssr failure

* Need require.resolve I think

* Add filepath + line/column to terminal error

* try try again

* Fixes hopefully

* typescript 😱

* lint

* Try tweaking jest settings

* Debuggin

* use default reporter

* explicitly init dev html worker pool so it doesn't start during tests

* restore original ci settings

* sup

* Update packages/gatsby-cli/src/structured-errors/error-map.ts

Co-authored-by: Lennart <[email protected]>

* console.logs seem to break jest-worker on CI

* Increase pageLoadTimeout

* try taskTimeout

* This might be confusing cypress

* Don't re-spawn the worker process on every change as that's very expensive. Just delete the module cache for 25 edits before re-spawning

* cleanups

* Update packages/gatsby/cache-dir/develop-static-entry.js

Co-authored-by: Ward Peeters <[email protected]>

* Cleanups suggested by @wardpeet

* fix lint

* Lazily compile page components

This makes the initial creation of the dev ssr bundle ~85% faster.

* fix typescript

* mock /lazy-sync-requires

* The lazy bundling created a race condition where two pages could be simultaneously requested but both would think they're done as soon as the first to arrive finishes — 'suspend' rendering until the pageComponent is found to avoid this

* Add more pages to make sure we're going to hit the race condition

* Check file directly that the page component has been added

This is a lot simpler & more reliable

* for some reason this lets log warnings from React not break jest-worker

* fix test & comment

* We can't use the gatsby reporter inside a child as it uses process.send for console.* which breaks jest-worker

* Move writing lazyComponents to requires-writer & still use old develop-static-entry if no flag

* update tests

* Don't render body of pages w/ matchPath

* use core util joinPath so works on windows

* try again

* try try again

* remove mistakenly added file

* fix pnp test

* Move lazy bundling changes to gatsbyjs/gatsby#27932

* More cleanup

* more cleanups

* Test both old & new develop-static-entry

* fix lint

* more lint fixes

* Use execa

* fix tests

* Update names

* Fail more gracefully when we source-maps don't work

* fix test & remove testing code

* Update snapshot

* fix dependency check

* remove unused import

* Update packages/gatsby/src/utils/dev-ssr/develop-html-route.ts

Co-authored-by: Ward Peeters <[email protected]>

* revert unnecessary change

* restore more old behavior + move all new requires behind the flag

* fix

* fix problem w/ merge w/ master

Co-authored-by: Sidhartha Chatterjee <[email protected]>
Co-authored-by: gatsbybot <[email protected]>
Co-authored-by: Peter van der Zee <[email protected]>
Co-authored-by: Michal Piechowiak <[email protected]>
Co-authored-by: Lennart <[email protected]>
Co-authored-by: Ward Peeters <[email protected]>
gatsbybot added a commit to gatsbyjs/gatsby-starter-hello-world that referenced this pull request Nov 17, 2020
* Prototype SSR wip

* Make it work

* Fix issue with running two instances of webpack

* Add a test suite for SSR

* Linting

* Run tests in CI

* Lint

* Show activity for HTML rendering + renable socket.io so server doesn't crash

* Add error page when templates don't render correctly

* Rebuild dev ssr bundle when source files change

* Fix some lint errors

* Fix building html

* use gatsby colors for syntax highlighting

* Add test script to compare dev & prod output

* Add return types

* Don't respond to dev server page loads until any sourcing/transforming is done

* typings

* fix types... maybe

* maybe typescript happy

* Add missing globals to tests & update snapshots

* moer merry type work

* Remove outdated typography.js test

* Start migrating route handler to spawned service

* back out moving dev html route into state machine

* cleanups

* more cleanups

* yet moer cleanups

* Add test for error parsing & codeframe creation

* add return type

* Feature flag functionality behind env variable

* Lighten how much the dev ssr html webpack instance is watching

* cleanup

* track usage while we're in experimental stage

* update snapshots

* cleanup

* Restore support for dev 404 page

* fix test

* Catch reading errors

* Make static queries work in dev-ssr

* Keep the renderer around

* Send 'loading' page if webpack is busy

* Show more error so can debug CI

* lazily build dev 404 page so it happens after webpack/queries are run

* just use /

* Add ansi-html as depdnency

* Run test w/ experimental flag

* meaningless change to run tests again

* use older version compatable with CI

* Update packages/gatsby/cache-dir/develop-static-entry.js

Co-authored-by: Peter van der Zee <[email protected]>

* remove unnecessary changes

* Consistently use imports

* fix

* Conditionally generate body str

* remove unneeded change

* Add find-page-by-path util from @pieh

* make typescript happy

* Switch to use @pieh's page finder util

* fix find-page-by-path tests

* Enable dev ssr for tests

* Add build:types again

* Do not await a flush

* Only delete the render-page.js module cache when it changes

* Try to reduce memory retention

* Fix recreating dev 404 page on every request + cache requires

* Add return

* this wasn't necessary

* Remove unused var

* fix return type

* Share cache across develop/develop-html instances of webpack

* This caused a lot of runtime tests to fail

* Use the webpack hash

* This didn't work

* fix lint error

* Meaningless change to try tests again

* SSR pages in jest-worker so memory doesn't accumulate in main process

* fix lint

* make typescript happy too

* fix test import

* Automatically fork the dev ssr renderer so it's ready to go when the user requests a page

* Add structured logging on dev ssr failure

* Need require.resolve I think

* Add filepath + line/column to terminal error

* try try again

* Fixes hopefully

* typescript 😱

* lint

* Try tweaking jest settings

* Debuggin

* use default reporter

* explicitly init dev html worker pool so it doesn't start during tests

* restore original ci settings

* sup

* Update packages/gatsby-cli/src/structured-errors/error-map.ts

Co-authored-by: Lennart <[email protected]>

* console.logs seem to break jest-worker on CI

* Increase pageLoadTimeout

* try taskTimeout

* This might be confusing cypress

* Don't re-spawn the worker process on every change as that's very expensive. Just delete the module cache for 25 edits before re-spawning

* cleanups

* Update packages/gatsby/cache-dir/develop-static-entry.js

Co-authored-by: Ward Peeters <[email protected]>

* Cleanups suggested by @wardpeet

* fix lint

* Lazily compile page components

This makes the initial creation of the dev ssr bundle ~85% faster.

* fix typescript

* mock /lazy-sync-requires

* The lazy bundling created a race condition where two pages could be simultaneously requested but both would think they're done as soon as the first to arrive finishes — 'suspend' rendering until the pageComponent is found to avoid this

* Add more pages to make sure we're going to hit the race condition

* Check file directly that the page component has been added

This is a lot simpler & more reliable

* for some reason this lets log warnings from React not break jest-worker

* fix test & comment

* We can't use the gatsby reporter inside a child as it uses process.send for console.* which breaks jest-worker

* Move writing lazyComponents to requires-writer & still use old develop-static-entry if no flag

* update tests

* Don't render body of pages w/ matchPath

* use core util joinPath so works on windows

* try again

* try try again

* remove mistakenly added file

* fix pnp test

* Move lazy bundling changes to gatsbyjs/gatsby#27932

* More cleanup

* more cleanups

* Test both old & new develop-static-entry

* fix lint

* more lint fixes

* Use execa

* fix tests

* Update names

* Fail more gracefully when we source-maps don't work

* fix test & remove testing code

* Update snapshot

* fix dependency check

* remove unused import

* Update packages/gatsby/src/utils/dev-ssr/develop-html-route.ts

Co-authored-by: Ward Peeters <[email protected]>

* revert unnecessary change

* restore more old behavior + move all new requires behind the flag

* fix

* fix problem w/ merge w/ master

Co-authored-by: Sidhartha Chatterjee <[email protected]>
Co-authored-by: gatsbybot <[email protected]>
Co-authored-by: Peter van der Zee <[email protected]>
Co-authored-by: Michal Piechowiak <[email protected]>
Co-authored-by: Lennart <[email protected]>
Co-authored-by: Ward Peeters <[email protected]>
@KyleAMathews
Copy link
Contributor Author

moved work to #28149

@LekoArts LekoArts deleted the lazy-dev-ssr-bundling branch April 23, 2021 11:07
fumierkm added a commit to fumierkm/gatsbyjs that referenced this pull request Nov 27, 2021
* Prototype SSR wip

* Make it work

* Fix issue with running two instances of webpack

* Add a test suite for SSR

* Linting

* Run tests in CI

* Lint

* Show activity for HTML rendering + renable socket.io so server doesn't crash

* Add error page when templates don't render correctly

* Rebuild dev ssr bundle when source files change

* Fix some lint errors

* Fix building html

* use gatsby colors for syntax highlighting

* Add test script to compare dev & prod output

* Add return types

* Don't respond to dev server page loads until any sourcing/transforming is done

* typings

* fix types... maybe

* maybe typescript happy

* Add missing globals to tests & update snapshots

* moer merry type work

* Remove outdated typography.js test

* Start migrating route handler to spawned service

* back out moving dev html route into state machine

* cleanups

* more cleanups

* yet moer cleanups

* Add test for error parsing & codeframe creation

* add return type

* Feature flag functionality behind env variable

* Lighten how much the dev ssr html webpack instance is watching

* cleanup

* track usage while we're in experimental stage

* update snapshots

* cleanup

* Restore support for dev 404 page

* fix test

* Catch reading errors

* Make static queries work in dev-ssr

* Keep the renderer around

* Send 'loading' page if webpack is busy

* Show more error so can debug CI

* lazily build dev 404 page so it happens after webpack/queries are run

* just use /

* Add ansi-html as depdnency

* Run test w/ experimental flag

* meaningless change to run tests again

* use older version compatable with CI

* Update packages/gatsby/cache-dir/develop-static-entry.js

Co-authored-by: Peter van der Zee <[email protected]>

* remove unnecessary changes

* Consistently use imports

* fix

* Conditionally generate body str

* remove unneeded change

* Add find-page-by-path util from @pieh

* make typescript happy

* Switch to use @pieh's page finder util

* fix find-page-by-path tests

* Enable dev ssr for tests

* Add build:types again

* Do not await a flush

* Only delete the render-page.js module cache when it changes

* Try to reduce memory retention

* Fix recreating dev 404 page on every request + cache requires

* Add return

* this wasn't necessary

* Remove unused var

* fix return type

* Share cache across develop/develop-html instances of webpack

* This caused a lot of runtime tests to fail

* Use the webpack hash

* This didn't work

* fix lint error

* Meaningless change to try tests again

* SSR pages in jest-worker so memory doesn't accumulate in main process

* fix lint

* make typescript happy too

* fix test import

* Automatically fork the dev ssr renderer so it's ready to go when the user requests a page

* Add structured logging on dev ssr failure

* Need require.resolve I think

* Add filepath + line/column to terminal error

* try try again

* Fixes hopefully

* typescript 😱

* lint

* Try tweaking jest settings

* Debuggin

* use default reporter

* explicitly init dev html worker pool so it doesn't start during tests

* restore original ci settings

* sup

* Update packages/gatsby-cli/src/structured-errors/error-map.ts

Co-authored-by: Lennart <[email protected]>

* console.logs seem to break jest-worker on CI

* Increase pageLoadTimeout

* try taskTimeout

* This might be confusing cypress

* Don't re-spawn the worker process on every change as that's very expensive. Just delete the module cache for 25 edits before re-spawning

* cleanups

* Update packages/gatsby/cache-dir/develop-static-entry.js

Co-authored-by: Ward Peeters <[email protected]>

* Cleanups suggested by @wardpeet

* fix lint

* Lazily compile page components

This makes the initial creation of the dev ssr bundle ~85% faster.

* fix typescript

* mock /lazy-sync-requires

* The lazy bundling created a race condition where two pages could be simultaneously requested but both would think they're done as soon as the first to arrive finishes — 'suspend' rendering until the pageComponent is found to avoid this

* Add more pages to make sure we're going to hit the race condition

* Check file directly that the page component has been added

This is a lot simpler & more reliable

* for some reason this lets log warnings from React not break jest-worker

* fix test & comment

* We can't use the gatsby reporter inside a child as it uses process.send for console.* which breaks jest-worker

* Move writing lazyComponents to requires-writer & still use old develop-static-entry if no flag

* update tests

* Don't render body of pages w/ matchPath

* use core util joinPath so works on windows

* try again

* try try again

* remove mistakenly added file

* fix pnp test

* Move lazy bundling changes to gatsbyjs/gatsby#27932

* More cleanup

* more cleanups

* Test both old & new develop-static-entry

* fix lint

* more lint fixes

* Use execa

* fix tests

* Update names

* Fail more gracefully when we source-maps don't work

* fix test & remove testing code

* Update snapshot

* fix dependency check

* remove unused import

* Update packages/gatsby/src/utils/dev-ssr/develop-html-route.ts

Co-authored-by: Ward Peeters <[email protected]>

* revert unnecessary change

* restore more old behavior + move all new requires behind the flag

* fix

* fix problem w/ merge w/ master

Co-authored-by: Sidhartha Chatterjee <[email protected]>
Co-authored-by: gatsbybot <[email protected]>
Co-authored-by: Peter van der Zee <[email protected]>
Co-authored-by: Michal Piechowiak <[email protected]>
Co-authored-by: Lennart <[email protected]>
Co-authored-by: Ward Peeters <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic: core Relates to Gatsby's core (e.g. page loading, reporter, state machine)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants