Performance Changes 🚀 #294
Conversation
This allows the caching of babel-loader across rebuilds of the same site. The user can override this by passing the cacheDirectory param to their babel config. A falsey value will disable the cacheDirectory. This change also copies the normalized plugins and presets over the babelConfig. This allows Gatsby to support additional babel options. --- Building `gatsby-starer-blog` on a Macbook Before cache implemented: ``` Command being timed: "gatsby build" User time (seconds): 31.49 System time (seconds): 2.56 Percent of CPU this job got: 107% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:31.72 ``` After cache implemented on rebuild: ``` Command being timed: "gatsby build" User time (seconds): 26.80 System time (seconds): 2.41 Percent of CPU this job got: 107% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:27.11 ```
This gets called 6 times over the course of the build process and hits the filesystem to glob every time. This change makes it such that the result of the first glob is cached in memory and returned on subsequent calls. Sites with more pages will see a greater performance gain with this change. --- `gatsby-starter-blog` Before: ``` Command being timed: "gatsby build" User time (seconds): 26.80 System time (seconds): 2.41 Percent of CPU this job got: 107% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:27.11 ``` After: ``` Command being timed: "gatsby build" User time (seconds): 25.29 System time (seconds): 2.47 Percent of CPU this job got: 106% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:26.17 ``` --- `kyleamathews/blog` Before: ``` Command being timed: "gatsby build" User time (seconds): 38.03 System time (seconds): 3.02 Percent of CPU this job got: 106% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:38.47 ``` After: ``` Command being timed: "gatsby build" User time (seconds): 32.10 System time (seconds): 2.57 Percent of CPU this job got: 107% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:32.31 ```
This is a performance optimization recommended in gatsbyjs#280 --- `gatsby-starter-blog` Before: ``` Command being timed: "gatsby build" User time (seconds): 25.29 System time (seconds): 2.47 Percent of CPU this job got: 106% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:26.17 ``` After: ``` Command being timed: "gatsby build" User time (seconds): 24.29 System time (seconds): 2.83 Percent of CPU this job got: 110% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:24.56 ``` --- `kyleamathews/blog` Before: ``` Command being timed: "gatsby build" User time (seconds): 32.10 System time (seconds): 2.57 Percent of CPU this job got: 107% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:32.31 ``` After: ``` Command being timed: "gatsby build" User time (seconds): 29.84 System time (seconds): 2.81 Percent of CPU this job got: 110% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:29.59 ```
This is a performance optimization recommended in gatsbyjs#280 There are a lot less loaders than general modules, so the performance gain is minimal, but every second counts. --- `gatsby-starter-blog` Before: ``` Command being timed: "gatsby build" User time (seconds): 24.29 System time (seconds): 2.83 Percent of CPU this job got: 110% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:24.56 ``` After: ``` Command being timed: "gatsby build" User time (seconds): 23.78 System time (seconds): 2.80 Percent of CPU this job got: 110% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:24.05 ``` --- `kyleamathews/blog` Before: ``` Command being timed: "gatsby build" User time (seconds): 29.84 System time (seconds): 2.81 Percent of CPU this job got: 110% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:29.59 After: ``` Command being timed: "gatsby build" User time (seconds): 26.70 System time (seconds): 2.79 Percent of CPU this job got: 112% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:26.30 ```
|
Ummm wow! 👏 👏 👏 I hadn't realized there were such big perf gains sitting around esp for such easy fixes. Awesome! Code looks great but won't have time to merge / create new release until tomorrow probably. BTW, curious if you've thought about parallelizing the various builds? I think there could be some big perf gains there as the builds are almost 100% CPU-bound. |
|
I tried it and actually it got slower for me. I have a Macbook at home so I think the CoreM processor just doesn't cope well with multi-threaded stuff. I'm going to run a few benchmarks at work to see if the forked processes makes it faster given a real CPU. |
|
hmmm... odd, CoreM has two cores right? http://www.laptopmag.com/articles/intel-core-m-details I have an I7 in my Mac Book Pro — is your code public? I'd love to test it out as well. |
|
I just pushed to origin/bs-parrallelbuilds so you should be able to fetch and checkout A lot of i/o from ProgressPlugin, but you can delete them. |
|
I'm not sure about what I'm about to say, but couldn't this cause some issues regarding file writing? Could you both please set me in and tell me what are the things that doesn't allow us to achieve the public resources on an unique build? |
I don't see this being a problem. You typically file lock to prevent writing the same file that is being read by multiple processes, but the webpack output for each stage is explicitly set to different files.
I'm sorry, I don't understand what you are asking. |
|
Ran some quick tests on my blog: Current master: real 0m33.742s
user 0m28.331s
sys 0m2.840sThis PR: real 0m23.164s
user 0m22.353s
sys 0m1.806sForking fork: real 0m25.560s
user 0m31.893s
sys 0m3.229sSo oddly when forking the user time goes up a lot while the real time goes up a little. Don't have a lot of experience with node.js + child processes but this strikes me as odd. I ran these all on my ~2015 Mac Book Pro EDITED |
|
The parrallelbuilds is really rough and doesn't take advantage of the memoized glob-pages here which gives a pretty big jump in performance. Especially right now when there is a lot of blocking i/o with fs.readFileSync inside of glob pages, running it more than once is very expensive. I'd imagine modifying glob pages to request pages from main process could benefit from the memoized result. Something like a One thing I noticed though is that when we separated the build-css from javascript, webpack still needs to go and compile every module that was in your bundle.js. You can see this with the ProgressPlugin counting up the modules in both child processes. I'm not sure if separating them was the best solution, but parallelizing build-css and build-javascript will probably be faster than not doing so. I'm not too sure if parallelizing will be faster than just doing both at the same time in one compilation though. |
|
Sounds great 👍 I'll do a quick release tomorrow morning of this existing PR and then if child stuff proves out, release that too. Thanks! |
|
My best time with process.send to use the cached glob pages still doesn't touch this PR by a pretty significant margin on a randomly generated 500 markdown post site. |
|
@benstepp am I reading that wrong? Both the forked versions win on clock time (which is what we care about) by 2 & 2.5 secs correct? |
|
It looks like you are correct. Elapsed time is the important one in this output. There are definite advantages to multi processing the build stages, especially with large sites shaving off 20%+ of the total build time. I'll try and clean up the code by the weekend because right now I turned it into a complete mess. |
|
👍 |
|
Released 0.11.1 with this. |
The latest version of this library doesn't currently work on Windows and we suspect it has to do with the loaders being handed a path. Using path.join ensures that proper Windows pathing will be returned if on a Windows system. Related gatsbyjs#293
3 participants
If you want to see incremental progress I put the
timeoutput on each commit. Here is the overall result: