kqueue: Only make a single syscall to add all the watches#454
Merged
0xpr03 merged 3 commits intonotify-rs:mainfrom Jan 14, 2023
Merged
kqueue: Only make a single syscall to add all the watches#4540xpr03 merged 3 commits intonotify-rs:mainfrom
0xpr03 merged 3 commits intonotify-rs:mainfrom
Conversation
Member
|
cc @xanderio who added this initially for BSD, my kqueue knowledge is kinda limited |
xanderio
approved these changes
Nov 16, 2022
0xpr03
reviewed
Nov 16, 2022
Member
|
I'm intending to accept this. I'd like to have some confirmation from BSD & Apple folks that this actually works as intended for single files & recursive folders, since we're now in stable v5. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hey!
We use notify v4 in
turbopackand we've recently tried updating to v5 to use thekqueueimplementation, as we found it to be much faster than FSEvents at detecting updates.However, we also noticed a very large drop in performance when watching large directories. After a bit of investigating, it turns out that
notifymakes akeventsyscall for each file it watches. Watching a folder with 15k files ends up taking about 11.5s.This PR changes this behavior to ensure we only make a single
keventcall when adding multiple files at the same time. This improves the above case to take 900ms.I've also filed an issue with
kqueueto fix another performance problem, which makes watching n files O(n²). This would further improve to 400ms.Finally, while profiling the above issue, I noticed that during the 12s of watching, only 2 cores of my computer were working. I have a prototype version that uses multiple threads to open files, which might be faster when watching large directories, but it would also require changes to
kqueue, so I'm filing it away for now.