Skip to content

kqueue: Only make a single syscall to add all the watches#454

Merged
0xpr03 merged 3 commits intonotify-rs:mainfrom
alexkirsz:alexkirsz/one-syscall
Jan 14, 2023
Merged

kqueue: Only make a single syscall to add all the watches#454
0xpr03 merged 3 commits intonotify-rs:mainfrom
alexkirsz:alexkirsz/one-syscall

Conversation

@alexkirsz
Copy link
Contributor

@alexkirsz alexkirsz commented Nov 16, 2022

Hey!

We use notify v4 in turbopack and we've recently tried updating to v5 to use the kqueue implementation, as we found it to be much faster than FSEvents at detecting updates.

However, we also noticed a very large drop in performance when watching large directories. After a bit of investigating, it turns out that notify makes a kevent syscall for each file it watches. Watching a folder with 15k files ends up taking about 11.5s.

This PR changes this behavior to ensure we only make a single kevent call when adding multiple files at the same time. This improves the above case to take 900ms.

I've also filed an issue with kqueue to fix another performance problem, which makes watching n files O(n²). This would further improve to 400ms.

Finally, while profiling the above issue, I noticed that during the 12s of watching, only 2 cores of my computer were working. I have a prototype version that uses multiple threads to open files, which might be faster when watching large directories, but it would also require changes to kqueue, so I'm filing it away for now.

@0xpr03
Copy link
Member

0xpr03 commented Nov 16, 2022

cc @xanderio who added this initially for BSD, my kqueue knowledge is kinda limited

@0xpr03
Copy link
Member

0xpr03 commented Nov 16, 2022

I'm intending to accept this. I'd like to have some confirmation from BSD & Apple folks that this actually works as intended for single files & recursive folders, since we're now in stable v5.

@0xpr03 0xpr03 merged commit d765321 into notify-rs:main Jan 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants