-
-
Notifications
You must be signed in to change notification settings - Fork 3.1k
feat: rewrite dag stat to be awesomerer #8795
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
bf9eb70 to
f1d673f
Compare
Fixes ipfs#8794 Fixes ipfs#8791 This adds: - skip-raw-leaves option, that avoids downloading raw leaves when you already know their size in the parrent block - multithreaded download - properly count blocks that show up multiple times with different multicodec - do not count size of inlined blocks Speed: Way way faster (it's hard to make an objective number because that depends a lot on how much multithreading helps you and how many raw-leaves block you skip), in my case: >100x faster.
9b4bf2a to
90dbf55
Compare
That bundles all raw leaves emissions in skip mode in one.
|
@aschmahmann will provide first round of feedback to determine if this is something that we want to land. |
|
General comment: it looks like this PR is covering multiple issues. We think it's best to have the PRs separate in case there are different levels of alignment on the issues themselves. |
This seems specific to dag-pb and won't work for anything else. I don't think we should be adding anything dag-pb specific in here. Almost all dag-pb data is UnixFS and so people could look at
This seems good, although this will become problematic with the introduction of selectors as there is currently no parallelizable selector traversal. I'd rather invest time/code complexity budget in parallelizable selector traversals if you're interested.
Seems reasonable.
Seems reasonable. |
UnixFS and
|
|
Closed in favor of #8843, I do not plan to work on rawleaf skipping or multithreaded downloads anytime soon. |
Fixes #8794
Fixes #8791
Todo:
This adds:
already know their size in the parrent block
multicodec
Speed:
Way way faster (it's hard to make an objective number because that depends a lot on how much multithreading helps you and how many raw-leaves block you skip), in my case:
>100xfaster.