Skip to content
This repository was archived by the owner on Jul 25, 2022. It is now read-only.

Add custom global allocator #30

Merged
jimexist merged 2 commits into from
Feb 25, 2022
Merged

Add custom global allocator #30

jimexist merged 2 commits into from
Feb 25, 2022

Conversation

@matthewmturner
Copy link
Contributor

@matthewmturner matthewmturner commented Feb 23, 2022 •
edited
Loading

Closes #27

@matthewmturner
Copy link
Contributor Author

matthewmturner commented Feb 23, 2022

@Dandandan FYI

@Dandandan
Copy link
Contributor

Dandandan commented Feb 23, 2022

Did you also try mimalloc (see https://github.com/pola-rs/polars/blob/master/py-polars/Cargo.toml#L22). Should give similar results, might be faster or slower on some queries. Still interested which one is the best to pick.

houqp
houqp reviewed Feb 23, 2022
View reviewed changes
Cargo.toml Outdated
datafusion-expr = { version = "^7.0.0" }
datafusion-common = { version = "^7.0.0", features = ["pyarrow"] }
uuid = { version = "0.8", features = ["v4"] }
snmalloc-rs = {version = "0.2", features = ["cache-friendly"]}
Copy link
Member

@houqp houqp Feb 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the cache-friendly feature has been dropped from upstream in newer releases I believe.

@matthewmturner
Copy link
Contributor Author

matthewmturner commented Feb 23, 2022

mimalloc results below - faster on almost all queries except Q8 and Q10. This may be the better option.

q1: 0.0366870829999999
q2: 0.32823987499999996
q3: 1.1415230410000001
q4: 0.028246750000000098
q5: 1.1575570830000004
q6: 1.2152348750000006
q7: 1.1023409590000002
q8: 2.9869192910000004
q9: 0.5675542499999988
q10: 25.437209041

@matthewmturner matthewmturner changed the title Add sn-malloc as global allocator Add custom global allocator Feb 23, 2022
@matthewmturner
Copy link
Contributor Author

matthewmturner commented Feb 24, 2022

@Dandandan to confirm - I haven't updated PR with mimalloc yet. Did you approve for snmalloc or mimalloc?

@Dandandan
Copy link
Contributor

Dandandan commented Feb 24, 2022

Ah I didn't catch that 😅. I think mimalloc would be a good choice to go forward with. I think it will be interesting in the future to keep experimenting with this (e.g. there is an upcoming smalloc 2 version).

@matthewmturner
Copy link
Contributor Author

matthewmturner commented Feb 24, 2022

Ok sounds good! Will update.

@jimexist jimexist merged commit 368b50e into datafusion-contrib:main Feb 25, 2022
@jonmmease jonmmease mentioned this pull request May 19, 2022
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

3 more reviewers

@houqp houqp houqp left review comments

@Dandandan Dandandan Dandandan approved these changes

@jimexist jimexist jimexist approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Use custom allocator in Python build

4 participants

@matthewmturner @Dandandan @jimexist @houqp