Conversation
4dc7fa0 to
60ef889
Compare
|
@joeloskarsson |
|
Thanks for working on this @Jayant-kernel! Could you write a test that measures that 1) the result of the operation is unchanged and 2) how long the operation takes? And once you've done that you can execute the test both against what is on uvx --with "git+https://<main-branch> your-test-script.py
uvx --with "git+https://<your-branch> your-test-script.pyAnd then print the result here together with the test script? |
76f446f to
8381060
Compare
|
@leifdenby |
Describe your changes
This PR significantly reduces the memory footprint and execution time for creating padded cropped domain masks on massive datasets, addressing the >2 day runtime identified in #83.
Changes:
Memory Scalability: In
distance_to_convex_hull_boundary(), we removed thenp.stack([...]).min(axis=0)array aggregation logic which was previously crashing caches and consumingO(M * N)memory. It has been replaced with an explicitnp.minimumstream reduction over the raw NumPy values footprint inside the loop, slashing memory explicitly down to exactlyO(N).Containment Dispatch Optimization: In
create_convex_hull_mask(), we removed the inherently slowxr.apply_ufunc(..., vectorize=True)Python mapping overhead overcontains_lonlat. We now pre-calculate the full(N, 3)unit-sphere Cartesian arrays leveraging fast trigonometric numpy dispatch without entering a Python loop, and then evaluateSphericalPolygonimplicitly on the bare coordinates.Issue Link
resolves #83
Type of change
Checklist before requesting a review
pullwith--rebaseoption if possible).Checklist for reviewers
Each PR comes with its own improvements and flaws. The reviewer should check the following:
Author checklist after completed review
reflecting type of change (add section where missing):
Checklist for assignee