Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
58da78f
Implement distributed groupby sum and apply_concat_apply function for…
kayibal Mar 7, 2018
51ae0a2
Merge branch 'master' into distributed-groupby-sum
kayibal Mar 7, 2018
3f3be49
update comments
kayibal Mar 9, 2018
63d4dc8
add test for different index datatypes
kayibal Mar 9, 2018
df110c9
implement sort_index
kayibal Mar 31, 2018
49cb182
implement __len__
kayibal Apr 1, 2018
fd10c00
implement rename, optimize groupby_sum and join
kayibal Apr 1, 2018
b89f7d4
implement distributed set_index
kayibal Apr 1, 2018
bb818ff
Merge branch 'optimization/distributed' into develop
kayibal Apr 1, 2018
f942a91
Merge branch 'master' into refactor/binning
kayibal Oct 8, 2018
d2b68e3
Formatting
kayibal Oct 8, 2018
2d2ab88
fix dask version until we have an update
kayibal Oct 8, 2018
8fc32b8
number of line ouput in __repr__ changed.
kayibal Oct 8, 2018
47782ed
stop checking for number of lines as it adjust to terminal size now...
kayibal Oct 8, 2018
9c352f9
Create folders when writing to local filesystem
kayibal Oct 9, 2018
cd2ce7b
Merge branch 'master' into refactor/binning
kayibal Oct 17, 2018
fcdd167
Fix empty dtype
kayibal Oct 10, 2018
18469ff
Implement distributed drop.
kayibal Oct 10, 2018
4876190
Always add npz extension when writing SparseFrame to npz format
kayibal Oct 10, 2018
4c18873
Fix metadata handling on set_index method
kayibal Oct 18, 2018
80aef8e
Correct dask requirement
kayibal Oct 22, 2018
414c69a
Add method for dask SparseFrame and tuple divisions type
kayibal Oct 29, 2018
f0aa2d8
Support empty divisions
kayibal Oct 30, 2018
3329492
Pass on divisions on sort_index
kayibal Oct 31, 2018
3223850
More restrictive pandas version as .drop method fails with pandas==0.…
kayibal Oct 31, 2018
a603d3d
Fix bug where empty dataframe would create wrongly sized shuffle array
kayibal Nov 1, 2018
92a8f5d
Fix bug where join with in memory sparse frame would return rows from…
kayibal Nov 15, 2018
c6fa6d0
update dask version
kayibal Dec 4, 2018
d60b758
Update dask version in setup.py
kayibal Dec 4, 2018
031d965
Update deprecated set_options call
kayibal Dec 4, 2018
734ff26
some fixes to tests
kayibal Dec 4, 2018
fbf0b17
Fix moto and boto versions
kayibal Dec 4, 2018
b836024
Update test dependencies
kayibal Dec 4, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ jobs:
- image: drtools/dask:latest
steps:
- checkout
- run: pip install pytest pytest-cov moto dask==0.19.2 .
- run: pip install boto3==1.7.84 botocore==1.10.84 moto==1.3.6
- run: pip install pytest pytest-cov dask==1.0.0 .
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be consistent with setup.py? Or shouldn't it be done by pip install .[test]?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx it is resolved

- run: py.test --cov sparsity --cov-report xml sparsity
- run: bash <(curl -s https://codecov.io/bash)
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
'scipy>=0.18.1',
'numpy>=1.12.0',
's3fs>=0.1.0',
'dask>0.19.2'
'dask>0.20.0'
],
test_requires=[
'moto',
Expand Down
2 changes: 1 addition & 1 deletion sparsity/dask/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ def join(self, other, on=None, how='left', lsuffix='',
# make empty meta
meta = meta.loc[[False] * meta.shape[0], :]
join_func = partial(sp.SparseFrame.join, other=other,
how='left')
how=how)
return self.map_partitions(join_func, meta=meta, name='simplejoin')
if not isinstance(other, (SparseFrame)):
raise ValueError('other must be SparseFrame')
Expand Down
2 changes: 1 addition & 1 deletion sparsity/test/test_dask_sparse_frame.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

from .conftest import tmpdir

dask.context.set_options(get=dask.local.get_sync)
dask.config.set(scheduler=dask.local.get_sync)


@pytest.fixture
Expand Down