Skip to content

s2: Add 'best' compression option #310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jan 13, 2021
Merged

s2: Add 'best' compression option #310

merged 4 commits into from
Jan 13, 2021

Conversation

@klauspost
Copy link
Owner

@klauspost klauspost commented Jan 12, 2021 •
edited
Loading

Add compression mode that will compress the data very well, but at a significant performance decrease.

Mainly for offline compression, but where decompression speed should still be high and compatible with other S2 compressed data.

Comparing on 16 core CPU:

* enwik10:
Default... 10000000000 -> 4761467548 [47.61%]; 1.098s, 8685.6MB/s
Better... 10000000000 -> 4225922984 [42.26%]; 2.817s, 3385.4MB/s
Best... 10000000000 -> 3667646858 [36.68%]; 35.995s, 264.9MB/s

* github-june-2days-2019.json:
Default... 6273951764 -> 1043196283 [16.63%]; 431ms, 13882.3MB/s
Better... 6273951764 -> 950079555 [15.14%]; 736ms, 8129.5MB/s
Best... 6273951764 -> 846260870 [13.49%]; 8.125s, 736.4MB/s

* nyc-taxi-data-10M.csv:
Default... 3325605752 -> 1095998837 [32.96%]; 324ms, 9788.7MB/s
Better... 3325605752 -> 960330423 [28.88%]; 602ms, 5268.4MB/s
Best... 3325605752 -> 794873295 [23.90%]; 6.619s, 479.1MB/s

* 10gb.tar
Default... 10065157632 -> 5916578242 [58.78%]; 1.028s, 9337.4MB/s
Better... 10065157632 -> 5650133605 [56.14%]; 2.172s, 4419.4MB/s
Best... 10065157632 -> 5246578570 [52.13%]; 25.696s, 373.6MB/s

* consensus.db.10gb:
Default... 10737418240 -> 4562648848 [42.49%]; 882ms, 11610.0MB/s
Better... 10737418240 -> 4542443833 [42.30%]; 3.3s, 3103.5MB/s
Best... 10737418240 -> 4272335558 [39.79%]; 38.955s, 262.9MB/s

@klauspost
s2: Add 'best' compression option
9ce8311 
Add compression mode that will compress the data very well, but at a significant performance decrease.

Mainly for offline compression, but where decompression speed should still be high.

Comparing to 'better':

```
github-june-2days-2019.json -> (discarded) 6273951764 -> 950079555 [15.14%]; 3541.4MB/s
github-june-2days-2019.json -> (discarded) 6273951764 -> 846260870 [13.49%]; 661.5MB/s

nyc-taxi-data-10M.csv -> (discarded) 3325605752 -> 960330423 [28.88%]; 3249.5MB/s
nyc-taxi-data-10M.csv -> (discarded) 3325605752 -> 794873295 [23.90%]; 437.5MB/s

10gb.tar -> (discarded) 10065157632 -> 5650133605 [56.14%]; 3137.9MB/s
10gb.tar -> (discarded) 10065157632 -> 5250102961 [52.16%]; 390.1MB/s

consensus.db.10gb -> (discarded) 10737418240 -> 4542443833 [42.30%]; 3008.2MB/s
consensus.db.10gb -> (discarded) 10737418240 -> 4272335541 [39.79%]; 248.8MB/s
```
@klauspost
Update tests and docs.
e0b6170
@klauspost
Don't retest same offsets.
e590bd2
@klauspost
Update benchmarks
bdb7aab
@klauspost klauspost merged commit fb371c6 into master Jan 13, 2021
@klauspost klauspost deleted the s2-add-best-compression branch January 13, 2021 14:16
This was referenced Mar 8, 2021
Open
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@klauspost