Select columns of Dataset or DatasetDict #5480

daskol · 2023-01-27T20:06:16Z

Close #5474 and #5468.

HuggingFaceDocBuilderDev · 2023-01-31T13:25:31Z

The documentation is not available anymore as the PR was closed or merged.

mariosasko

Thanks! I've suggested some nits for consistency with the rest of the codebase, but looks great other than that!

src/datasets/arrow_dataset.py

src/datasets/iterable_dataset.py

Co-authored-by: Mario Šaško <[email protected]>

mariosasko

Last nit

src/datasets/iterable_dataset.py

Co-authored-by: Mario Šaško <[email protected]>

mariosasko

Looks all good now, thanks!

lhoestq

Awesome thank you ! I just added some small suggestions :)

src/datasets/arrow_dataset.py

Co-authored-by: Quentin Lhoest <[email protected]>

lhoestq

Thanks :)

github-actions · 2023-02-13T10:07:04Z

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009963 / 0.011353 (-0.001390)	0.005512 / 0.011008 (-0.005496)	0.100495 / 0.038508 (0.061987)	0.039929 / 0.023109 (0.016820)	0.299749 / 0.275898 (0.023850)	0.372330 / 0.323480 (0.048850)	0.008689 / 0.007986 (0.000703)	0.004334 / 0.004328 (0.000006)	0.076469 / 0.004250 (0.072218)	0.048091 / 0.037052 (0.011039)	0.303884 / 0.258489 (0.045395)	0.352747 / 0.293841 (0.058906)	0.038941 / 0.128546 (-0.089605)	0.012541 / 0.075646 (-0.063105)	0.334227 / 0.419271 (-0.085044)	0.048802 / 0.043533 (0.005269)	0.295800 / 0.255139 (0.040661)	0.316222 / 0.283200 (0.033022)	0.108246 / 0.141683 (-0.033437)	1.452735 / 1.452155 (0.000580)	1.466293 / 1.492716 (-0.026423)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.010497 / 0.018006 (-0.007510)	0.507427 / 0.000490 (0.506937)	0.003054 / 0.000200 (0.002854)	0.000084 / 0.000054 (0.000030)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.029529 / 0.037411 (-0.007883)	0.114151 / 0.014526 (0.099625)	0.120599 / 0.176557 (-0.055957)	0.161881 / 0.737135 (-0.575255)	0.127669 / 0.296338 (-0.168669)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.399631 / 0.215209 (0.184421)	3.992997 / 2.077655 (1.915343)	1.803770 / 1.504120 (0.299650)	1.612301 / 1.541195 (0.071106)	1.717846 / 1.468490 (0.249356)	0.706753 / 4.584777 (-3.878024)	3.798224 / 3.745712 (0.052512)	2.169733 / 5.269862 (-3.100128)	1.358264 / 4.565676 (-3.207413)	0.086828 / 0.424275 (-0.337447)	0.012606 / 0.007607 (0.004999)	0.512085 / 0.226044 (0.286041)	5.101491 / 2.268929 (2.832563)	2.285688 / 55.444624 (-53.158936)	1.955160 / 6.876477 (-4.921317)	2.045887 / 2.142072 (-0.096186)	0.878836 / 4.805227 (-3.926392)	0.166483 / 6.500664 (-6.334181)	0.062656 / 0.075469 (-0.012814)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.215152 / 1.841788 (-0.626636)	15.436187 / 8.074308 (7.361879)	14.489951 / 10.191392 (4.298559)	0.199019 / 0.680424 (-0.481404)	0.029148 / 0.534201 (-0.505053)	0.440309 / 0.579283 (-0.138974)	0.452041 / 0.434364 (0.017677)	0.527102 / 0.540337 (-0.013236)	0.634302 / 1.386936 (-0.752634)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007814 / 0.011353 (-0.003539)	0.005582 / 0.011008 (-0.005427)	0.075466 / 0.038508 (0.036958)	0.034421 / 0.023109 (0.011312)	0.342345 / 0.275898 (0.066447)	0.389943 / 0.323480 (0.066463)	0.006346 / 0.007986 (-0.001639)	0.004442 / 0.004328 (0.000113)	0.074440 / 0.004250 (0.070190)	0.056383 / 0.037052 (0.019331)	0.340293 / 0.258489 (0.081804)	0.394416 / 0.293841 (0.100575)	0.037217 / 0.128546 (-0.091330)	0.012597 / 0.075646 (-0.063050)	0.087005 / 0.419271 (-0.332267)	0.051626 / 0.043533 (0.008094)	0.336690 / 0.255139 (0.081551)	0.369143 / 0.283200 (0.085943)	0.110764 / 0.141683 (-0.030919)	1.459003 / 1.452155 (0.006849)	1.557333 / 1.492716 (0.064617)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.319596 / 0.018006 (0.301590)	0.514697 / 0.000490 (0.514207)	0.005286 / 0.000200 (0.005086)	0.000086 / 0.000054 (0.000032)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.032579 / 0.037411 (-0.004832)	0.111094 / 0.014526 (0.096568)	0.127827 / 0.176557 (-0.048730)	0.169967 / 0.737135 (-0.567168)	0.133149 / 0.296338 (-0.163189)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.424637 / 0.215209 (0.209428)	4.217889 / 2.077655 (2.140235)	2.044844 / 1.504120 (0.540724)	1.863513 / 1.541195 (0.322319)	1.975674 / 1.468490 (0.507184)	0.695493 / 4.584777 (-3.889284)	3.815562 / 3.745712 (0.069850)	3.534427 / 5.269862 (-1.735435)	1.684874 / 4.565676 (-2.880802)	0.085560 / 0.424275 (-0.338715)	0.012439 / 0.007607 (0.004832)	0.541231 / 0.226044 (0.315187)	5.287166 / 2.268929 (3.018237)	2.596622 / 55.444624 (-52.848002)	2.315913 / 6.876477 (-4.560564)	2.418454 / 2.142072 (0.276381)	0.838947 / 4.805227 (-3.966281)	0.168149 / 6.500664 (-6.332515)	0.066439 / 0.075469 (-0.009030)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.264814 / 1.841788 (-0.576974)	15.861324 / 8.074308 (7.787016)	14.352515 / 10.191392 (4.161123)	0.167032 / 0.680424 (-0.513391)	0.017766 / 0.534201 (-0.516435)	0.421821 / 0.579283 (-0.157462)	0.426657 / 0.434364 (-0.007707)	0.526742 / 0.540337 (-0.013595)	0.623851 / 1.386936 (-0.763085)

* Allow to select columns of Dataset or DatasetDict * Add column selection method to autodoc index * Allow to select columns of IterableDataset or IterableDatasetDict * Fix formatting issues * Prefer sequential containers for building feature table Co-authored-by: Mario Šaško <[email protected]> * Rework feature descriptor update in select_columns Co-authored-by: Mario Šaško <[email protected]> * Apply suggestions from code review Co-authored-by: Quentin Lhoest <[email protected]> --------- Co-authored-by: Mario Šaško <[email protected]> Co-authored-by: Quentin Lhoest <[email protected]>

Allow to select columns of Dataset or DatasetDict

cc80c5d

daskol added 2 commits January 31, 2023 18:49

Add column selection method to autodoc index

7db71e4

Allow to select columns of IterableDataset or IterableDatasetDict

2ee8ba7

mariosasko linked an issue Feb 1, 2023 that may be closed by this pull request

Allow opposite of remove_columns on Dataset and DatasetDict #5468

Closed

Fix formatting issues

ff4bf9e

mariosasko reviewed Feb 7, 2023

View reviewed changes

src/datasets/arrow_dataset.py Show resolved Hide resolved

src/datasets/iterable_dataset.py Show resolved Hide resolved

Prefer sequential containers for building feature table

abedcb1

Co-authored-by: Mario Šaško <[email protected]>

This comment was marked as resolved.

Sign in to view

mariosasko reviewed Feb 8, 2023

View reviewed changes

src/datasets/iterable_dataset.py Outdated Show resolved Hide resolved

Rework feature descriptor update in select_columns

6e8de44

Co-authored-by: Mario Šaško <[email protected]>

mariosasko approved these changes Feb 9, 2023

View reviewed changes

lhoestq reviewed Feb 12, 2023

View reviewed changes

src/datasets/arrow_dataset.py Outdated Show resolved Hide resolved

src/datasets/arrow_dataset.py Outdated Show resolved Hide resolved

Apply suggestions from code review

c5c839e

Co-authored-by: Quentin Lhoest <[email protected]>

lhoestq approved these changes Feb 13, 2023

View reviewed changes

lhoestq merged commit 69b1975 into huggingface:main Feb 13, 2023

daskol deleted the add/column-projection branch February 13, 2023 11:10

unifyh mentioned this pull request Aug 3, 2023

[Docs] The "Process" how-to guide lacks description of select_columns function #6116

Closed

renovate bot mentioned this pull request Nov 14, 2023

chore(deps): update dependency datasets to v2 - autoclosed corypaik/renovate-loop#3

Closed

1 task

Select columns of Dataset or DatasetDict #5480

Select columns of Dataset or DatasetDict #5480

Uh oh!

Conversation

daskol commented Jan 27, 2023

Uh oh!

HuggingFaceDocBuilderDev commented Jan 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mariosasko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

mariosasko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mariosasko left a comment

Choose a reason for hiding this comment

Uh oh!

lhoestq left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lhoestq left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 13, 2023

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HuggingFaceDocBuilderDev commented Jan 31, 2023 •

edited

Loading