Add args description to DatasetInfo #2384
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lhoestq
approved these changes
May 21, 2021
Member
lhoestq
left a comment
lhoestq
left a comment
There was a problem hiding this comment.
Thanks :) added some suggestions
src/datasets/info.py
Outdated
| homepage (str): A URL to the official homepage for the dataset. | ||
| license (str): The dataset's license. | ||
| features (Features, optional): The features used to specify the dataset's columns, types and conversion methods. | ||
| post_processed (PostProcessedInfo, optional): |
Member
There was a problem hiding this comment.
Suggested change
| post_processed (PostProcessedInfo, optional): | |
| post_processed (PostProcessedInfo, optional): Info regarding the resources of a possible post-processing of a dataset. It can contain the information of an index for example. |
src/datasets/info.py
Outdated
| splits (dict, optional): The mapping between split name and metadata. | ||
| download_checksums (dict, optional): The mapping between the URL to download the dataset's checksums and corresponding metadata. | ||
| download_size (int, optional): The size of the compressed dataset in bytes. | ||
| post_processing_size (int, optional): |
Member
There was a problem hiding this comment.
Suggested change
| post_processing_size (int, optional): | |
| post_processing_size (int, optional): Size of the dataset after the post-processing, if any. |
src/datasets/info.py
Outdated
| download_checksums (dict, optional): | ||
| download_size (int, optional): | ||
| supervised_keys (SupervisedKeysData, optional): Specifies the input feature and the label for supervised learning if applicable for the dataset. | ||
| builder_name (str, optional): The name of the :class:`GeneratorBasedBuilder` subclass used to create the dataset. Usually matched to the corresponding script name, but with CamelCase instead of snake_case. |
Member
There was a problem hiding this comment.
Suggested change
| builder_name (str, optional): The name of the :class:`GeneratorBasedBuilder` subclass used to create the dataset. Usually matched to the corresponding script name, but with CamelCase instead of snake_case. | |
| builder_name (str, optional): The name of the :class:`GeneratorBasedBuilder` subclass used to create the dataset. Usually matched to the corresponding script name. It is also the snake_case version of the dataset builder class name. |
src/datasets/info.py
Outdated
| splits (dict, optional): | ||
| download_checksums (dict, optional): | ||
| download_size (int, optional): | ||
| supervised_keys (SupervisedKeysData, optional): Specifies the input feature and the label for supervised learning if applicable for the dataset. |
Member
There was a problem hiding this comment.
Suggested change
| supervised_keys (SupervisedKeysData, optional): Specifies the input feature and the label for supervised learning if applicable for the dataset. | |
| supervised_keys (SupervisedKeysData, optional): Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS). |
src/datasets/info.py
Outdated
| description (str): A description of the dataset. | ||
| citation (str): A BibTeX citation of the dataset. | ||
| homepage (str): A URL to the official homepage for the dataset. | ||
| license (str): The dataset's license. |
Member
There was a problem hiding this comment.
Suggested change
| license (str): The dataset's license. | |
| license (str): The dataset's license. It can be the name of the license or a paragraph containing the terms of the license. |
src/datasets/info.py
Outdated
| size_in_bytes (int, optional): | ||
| task_templates (List[TaskTemplate], optional): | ||
| dataset_size (int, optional): The combined size of the Apache Arrow tables for all splits in bytes. | ||
| size_in_bytes (int, optional): The combined size of all files associated with the dataset. |
Member
There was a problem hiding this comment.
Suggested change
| size_in_bytes (int, optional): The combined size of all files associated with the dataset. | |
| size_in_bytes (int, optional): The combined size of all files associated with the dataset (downloaded files + arrow files) |
src/datasets/info.py
Outdated
| version (str or Version, optional): The version of the dataset. | ||
| splits (dict, optional): The mapping between split name and metadata. | ||
| download_checksums (dict, optional): The mapping between the URL to download the dataset's checksums and corresponding metadata. | ||
| download_size (int, optional): The size of the compressed dataset in bytes. |
Member
There was a problem hiding this comment.
Suggested change
| download_size (int, optional): The size of the compressed dataset in bytes. | |
| download_size (int, optional): The size of the files to download to generate the dataset, in bytes. |
src/datasets/info.py
Outdated
| citation (str): A BibTeX citation of the dataset. | ||
| homepage (str): A URL to the official homepage for the dataset. | ||
| license (str): The dataset's license. | ||
| features (Features, optional): The features used to specify the dataset's columns, types and conversion methods. |
Member
There was a problem hiding this comment.
Suggested change
| features (Features, optional): The features used to specify the dataset's columns, types and conversion methods. | |
| features (Features, optional): The features used to specify the dataset's columns types. |
Member
Author
|
Thanks for the suggestions! I've included them and made a few minor tweaks along the way |
Member
|
Please merge master into this branch to fix the CI, I just fixed metadata validation tests. |
JayantGoel001
added a commit
to JayantGoel001/datasets-1
that referenced
this pull request
May 22, 2021
Add args description to DatasetInfo (huggingface#2384)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #2354
I am not sure what
post_processedandpost_processing_sizecorrespond to, so have left them empty for now. I also took a guess at some of the other fields likedataset_sizevssize_in_bytes, so might have misunderstood their meaning.