Dataset.push_to_hub of a canonical dataset creates an additional dataset under the user namespace

[albertvillanova]

While using Dataset.push_to_hub of a canonical dataset, an additional dataset was created under my user namespace.

Steps to reproduce the bug

The command:

commit_info = ds.push_to_hub(
    "caner",
    config_name="default",
    commit_message="Convert dataset to Parquet",
    commit_description="Convert dataset to Parquet.",
    create_pr=True,
    token=token,
)

creates the additional dataset albertvillanova/caner.

[albertvillanova]

It is caused by these code lines:

datasets/src/datasets/dataset_dict.py

Lines 1688 to 1694 in 9d6d161

	_ = api.create_repo(
	repo_id,
	token=token,
	repo_type="dataset",
	private=private,
	exist_ok=True,
	)

[albertvillanova]

Also note the information in the docstring:

datasets/src/datasets/dataset_dict.py

Lines 1582 to 1585 in 9d6d161

	repo_id (`str`):
	The ID of the repository to push to in the following format: `<user>/<dataset_name>` or
	`<org>/<dataset_name>`. Also accepts `<dataset_name>`, which will default to the namespace
	of the logged-in user.

Also accepts <dataset_name>, which will default to the namespace of the logged-in user.

This behavior was "reverted" by the PR:

• Support push_to_hub canonical datasets #6519

We have therefore contradictory requirements. We should decide:

• whether to support passing dataset_namespace without user/org that defaults to the logged-in user (and not support canonical datasets)
• or vice-versa, to support canonical datasets and not support passing only dataset_name

As canonical datasets are "deprecated" (and will eventually disappear), I would choose the first option. However, if so, the Space to convert datasets to Parquet will not work for canonical datasets: https://huggingface.co/spaces/albertvillanova/convert-dataset-to-parquet

[mariosasko]

IIUC, this could also be "fixed" by create_repo("dataset_name") not defaulting to create_repo("user/dataset_name") (when the user's token is available), which would be consistent with the rest of the HfApi ops used in the push_to_hub implementation. This is a (small) breaking change for huggingface_hub, but justified to make the API more consistent.

[albertvillanova]

I tag @Wauplin to have his opinion as well.

[Wauplin]

Hmm, creating repo with implicit namespace (e.g. create_repo("dataset_name")) is a convenient feature used in a lot of integrations. It is not consistent with other HfApi methods specifically because it is the method to create repos. Once the repo is created, the return value provides the explicit repo_id (namespace/repo_name) that has to be passed to every HfApi method. Otherwise, libraries/scripts would often need to do a whoami call to get the namespace before creating a repo.

Another solution for #6597 (comment) could be that implicit namespace is allowed (same as today) except if the repo_id is in a hard-coded list of canonical datasets. This list can be maintained automatically and should be slowly decreasing. Caveat: as a normal user I wouldn't be able to implicitly push to imagenet-1k if I wanted to push to Wauplin/imagenet-1k. Shouldn't be too problematic, no? Worse case, would need to add a whoami call and allow implicit-canonical-name for non-HF users for instance (a bit too over-engineered IMO but doable).

[albertvillanova]

As canonical datasets are going to disappear in the following couple of months, I would not make any effort on their support.

I propose reverting #6519, so that the behavior of push_to_hub is aligned with the one described in its dosctring: "Also accepts <dataset_name>, which will default to the namespace of the logged-in user."

I'm opening a PR.