HfHubHTTPError - exceeded our hourly quotas for action: commit

[yuvalkirstain]

Describe the bug

I try to upload a very large dataset of images, and get the following error:

File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/hf_api.py:2712, in HfApi.create_commit(self, repo_id, operations, commit_message, commit_description, token, repo_type, revision, create_pr, num_threads, parent_commit, run_as_future)
   2710 try:
   2711     commit_resp = get_session().post(url=commit_url, headers=headers, data=data, params=params)
-> 2712     hf_raise_for_status(commit_resp, endpoint_name="commit")
   2713 except RepositoryNotFoundError as e:
   2714     e.append_to_message(_CREATE_COMMIT_NO_REPO_ERROR_MESSAGE)

File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py:301, in hf_raise_for_status(response, endpoint_name)
    297     raise BadRequestError(message, response=response) from e
    299 # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    300 # as well (request id and/or server error message)
--> 301 raise HfHubHTTPError(str(e), response=response) from e

HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/yuvalkirstain/pickapic_v2/commit/main (Request ID: Root=1-65112399-12d63f7d7f28bfa40a36a0fd)

You have exceeded our hourly quotas for action: commit. We invite you to retry later.

this makes it much less convenient to host large datasets on HF hub.

Steps to reproduce the bug

Upload a very large dataset of images

Expected behavior

the upload to work well

Environment info

• datasets version: 2.13.1
• Platform: Linux-5.15.0-1033-aws-x86_64-with-glibc2.31
• Python version: 3.10.11
• Huggingface_hub version: 0.15.1
• PyArrow version: 12.0.1
• Pandas version: 1.5.3

[julien-c]

how is your dataset structured? (file types, how many commits and files are you trying to push, etc)

[yuvalkirstain]

I succeeded in uploading it after several attempts with an hour gap between each attempt (inconvenient but worked). The final dataset is here, code and context to the dataset can be found here.
I can close the issue if this behavior is intended, as most users probably do not need to upload large-scale datasets.

[mariosasko]

We could fix this by creating a single commit for all the (Parquet) shards in push_to_hub instead of one commit per shard, as we currently do.

@Wauplin Any updates on the 2-step commit process suggested by you that we need to implement this?

[Wauplin]

Any updates on the 2-step commit process suggested by you that we need to implement this?

Re-prioritizing this, sorry. Will let you know but probably can be done this week.