AttributeError: type object 'tqdm' has no attribute '_lock'

[NielsRogge]

Describe the bug

Getting a tqdm issue when writing a Dask dataframe to the hub.

Similar to huggingface/datasets#6066. Using latest Datasets version doesn't seem to resolve it

Steps to reproduce the bug

This is a minimal reproducer:

import dask.dataframe as dd
import pandas as pd
import random

import huggingface_hub

data = {"number": [random.randint(0,10) for _ in range(1000)]}
df = pd.DataFrame.from_dict(data)
dataframe = dd.from_pandas(df, npartitions=1)
dataframe = dataframe.repartition(npartitions=2)

repo_id = "nielsr/test-dask"
repo_path = f"hf://datasets/{repo_id}"
huggingface_hub.create_repo(repo_id=repo_id, repo_type="dataset", exist_ok=True)
dd.to_parquet(dataframe, path=f"{repo_path}/data")

Note: I'm intentionally repartioning the Dask dataframe to 2 partitions, as it does work when only having one partition.

Expected behavior

Would expect to write to the hub without any problem.

Environment info

Datasets version 2.14.4

[mariosasko]

This snippet doesn't contain datasets code, so I think it can be transferred to huggingface_hub, no?

[NielsRogge]

Oh yes, feel free to transfer

[Wauplin]

@NielsRogge What's your full stacktrace? And which version of tqdm are you using ?

pip freeze | grep tqdm

[Wauplin]

Ah I see, so the issue has already being identified and fixed in huggingface/datasets#6068 ?
We can make the exact same fix for huggingface_hub as well. But very weird that I can't reproduce myself and that no-one complained so far 😕

EDIT: I'll not have time to fix this today and I'll be off next week. @NielsRogge If you need it right now, feel free to edit the tqdm class here. I don't want to rush the fix now as I'm not able to reproduce the bug yet.