Add a utility to list cached things

[sgugger]

In Transformers, we have this function that returns the list of cached models. It would be nice to rely on an util in this repository instead (not limited to models) that would list:

• cached repos
• revisions cached for each repos (replacing commit hashes with refs when they exist)
• files cached for a given repo and revision

[LysandreJik]

This would be a nice exercise to dive into the way the cache is structured and understand it thoroughly. @Wauplin, would you like to try your hand at it?

[Wauplin]

@LysandreJik Thanks for pinging me on this one. It's a great idea, I'll assign the issue to myself and get back to you when I'm starting to work on it.

[Wauplin]

@sgugger (cc @LysandreJik) Quick question as I am trying to understand where caching logic belongs. You said that the get_cached_models will most probably live in huggingface_repo since it's not specific to models.

Does that mean that functions like cached_file and try_to_load_from_cache are also meant to be moved at some point ? And if yes, would that be part of the same process ? I am asking since I feel get_cached_models will heavily depends on the cache structure.

[sgugger]

cached_file is just a wrapper around hf_hub_download which is in huggingface_hub. I'll actually make a PR to move try_to_load_from_cache to huggingface_hub later today/tomorrow as it should live there IMO.

You should rely on the new cache structure (so as created by hf_hub_downlaod for instance) I think, but I'll let others correct me if I'm wrong.

[Wauplin]

Ok thanks for the info ! This precision definitely helps as caching in huggingface_hub is still a bit blurry for me 🙄

[mariosasko]

Great proposal! I guess this is somewhat related to huggingface/datasets#2631, in which we discussed how great it would be to have a CLI tool for easy inspection/deletion of the datasets cache.

[julien-c]

NB: for shared terminology, when we talk about "new cache structure" (aka. "cache file layout"), it's = what we shipped in #801 i.e. where do we store cached files on local disk

[Wauplin]

Closed by #990.
See doc and usage here: https://huggingface.co/docs/huggingface_hub/main/en/how-to-cache#scan-your-cache

[julien-c]

feel free to publicize/announce this =)