bug: file leak in object store cache

[killme2008]

What type of bug is this?

Other

What subsystems are affected?

Datanode

What happened?

We already implemented an object store cache in #928 , but it has a potential issue.

The LruCachePolicy keeps all active cached items in memory, and if we restart the datanode, those items are lost.We don't recover the cached items from disk when initializing the LRU cache. The cache will be reconstructed by file accessing.

So if a file is not accessed after that at all, the file is leaked on disk forever and may consume too much disk consumption.

What operating system did you use?

All os systems.

Relevant log output and stack trace

No response

How can we reproduce the bug?

1. Enable the object store caching by setting cache_path in storage config when using s3 or OSS.
2. Write and read some data.
3. Restart the datanode(or standalone node).
4. The files in cache_path may leak.

[e1ijah1]

We can set a maximum cache file size limit. When the total size of the cache files exceeds this limit, we can delete a portion of the cache files based on the last access time or by access count, ensuring that disk space is not consumed indefinitely.

[killme2008]

@e1ijah1 The object store cache is already an LRU cache with a maximum capacity. But the LRU cache doesn't recover the cached items from the disk when it's initialized. So the previous cached items may leak in file systems because the LRU cache doesn't manage them.

I am working on this issue at #1482