warning - add: Failed to allocate an entry for handle 0

[yongshengma]

Hello,

This message repeatedly shows up:

Mar 5 11:53:16 node07 volumedriver_fs.sh: 2019-03-05 11:53:16 626813 +0800 - node07 - 3106/0x00007fe465663700 - volumedriverfs/ClusterClusterCache - 0000000000278f14 - warning - add: Failed to allocate an entry for handle 0 - are all devices gone or all entries consumed by other namespaces?

Is it because those SSDs with Write role are exhausted?

Best regards,
Yongsheng

[redlicha]

Hi Yongsheng,

Some background: the read (aka cluster) cache can hold multiple cache namespaces (not to be confused with the backend namespaces). These are identified by handles. "Handle 0" is a special cache namespace used for content based caching. Other cache namespaces are used for location based caching per volume.
Volumes can be configured to use (and update) the read cache either on write, on read or not at all.

With that, the above message suggests that at least one volume is configured to use the read cache in content based mode but cannot add an entry to it. This is expected if either

• all read cache devices were offlined due to errors (if there were any configured in the first place)
• all read cache entries are in use by other cache namespaces for volumes in location based cache mode
  .

The relevant volumedriver Python API to inspect/modify the per-volume settings is:

• StorageRouterClient.{get,set}_readcache_behaviour
• StorageRouterClient.{get,set}_readcache_mode
• StorageRouterClient.{get,set}_readcache_limit
• the handle used by a volume is in the cluster_cache_handle property of the object returned by StorageRouterClient.info_volume
  .