Add ACTIVE-EXPIRY and ACTIVE-DEFRAG for hash objects with volatile items

[xbasel]

This change adds support for active expiration of hash fields with TTLs (Hash Field Expiration), building on the existing key-level expiry system.

Field TTL metadata is tracked in volatile sets associated with each hash key. Expired fields are reclaimed incrementally by the active expiration loop, using a new job type to alternate between key expiry and field expiry within the same logic and effort budget.

Both key and field expiration now share the same scheduler infrastructure.

Alternating job types ensures fairness and avoids starvation, while keeping CPU usage predictable.

+-----------------+
|     DB          |
+-----------------+
        |
        v
+---------------------+
|     myhash          |  (key with TTL)
+---------------------+
        |
        v
+------------------------------------+
| fields (hashType)                  |
|  - field1                          |
|  - field2                          |
|  - fieldN                          |
+------------------------------------+
        |
        v
+------------------------------------+
| volatile set (field-level TTL)     |
|  - field1 expires at T1            |
|  - field5 expires at T5            |
+------------------------------------+

No new configuration was introduced; the existing active-expire-effort and time budget are reused for both key and field expiry.

Also active defrag for volatile sets is added.

[xbasel]

THIS IS NOT LONGER RELEVANT

Please note that at first glance, it seems logical to try and unify the active expiry cycle for keys and for hash fields with TTLs. After all, both processes are “expire something with a timestamp,” and both follow an incremental, time-bounded loop, and esepcially that the existing key expiry mechnaism is already optmized to not starve the engine. It feels natural to try to generalize the mechanism using function pointers, callbacks, or generic iterators to share the same control flow.

However, for me this fell apart because of a fundamental structural difference:

• Key expiry operates on the top-level keyspace, where every db->key is an independent unit, randomly sampled, with a well-defined expiry time. Once you pick a key, you only have to check its expiry and delete it if needed. That is a single-level sampling problem.

-Field expiry is different. You do not have a flat namespace of all fields across the DB. Instead, you have a two-level hierarchy:

    sample a key from the DB containing volatile fields

    then scan inside its attached volatile set to find expired fields

    then decide if the hash itself should be deleted if all fields expire

Sampling: sampling volatile keys is meaningless since a single volatile key might hold 99% of the expirables, and treating all entries as a flat sample pool doesn’t work because volatile sets are already ordered and are scanned semi-deterministically. Sampling fundamentally behaves differently for keys vs. fields, making separate loops cleaner and correct.

Unifying expiry would require sampling across multiple levels, separate iterators for keys and volatile sets, tracking state across two iteration layers, handling mid-scan key deletions, and preventing starvation from large volatile sets.
While technically possible with a generic framework, the cost in complexity abd risk of subtle bugs, and maintenance is simply not justified IMHO.

[ranshid]

just dumping some immediate comments. will follow up soon

[ranshid]

Did not review the entire change yet.

Lets re-iterate after this bulk

[ranshid]

It suddenly hit me that we are no longer calculating the expire cycle latency correctly.
Also the expired_stale percentage should be calculated differently per each job type.

@xbasel / @JimB123 care to take a quick look?

[xbasel]

It suddenly hit me that we are no longer calculating the expire cycle latency correctly. Also the expired_stale percentage should be calculated differently per each job type.

@xbasel / @JimB123 care to take a quick look?

Mixing both key and field expiry in stat_expired_stale_perc doesn't seem right. We should track them independently.
As for latency, expire-latency now reflects both key and field TTL processing since they share the same handler, but it's unclear if we should record latency once in the outer function or separately (as its done now).

[ranshid]

It suddenly hit me that we are no longer calculating the expire cycle latency correctly. Also the expired_stale percentage should be calculated differently per each job type.
@xbasel / @JimB123 care to take a quick look?

Mixing both key and field expiry in stat_expired_stale_perc doesn't seem right. We should track them independently. As for latency, expire-latency now reflects both key and field TTL processing since they share the same handler, but it's unclear if we should record latency once in the outer function or separately (as its done now).

Right now I am NOT mixing the stat_expired_stale_perc, which is not the case before the last fix. NOW we do track them independently.

expire-latency now reflects both key and field TTL processing since they share the same handler, but it's unclear if we should record latency once in the outer function or separately (as its done now).

IMO doing it differently will just change the reporting of the active expiry statistic. Since we decide to share the CPU time for both expiration jobs, it should be tracked for both.

[JimB123]

Active Expiry code and Active Defrag code reviewed.

[ranshid]

This was manually cherry-picked into valkey-io#2089