Recreated objects do not pass through predicate filters

[aharbis]

Current and expected behavior

    let (reader, writer) = reflector::store();
    let stream = reflector(writer, watcher(api, watcher::Config::default()))
        .default_backoff()
        .applied_objects()
        .predicate_filter(predicates::generation);

    Controller::for_stream(stream, reader)

Using a predicate (such as generation) filter, events are dropped for objects which are replaced (create-delete-create) in scenarios where the replaced object is the same (according to the filter alone). In other words, using the generation filter, if you delete and recreate a resource with the same generation, the new resource will not pass through the filter even though it is a new / distinct resource at that time.

For example:

1. Create resource (generation=1), controller reconciles
2. Delete resource
3. Recreate resource (generation=1), controller does not reconcile
4. Edit resource (generation=2), controller reconciles

IMO, this is a bug because the resources in (1) and (3) are actually distinct (different metadata.uid). Consider that the controller is responsible for managing some operand (e.g. Deployment). Using owner references and garbage collection, the Deployment would be deleted as part of (2). I would expect the Deployment to be recreated at (3), but it is not due to the predicate filter.

Possible solution

The predicate cache could consider a resource's uid as part of the key hashing algorithm.

Additional context

This appears to be due to kube's ObjectRef type not considering extra for its Hash impl, as ObjectRef is the key for the Predicate cache (HashMap).

pub struct ObjectRef<K: Lookup + ?Sized> {
    // ...

    /// Extra information about the object being referred to
    ///
    /// This is *not* considered when comparing objects, but may be used when converting to and from other representations,
    /// such as [`OwnerReference`] or [`ObjectReference`].
    #[educe(Hash(ignore), PartialEq(ignore))]
    pub extra: Extra,
}

I'm guessing there was a specific reason this extra data was ignored in the Hash impl (perhaps memory footprint / overhead for deleted objects?). However, it leads to (imo) unintuitive behavior in the predicate filter implementation so may be worth reconsidering or using a different type for the cache.

Environment

OCP 4.18

Configuration and features

k8s-openapi = { version = "0.25", features = ["latest", "schemars"] }
kube = { version = "1", features = [
    "runtime",
    "derive",
    "client",
    "unstable-runtime",
] }

YAML

No response

Affected crates

kube-runtime

Would you like to work on fixing this bug?

maybe