feat: cache updated workflow to avoid reconciling outdated objects. Fixes: #14833 #14780

[shuangkun]

Create new cache instead of writeInformerBack.

Fixes #14833
Fixes #14780
Fixes #12352

Motivation

Modifications

Verification

Documentation

[eduardodbr]

This PR is very similar to #14532 , both are adding a new cache in front of the informar cache, which might feel weird. Please review and join the discussion if you have a different opinion

[shuangkun]

This PR is very similar to #14532 , both are adding a new cache in front of the informar cache, which might feel weird. Please review and join the discussion if you have a different opinion

Oh, the problems we're trying to solve may be somewhat different, but there's also some overlap, such as double pod creation. I'm trying to address controller informer anomalies at very large scales (300,000+ workflows), such as event overwriting and workflow controllers missing workflows. Therefore, I'd like to remove the "writeInformerBack" method, which is the official recommended approach: instead of writing to the same InformerCache, create new expectations.