Objectmeta of embedded object goes missing during CRD conversion

[champak]

Hi folks,
I ran into an odd issue with CRD conversion using controller-tools v0.2.4/kubebuilder 2.2.0/apiserver 17.1 This CRD has embedded native specs like the PodTemplateSpec in its spec. I found that in the conversion functions the ObjectMeta of the embedded PodTemplateSpec is nulled out. Other attributes look OK. There is nothing in the validation callback. I had needed to add preserveUnknownFields: false to the CRD to be able to deploy the yaml; not sure if that has some sideffect

The ObjectMeta for the CRD itself shows up fine. It is the embedded spec's ObjectMeta that seemed to be affected. Does this sound like any known issue ? One that looked somewhat related from a search was #279 with comments from @DirectXMan12 but was not quite the same problem.
Thanks for any insight !

[DirectXMan12]

/kind bug
/priority critical-urgent
/needs-more-info

any idea if this is happening during the conversion webhook? Can you post a reproducer?

[champak]

I had added logs to the ConvertTo(dstRaw conversion.Hub) and ConvertFrom(srcRaw conversion.Hub) functions in the file api/v<>/<>_conversion.go to display the contents of the ObjectMeta from the CRD and the embedded PodTemplateSpec. The ObjectMeta.Name etc from the CRD level had the expected contents but the contents from the embedded podtemplatespec were nulled. Are there other places that may be useful to log ? I was wondering if the api-server itself could have chosen to not send the metadata of the embedded spec to the webhook. Have not looked yet at what all layers are in play between the api-server and the ConvertTo/From functions. I am wondering if I am missing some knob. I was following the kubebuilder book on the conversion.

The CRD structure is on the lines of ;
CRDfooSpec {
barspec
}
barspec {
Pod corev1.PodTemplateSpec ...
...
}

The embedded podtemplate spec contents are visible in the last-applied-configuration annotation of the CRD so the info did make it to the api-server. I will try to post a repro. Thanks !

[champak]

I created a simple repro for this issue. @DirectXMan12 please see https://github.com/champak/repro-kbuilder-version-issue .

If the sample yaml at https://github.com/champak/repro-kbuilder-version-issue/blob/master/config/samples/left_v1alpha1_repro.yaml

is applied, the logs at the webhook converter look like

2020/06/02 19:42:14 ConvertFrom : working on repro-sample 
2020/06/02 19:42:14 ConvertFrom working on Pod  attr bar  ===> I tried printing the pod name; nothing shows up.
2020/06/02 19:42:30 ConvertFrom : working on repro-sample 
2020/06/02 19:42:30 ConvertFrom working on Pod  attr bar 

The name of the CRD "repro-sample" show up. The name of the Pod does not. The logs are from api/v1/repro_conversion.go. The versions are going between v1alpha1 and v1. I followed the kubebuilder scaffolds and book.

If logs elsewhere would tell us more send a ptr. Thanks for looking !

[champak]

And a get on the CRD looks like

$ kctl get repro.left.play.zone/repro-sample -o yaml | more
apiVersion: left.play.zone/v1
kind: Repro
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"left.play.zone/v1alpha1","kind":"Repro","metadata":{"annota
tions":{},"name":"repro-sample","namespace":"default"},"spec":{"foo":"bar","grap
h":{"dummy":"isthere","pod":{"metadata":{"name":"mia-pod-name"},"spec":{"contain
ers":[{"image":"busybox","name":"a-container"}]}}}}}
  creationTimestamp: "2020-06-02T19:42:30Z"
  generation: 1
  name: repro-sample
  namespace: default
  resourceVersion: "22387679"
  selfLink: /apis/left.play.zone/v1/namespaces/default/reproes/repro-sample
  uid: fac7fae5-3592-4925-a454-69a98e20e1bf
spec:
  foo: bar
  graph:
    dummy: ""
    pod:
      metadata: {}
      spec:
        containers: null
status: {}

The pod metadata is missing. Similar behavior for embedding other native types like PVCs or services. Perhaps I am missing some setting but found nothing yet.

[champak]

@DirectXMan12 let me know if I try to get any other details etc. that may help. I hope the example provides enough info. One thing I had noted is that the meta-data was missing even before we hit the convert functions. For the v1alpha1 CR version we hit the reconcile loop before going to the convert callbacks and the bits are already missing. That made me wonder if it is the apiserver that may have pruned the info due to some setting. The operator initialization log had

2020-06-04T19:15:50.982Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": "127.0.0.1:8080"}
2020-06-04T19:15:50.982Z	INFO	controller-runtime.builder	skip registering a mutating webhook, admission.Defaulter interface is not implemented	{"GVK": "left.play.zone/v1alpha1, Kind=Repro"}
2020-06-04T19:15:50.982Z	INFO	controller-runtime.builder	skip registering a validating webhook, admission.Validator interface is not implemented	{"GVK": "left.play.zone/v1alpha1, Kind=Repro"}
2020-06-04T19:15:50.982Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/convert"}
2020-06-04T19:15:50.982Z	INFO	controller-runtime.builder	conversion webhook enabled	{"object": {"metadata":{"creationTimestamp":null},"spec":{"graph":{"pod":{"metadata":{"creationTimestamp":null},"spec":{"containers":null}},"dummy":""}},"status":{}}}
2020-06-04T19:15:51.080Z	INFO	setup	starting manager
2020-06-04T19:15:51.082Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2020-06-04T19:15:51.182Z	INFO	controller-runtime.webhook.webhooks	starting webhook server
2020-06-04T19:15:51.183Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2020-06-04T19:15:51.185Z	INFO	controller-runtime.webhook	serving webhook server	{"host": "", "port": 9443}
2020-06-04T19:15:51.186Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
2020-06-04T19:16:08.575Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ConfigMap","namespace":"metarepro-system","name":"controller-leader-election-helper","uid":"9bb0c9fc-a56d-4436-9a62-57329c1a3e77","apiVersion":"v1","resourceVersion":"22876473"}, "reason": "LeaderElection", "message": "metarepro-controller-manager-5575c95b5c-c9n48_6428cfcb-da12-4e36-8a6b-9d1e6e31aff6 became leader"}
2020-06-04T19:16:08.576Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "repro", "source": "kind source: /, Kind="}
2020-06-04T19:16:08.877Z	INFO	controller-runtime.controller	Starting Controller	{"controller": "repro"}
2020-06-04T19:16:08.977Z	INFO	controller-runtime.controller	Starting workers	{"controller": "repro", "worker count": 1}

[champak]

@DirectXMan12 please let me know if you need any other info to take a look into this issue. Thanks !

[DirectXMan12]

sorry. Did some digging thursday / friday. I think you're hitting unknown field elision -- need to confirm that, but I think it's because conversion requires preserveUnknownFields: false, which will strip fields from the under-specified ObjectMeta. Related: #385, #395

[champak]

@DirectXMan12 Thanks for taking a look ! If there is any hack that would allow crd conversion to work w/o needing preserveUnknownFields: false let me know and I can try it out. I could not tell if the changes from #385, #395 will get pulled into controller-gen main. I did not follow what functional need makes preserveUnknownFields: false mandatory. I can see it is desirable since dealing with conversion of unstructured data is ill-defined. But in many cases we merely want the unstructured blob to flow over to the new CRD version without changes. Would be nice to have a means for that.

[DirectXMan12]

kubernetes itself requires preserveUnknownFields: false if conversion is enabled for v1beta1, and IIRC has it on by default in v1

[champak]

@DirectXMan12 thanks for explaining ; in that case it sounds like something on the lines of #395 would be needed. I did not see updates on the issue since Jan ; is a different approach under consideration ? Think it would be essential to preserve at least a few common fields from ObjectMeta like name, namespace to be able to make embeddable native resources work well in a versioned CRD. Base types like replicasets embed podspecs and can handle version changes. Would be awesome to be able to do that with CRDs as well.

[champak]

@DirectXMan12 is there any possibility of a fix or workaround for this issue in the near future ? Thanks.