knative serving not getting ready due to post install job `storage-version-migration-serving-serving-1.17.0`

[alexanderwanyoike]

Describe the bug

Kubernetes running on a GKE cluster

Server Version: v1.31.5-gke.1169000

Having issues when tring to setup knative serving via the operator

Installation was done via Helm charts with fluxcd using the following guide

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: knative-operator
  namespace: knative
spec:
  interval: 10m
  chart:
    spec:
      chart: knative-operator
      version: 1.17.3
      sourceRef:
        kind: HelmRepository
        name: knative-operator
---

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: knative-operator
  namespace: knative
spec:
  interval: 1m0s
  url: https://knative.github.io/operator

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving

Expected behavior
After installation of knative serving running

kubectl get KnativeServing knative-serving -n knative-serving

should result in a ready KnativeServing resource

Actual Behaviour

After installation of knative serving resource returns the following error KnativeServing

kubectl get KnativeServing -n knative-serving -o yaml                     

apiVersion: v1
items:
- apiVersion: operator.knative.dev/v1beta1
  kind: KnativeServing
  metadata:
    creationTimestamp: "2025-03-01T16:14:26Z"
    finalizers:
    - knativeservings.operator.knative.dev
    generation: 1
    labels:
      kustomize.toolkit.fluxcd.io/name: flux-system
      kustomize.toolkit.fluxcd.io/namespace: flux-system
    name: knative-serving
    namespace: knative-serving
    resourceVersion: "10052980"
    uid: c3c3ea44-ed6d-4e70-9a76-82e08169ea3f
  status:
    conditions:
    - lastTransitionTime: "2025-03-01T16:14:40Z"
      status: "True"
      type: DependenciesInstalled
    - lastTransitionTime: "2025-03-01T16:36:47Z"
      status: "True"
      type: DeploymentsAvailable
    - lastTransitionTime: "2025-03-01T18:05:01Z"
      message: 'Install failed with message: Job.batch "storage-version-migration-serving-serving-1.17.0"
        is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"",
        GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0,
        CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>,
        DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"storage-version-migration-serving",
        "app.kubernetes.io/component":"storage-version-migration-job", "app.kubernetes.io/name":"knative-serving",
        "app.kubernetes.io/version":"1.17.0", "batch.kubernetes.io/controller-uid":"2ec2d1ee-d0df-45d9-b84a-da158441a069",
        "batch.kubernetes.io/job-name":"storage-version-migration-serving-serving-1.17.0",
        "controller-uid":"2ec2d1ee-d0df-45d9-b84a-da158441a069", "job-name":"storage-version-migration-serving-serving-1.17.0",
        "sidecar.istio.io/inject":"false"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil),
        Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil),
        InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"migrate",
        Image:"gcr.io/knative-releases/knative.dev/pkg/apiextensions/storageversion/cmd/migrate@sha256:c7ae2bf09e8f4236fad16705aa2ec69ab69e0f31ebd53159a6b0d734039538b4",
        Command:[]string(nil), Args:[]string{"services.serving.knative.dev", "configurations.serving.knative.dev",
        "revisions.serving.knative.dev", "routes.serving.knative.dev", "domainmappings.serving.knative.dev"},
        WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil),
        Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:1,
        scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1", Format:"DecimalSI"},
        "ephemeral-storage":resource.Quantity{i:resource.int64Amount{value:1073741824,
        scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1Gi", Format:"BinarySI"},
        "memory":resource.Quantity{i:resource.int64Amount{value:1048576000, scale:0},
        d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:100,
        scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"},
        "ephemeral-storage":resource.Quantity{i:resource.int64Amount{value:1073741824,
        scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1Gi", Format:"BinarySI"},
        "memory":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0},
        d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100Mi", Format:"BinarySI"}},
        Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil),
        RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil),
        VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil),
        ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil),
        TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File",
        ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(0xc02004d500),
        Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil),
        RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc03811cac0),
        ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil),
        ServiceAccountName:"controller", AutomountServiceAccountToken:(*bool)(nil),
        NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc014e47540), ImagePullSecrets:[]core.LocalObjectReference(nil),
        Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil),
        SchedulerName:"default-scheduler", Tolerations:[]core.Toleration{core.Toleration{Key:"kubernetes.io/arch",
        Operator:"Equal", Value:"amd64", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}},
        HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil),
        PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil),
        ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil),
        Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil),
        OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}:
        field is immutable'
      reason: Error
      status: "False"
      type: InstallSucceeded
    - lastTransitionTime: "2025-03-01T18:05:01Z"
      message: 'Install failed with message: Job.batch "storage-version-migration-serving-serving-1.17.0"
        is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"",
        GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0,
        CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>,
        DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"storage-version-migration-serving",
        "app.kubernetes.io/component":"storage-version-migration-job", "app.kubernetes.io/name":"knative-serving",
        "app.kubernetes.io/version":"1.17.0", "batch.kubernetes.io/controller-uid":"2ec2d1ee-d0df-45d9-b84a-da158441a069",
        "batch.kubernetes.io/job-name":"storage-version-migration-serving-serving-1.17.0",
        "controller-uid":"2ec2d1ee-d0df-45d9-b84a-da158441a069", "job-name":"storage-version-migration-serving-serving-1.17.0",
        "sidecar.istio.io/inject":"false"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil),
        Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil),
        InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"migrate",
        Image:"gcr.io/knative-releases/knative.dev/pkg/apiextensions/storageversion/cmd/migrate@sha256:c7ae2bf09e8f4236fad16705aa2ec69ab69e0f31ebd53159a6b0d734039538b4",
        Command:[]string(nil), Args:[]string{"services.serving.knative.dev", "configurations.serving.knative.dev",
        "revisions.serving.knative.dev", "routes.serving.knative.dev", "domainmappings.serving.knative.dev"},
        WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil),
        Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:1,
        scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1", Format:"DecimalSI"},
        "ephemeral-storage":resource.Quantity{i:resource.int64Amount{value:1073741824,
        scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1Gi", Format:"BinarySI"},
        "memory":resource.Quantity{i:resource.int64Amount{value:1048576000, scale:0},
        d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:100,
        scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"},
        "ephemeral-storage":resource.Quantity{i:resource.int64Amount{value:1073741824,
        scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1Gi", Format:"BinarySI"},
        "memory":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0},
        d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100Mi", Format:"BinarySI"}},
        Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil),
        RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil),
        VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil),
        ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil),
        TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File",
        ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(0xc02004d500),
        Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil),
        RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc03811cac0),
        ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil),
        ServiceAccountName:"controller", AutomountServiceAccountToken:(*bool)(nil),
        NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc014e47540), ImagePullSecrets:[]core.LocalObjectReference(nil),
        Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil),
        SchedulerName:"default-scheduler", Tolerations:[]core.Toleration{core.Toleration{Key:"kubernetes.io/arch",
        Operator:"Equal", Value:"amd64", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}},
        HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil),
        PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil),
        ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil),
        Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil),
        OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}:
        field is immutable'
      reason: Error
      status: "False"
      type: Ready
    - lastTransitionTime: "2025-03-01T16:14:26Z"
      status: "True"
      type: VersionMigrationEligible
    manifests:
    - /var/run/ko/knative-serving/1.17.0
    - /var/run/ko/ingress/1.17/istio
    observedGeneration: 1
    version: 1.17.0
kind: List
metadata:
  resourceVersion: ""

Knative release version

knative-operator version: v1.17
knative-serving version: latest (v1.17.0)
storage-version-migration-serving-serving version: 1.17.0

Additional context

• Networking layer set up with istio and istio installed separately

[alexanderwanyoike]

will move this to serving as its serving specific

[alexanderwanyoike]

knative/serving#15799

[alexanderwanyoike]

Actually been digging through the charts and seems that this could issue with the operator will re-open

[alexanderwanyoike]

For all who are struggling I just excluded the post install job not ideal but its a workaround 😢

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  version: "1.17.0"
  manifests:
  - URL: https://raw.githubusercontent.com/knative/operator/2f4a10ff2fc7c03afa449116df23c754905a9eda/cmd/operator/kodata/knative-serving/1.17.0/1-serving-crds.yaml
  - URL: https://raw.githubusercontent.com/knative/operator/2f4a10ff2fc7c03afa449116df23c754905a9eda/cmd/operator/kodata/knative-serving/1.17.0/2-serving-core.yaml
  - URL: https://raw.githubusercontent.com/knative/operator/2f4a10ff2fc7c03afa449116df23c754905a9eda/cmd/operator/kodata/knative-serving/1.17.0/3-serving-hpa.yaml

[dprotaso]

cc @houshengbo

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[dprotaso]

bump @houshengbo

[csechrist]

I am also having this issue, I am getting the following errors:

{"severity":"INFO","timestamp":"2025-07-21T22:02:40.762152094Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:145","message":"Merging","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","diff":"{\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"controller\"}],\"containers\":[{\"name\":\"controller\",\"resources\":{\"requests\":{\"memory\":\"100Mi\"}}}]}}}}"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:40.762895244Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:208","message":"Updating","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","name":"knative-serving/controller","type":"apps/v1, Kind=Deployment"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:41.161671064Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:145","message":"Merging","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","diff":"{\"spec\":{\"selector\":{\"matchLabels\":{\"app\":\"webhook\"}}}}"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:41.162043644Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:208","message":"Updating","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","name":"knative-serving/webhook-pdb","type":"policy/v1, Kind=PodDisruptionBudget"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:41.356338184Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:145","message":"Merging","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","diff":"{\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"webhook\"}],\"containers\":[{\"name\":\"webhook\",\"resources\":{\"requests\":{\"memory\":\"100Mi\"}}}]}}}}"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:41.357446534Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:208","message":"Updating","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","name":"knative-serving/webhook","type":"apps/v1, Kind=Deployment"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:41.754035535Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:145","message":"Merging","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","diff":"{\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"autoscaler-hpa\"}],\"containers\":[{\"name\":\"autoscaler-hpa\",\"resources\":{\"requests\":{\"cpu\":\"30m\",\"memory\":\"40Mi\"}}}]}}}}"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:41.754802195Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:208","message":"Updating","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","name":"knative-serving/autoscaler-hpa","type":"apps/v1, Kind=Deployment"}
{"severity":"WARNING","timestamp":"2025-07-21T22:02:41.921548434Z","logger":"knative-operator","caller":"logging/warning_handler.go:32","message":"API Warning: autopilot-default-resources-mutator:Autopilot updated Deployment knative-serving/autoscaler-hpa: adjusted 'cpu' resource to meet requirements for containers [autoscaler-hpa] (see http://g.co/gke/autopilot-defaults).","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:42.054004595Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:145","message":"Merging","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","diff":"{\"metadata\":{\"annotations\":null,\"creationTimestamp\":null},\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"migrate\"}],\"containers\":[{\"name\":\"migrate\",\"resources\":{\"requests\":{\"memory\":\"100Mi\"}}}]}}}}"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:42.054448825Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:208","message":"Updating","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","name":"knative-serving/storage-version-migration-serving-serving-1.17.1","type":"batch/v1, Kind=Job"}
{"severity":"ERROR","timestamp":"2025-07-21T22:02:42.160043935Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:165","message":"Failed to update merged resource, trying overwrite","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","error":"Job.batch \"storage-version-migration-serving-serving-1.17.1\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app\":\"storage-version-migration-serving\", \"app.kubernetes.io/component\":\"storage-version-migration-job\", \"app.kubernetes.io/name\":\"knative-serving\", \"app.kubernetes.io/version\":\"1.17.1\", \"batch.kubernetes.io/controller-uid\":\"90694af7-e4d9-4ddc-89fb-2f8f7ef031a1\", \"batch.kubernetes.io/job-name\":\"storage-version-migration-serving-serving-1.17.1\", \"controller-uid\":\"90694af7-e4d9-4ddc-89fb-2f8f7ef031a1\", \"job-name\":\"storage-version-migration-serving-serving-1.17.1\", \"sidecar.istio.io/inject\":\"false\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"migrate\", Image:\"gcr.io/knative-releases/knative.dev/pkg/apiextensions/storageversion/cmd/migrate@sha256:6ae0953fdc1d44b18299aad228cb3496fe435f496065f3142ea41013cd27e941\", Command:[]string(nil), Args:[]string{\"services.serving.knative.dev\", \"configurations.serving.knative.dev\", \"revisions.serving.knative.dev\", \"routes.serving.knative.dev\", \"domainmappings.serving.knative.dev\"}, WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{\"cpu\":resource.Quantity{i:resource.int64Amount{value:1, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1\", Format:\"DecimalSI\"}, \"ephemeral-storage\":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1Gi\", Format:\"BinarySI\"}, \"memory\":resource.Quantity{i:resource.int64Amount{value:1048576000, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"\", Format:\"BinarySI\"}}, Requests:core.ResourceList{\"cpu\":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"100m\", Format:\"DecimalSI\"}, \"ephemeral-storage\":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1Gi\", Format:\"BinarySI\"}, \"memory\":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"100Mi\", Format:\"BinarySI\"}}, Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc034eb3200), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"OnFailure\", TerminationGracePeriodSeconds:(*int64)(0xc0319d8160), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"controller\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc03883a1e0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration{core.Toleration{Key:\"kubernetes.io/arch\", Operator:\"Equal\", Value:\"amd64\", Effect:\"NoSchedule\", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil), Resources:(*core.ResourceRequirements)(nil)}}: field is immutable","stacktrace":"github.com/manifestival/manifestival.Manifest.update\n\tgithub.com/manifestival/manifestival@v0.7.2/manifestival.go:165\ngithub.com/manifestival/manifestival.Manifest.apply\n\tgithub.com/manifestival/manifestival@v0.7.2/manifestival.go:155\ngithub.com/manifestival/manifestival.Manifest.Apply\n\tgithub.com/manifestival/manifestival@v0.7.2/manifestival.go:100\nknative.dev/operator/pkg/reconciler/common.Install\n\tknative.dev/operator/pkg/reconciler/common/install.go:58\nknative.dev/operator/pkg/reconciler/manifests.Install\n\tknative.dev/operator/pkg/reconciler/manifests/install.go:33\nknative.dev/operator/pkg/reconciler/common.Stages.Execute\n\tknative.dev/operator/pkg/reconciler/common/stages.go:40\nknative.dev/operator/pkg/reconciler/knativeserving.(*Reconciler).ReconcileKind\n\tknative.dev/operator/pkg/reconciler/knativeserving/knativeserving.go:134\nknative.dev/operator/pkg/client/injection/reconciler/operator/v1beta1/knativeserving.(*reconcilerImpl).Reconcile\n\tknative.dev/operator/pkg/client/injection/reconciler/operator/v1beta1/knativeserving/reconciler.go:236\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:489"}
{"severity":"ERROR","timestamp":"2025-07-21T22:02:42.302815665Z","logger":"knative-operator","caller":"knativeserving/reconciler.go:295","message":"Returned an error","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing","knative.dev/traceid":"b2802548-0828-4bc9-88e1-95c97ca2db02","knative.dev/key":"knative-serving/knative-serving","targetMethod":"ReconcileKind","error":"failed to apply non rbac manifest: Job.batch \"storage-version-migration-serving-serving-1.17.1\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app\":\"storage-version-migration-serving\", \"app.kubernetes.io/component\":\"storage-version-migration-job\", \"app.kubernetes.io/name\":\"knative-serving\", \"app.kubernetes.io/version\":\"1.17.1\", \"batch.kubernetes.io/controller-uid\":\"90694af7-e4d9-4ddc-89fb-2f8f7ef031a1\", \"batch.kubernetes.io/job-name\":\"storage-version-migration-serving-serving-1.17.1\", \"controller-uid\":\"90694af7-e4d9-4ddc-89fb-2f8f7ef031a1\", \"job-name\":\"storage-version-migration-serving-serving-1.17.1\", \"sidecar.istio.io/inject\":\"false\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"migrate\", Image:\"gcr.io/knative-releases/knative.dev/pkg/apiextensions/storageversion/cmd/migrate@sha256:6ae0953fdc1d44b18299aad228cb3496fe435f496065f3142ea41013cd27e941\", Command:[]string(nil), Args:[]string{\"services.serving.knative.dev\", \"configurations.serving.knative.dev\", \"revisions.serving.knative.dev\", \"routes.serving.knative.dev\", \"domainmappings.serving.knative.dev\"}, WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{\"cpu\":resource.Quantity{i:resource.int64Amount{value:1, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1\", Format:\"DecimalSI\"}, \"ephemeral-storage\":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1Gi\", Format:\"BinarySI\"}, \"memory\":resource.Quantity{i:resource.int64Amount{value:1048576000, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"\", Format:\"BinarySI\"}}, Requests:core.ResourceList{\"cpu\":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"100m\", Format:\"DecimalSI\"}, \"ephemeral-storage\":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1Gi\", Format:\"BinarySI\"}, \"memory\":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"100Mi\", Format:\"BinarySI\"}}, Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc00ea476e0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"OnFailure\", TerminationGracePeriodSeconds:(*int64)(0xc034625d00), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"controller\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc01543dae0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration{core.Toleration{Key:\"kubernetes.io/arch\", Operator:\"Equal\", Value:\"amd64\", Effect:\"NoSchedule\", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil), Resources:(*core.ResourceRequirements)(nil)}}: field is immutable","stacktrace":"knative.dev/operator/pkg/client/injection/reconciler/operator/v1beta1/knativeserving.(*reconcilerImpl).Reconcile\n\tknative.dev/operator/pkg/client/injection/reconciler/operator/v1beta1/knativeserving/reconciler.go:295\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:489"}
{"severity":"ERROR","timestamp":"2025-07-21T22:02:42.302905445Z","logger":"knative-operator","caller":"controller/controller.go:564","message":"Reconcile error","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing","knative.dev/traceid":"b2802548-0828-4bc9-88e1-95c97ca2db02","knative.dev/key":"knative-serving/knative-serving","duration":"9.831334252s","error":"failed to apply non rbac manifest: Job.batch \"storage-version-migration-serving-serving-1.17.1\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app\":\"storage-version-migration-serving\", \"app.kubernetes.io/component\":\"storage-version-migration-job\", \"app.kubernetes.io/name\":\"knative-serving\", \"app.kubernetes.io/version\":\"1.17.1\", \"batch.kubernetes.io/controller-uid\":\"90694af7-e4d9-4ddc-89fb-2f8f7ef031a1\", \"batch.kubernetes.io/job-name\":\"storage-version-migration-serving-serving-1.17.1\", \"controller-uid\":\"90694af7-e4d9-4ddc-89fb-2f8f7ef031a1\", \"job-name\":\"storage-version-migration-serving-serving-1.17.1\", \"sidecar.istio.io/inject\":\"false\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"migrate\", Image:\"gcr.io/knative-releases/knative.dev/pkg/apiextensions/storageversion/cmd/migrate@sha256:6ae0953fdc1d44b18299aad228cb3496fe435f496065f3142ea41013cd27e941\", Command:[]string(nil), Args:[]string{\"services.serving.knative.dev\", \"configurations.serving.knative.dev\", \"revisions.serving.knative.dev\", \"routes.serving.knative.dev\", \"domainmappings.serving.knative.dev\"}, WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{\"cpu\":resource.Quantity{i:resource.int64Amount{value:1, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1\", Format:\"DecimalSI\"}, \"ephemeral-storage\":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1Gi\", Format:\"BinarySI\"}, \"memory\":resource.Quantity{i:resource.int64Amount{value:1048576000, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"\", Format:\"BinarySI\"}}, Requests:core.ResourceList{\"cpu\":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"100m\", Format:\"DecimalSI\"}, \"ephemeral-storage\":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1Gi\", Format:\"BinarySI\"}, \"memory\":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"100Mi\", Format:\"BinarySI\"}}, Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc00ea476e0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"OnFailure\", TerminationGracePeriodSeconds:(*int64)(0xc034625d00), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"controller\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc01543dae0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration{core.Toleration{Key:\"kubernetes.io/arch\", Operator:\"Equal\", Value:\"amd64\", Effect:\"NoSchedule\", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil), Resources:(*core.ResourceRequirements)(nil)}}: field is immutable","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:564\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:541\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:489"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:42.303301135Z","logger":"knative-operator.event-broadcaster","caller":"record/event.go:377","message":"Event(v1.ObjectReference{Kind:\"KnativeServing\", Namespace:\"knative-serving\", Name:\"knative-serving\", UID:\"66adf2d9-7801-4522-a8f4-69eb8a5296e8\", APIVersion:\"operator.knative.dev/v1beta1\", ResourceVersion:\"1753135352452527014\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' failed to apply non rbac manifest: Job.batch \"storage-version-migration-serving-serving-1.17.1\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app\":\"storage-version-migration-serving\", \"app.kubernetes.io/component\":\"storage-version-migration-job\", \"app.kubernetes.io/name\":\"knative-serving\", \"app.kubernetes.io/version\":\"1.17.1\", \"batch.kubernetes.io/controller-uid\":\"90694af7-e4d9-4ddc-89fb-2f8f7ef031a1\", \"batch.kubernetes.io/job-name\":\"storage-version-migration-serving-serving-1.17.1\", \"controller-uid\":\"90694af7-e4d9-4ddc-89fb-2f8f7ef031a1\", \"job-name\":\"storage-version-migration-serving-serving-1.17.1\", \"sidecar.istio.io/inject\":\"false\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"migrate\", Image:\"gcr.io/knative-releases/knative.dev/pkg/apiextensions/storageversion/cmd/migrate@sha256:6ae0953fdc1d44b18299aad228cb3496fe435f496065f3142ea41013cd27e941\", Command:[]string(nil), Args:[]string{\"services.serving.knative.dev\", \"configurations.serving.knative.dev\", \"revisions.serving.knative.dev\", \"routes.serving.knative.dev\", \"domainmappings.serving.knative.dev\"}, WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{\"cpu\":resource.Quantity{i:resource.int64Amount{value:1, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1\", Format:\"DecimalSI\"}, \"ephemeral-storage\":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1Gi\", Format:\"BinarySI\"}, \"memory\":resource.Quantity{i:resource.int64Amount{value:1048576000, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"\", Format:\"BinarySI\"}}, Requests:core.ResourceList{\"cpu\":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"100m\", Format:\"DecimalSI\"}, \"ephemeral-storage\":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1Gi\", Format:\"BinarySI\"}, \"memory\":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"100Mi\", Format:\"BinarySI\"}}, Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc00ea476e0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"OnFailure\", TerminationGracePeriodSeconds:(*int64)(0xc034625d00), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"controller\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc01543dae0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration{core.Toleration{Key:\"kubernetes.io/arch\", Operator:\"Equal\", Value:\"amd64\", Effect:\"NoSchedule\", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil), Resources:(*core.ResourceRequirements)(nil)}}: field is immutable","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:42.308363685Z","logger":"knative-operator","caller":"knativeserving/knativeserving.go:107","message":"Reconciling KnativeServing","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing","knative.dev/traceid":"69d77884-f862-46a8-80fc-7b44afd8080a","knative.dev/key":"knative-serving/knative-serving","status":{"observedGeneration":1,"conditions":[{"type":"DependenciesInstalled","status":"Unknown","lastTransitionTime":"2025-07-21T21:32:39Z"},{"type":"DeploymentsAvailable","status":"False","lastTransitionTime":"2025-07-21T21:33:00Z","reason":"NotReady","message":"Waiting on deployments: activator, autoscaler, controller, webhook, autoscaler-hpa, net-istio-controller, net-istio-webhook"},{"type":"InstallSucceeded","status":"False","lastTransitionTime":"2025-07-21T22:02:42Z","reason":"Error","message":"Install failed with message: Job.batch \"storage-version-migration-serving-serving-1.17.1\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app\":\"storage-version-migration-serving\", \"app.kubernetes.io/component\":\"storage-version-migration-job\", \"app.kubernetes.io/name\":\"knative-serving\", \"app.kubernetes.io/version\":\"1.17.1\", \"batch.kubernetes.io/controller-uid\":\"90694af7-e4d9-4ddc-89fb-2f8f7ef031a1\", \"batch.kubernetes.io/job-name\":\"storage-version-migration-serving-serving-1.17.1\", \"controller-uid\":\"90694af7-e4d9-4ddc-89fb-2f8f7ef031a1\", \"job-name\":\"storage-version-migration-serving-serving-1.17.1\", \"sidecar.istio.io/inject\":\"false\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"migrate\", Image:\"gcr.io/knative-releases/knative.dev/pkg/apiextensions/storageversion/cmd/migrate@sha256:6ae0953fdc1d44b18299aad228cb3496fe435f496065f3142ea41013cd27e941\", Command:[]string(nil), Args:[]string{\"services.serving.knative.dev\", \"configurations.serving.knative.dev\", \"revisions.serving.knative.dev\", \"routes.serving.knative.dev\", \"domainmappings.serving.knative.dev\"}, WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{\"cpu\":resource.Quantity{i:resource.int64Amount{value:1, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1\", Format:\"DecimalSI\"}, \"ephemeral-storage\":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1Gi\", Format:\"BinarySI\"}, \"memory\":resource.Quantity{i:resource.int64Amount{value:1048576000, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"\", Format:\"BinarySI\"}}, Requests:core.ResourceList{\"cpu\":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"100m\", Format:\"DecimalSI\"}, \"ephemeral-storage\":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1Gi\", Format:\"BinarySI\"}, \"memory\":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"100Mi\", Format:\"BinarySI\"}}, Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc00ea476e0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"OnFailure\", TerminationGracePeriodSeconds:(*int64)(0xc034625d00), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"controller\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc01543dae0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration{core.Toleration{Key:\"kubernetes.io/arch\", Operator:\"Equal\", Value:\"amd64\", Effect:\"NoSchedule\", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil), Resources:(*core.ResourceRequirements)(nil)}}: field is immutable"},{"type":"Ready","status":"False","lastTransitionTime":"2025-07-21T22:02:42Z","reason":"Error","message":"Install failed with message: Job.batch \"storage-version-migration-serving-serving-1.17.1\" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app\":\"storage-version-migration-serving\", \"app.kubernetes.io/component\":\"storage-version-migration-job\", \"app.kubernetes.io/name\":\"knative-serving\", \"app.kubernetes.io/version\":\"1.17.1\", \"batch.kubernetes.io/controller-uid\":\"90694af7-e4d9-4ddc-89fb-2f8f7ef031a1\", \"batch.kubernetes.io/job-name\":\"storage-version-migration-serving-serving-1.17.1\", \"controller-uid\":\"90694af7-e4d9-4ddc-89fb-2f8f7ef031a1\", \"job-name\":\"storage-version-migration-serving-serving-1.17.1\", \"sidecar.istio.io/inject\":\"false\"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"migrate\", Image:\"gcr.io/knative-releases/knative.dev/pkg/apiextensions/storageversion/cmd/migrate@sha256:6ae0953fdc1d44b18299aad228cb3496fe435f496065f3142ea41013cd27e941\", Command:[]string(nil), Args:[]string{\"services.serving.knative.dev\", \"configurations.serving.knative.dev\", \"revisions.serving.knative.dev\", \"routes.serving.knative.dev\", \"domainmappings.serving.knative.dev\"}, WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{\"cpu\":resource.Quantity{i:resource.int64Amount{value:1, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1\", Format:\"DecimalSI\"}, \"ephemeral-storage\":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1Gi\", Format:\"BinarySI\"}, \"memory\":resource.Quantity{i:resource.int64Amount{value:1048576000, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"\", Format:\"BinarySI\"}}, Requests:core.ResourceList{\"cpu\":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"100m\", Format:\"DecimalSI\"}, \"ephemeral-storage\":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"1Gi\", Format:\"BinarySI\"}, \"memory\":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"100Mi\", Format:\"BinarySI\"}}, Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc00ea476e0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"OnFailure\", TerminationGracePeriodSeconds:(*int64)(0xc034625d00), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"controller\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc01543dae0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration{core.Toleration{Key:\"kubernetes.io/arch\", Operator:\"Equal\", Value:\"amd64\", Effect:\"NoSchedule\", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil), Resources:(*core.ResourceRequirements)(nil)}}: field is immutable"},{"type":"VersionMigrationEligible","status":"True","lastTransitionTime":"2025-07-21T21:32:39Z"}]}}
{"severity":"INFO","timestamp":"2025-07-21T22:02:48.154112285Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:145","message":"Merging","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","diff":"{\"spec\":{\"selector\":{\"matchLabels\":{\"app\":\"activator\"}}}}"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:48.154431445Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:208","message":"Updating","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","name":"knative-serving/activator-pdb","type":"policy/v1, Kind=PodDisruptionBudget"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:48.352317255Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:145","message":"Merging","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","diff":"{\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"activator\"}],\"containers\":[{\"name\":\"activator\",\"resources\":{\"requests\":{\"memory\":\"60Mi\"}}}]}}}}"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:48.353241575Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:208","message":"Updating","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","name":"knative-serving/activator","type":"apps/v1, Kind=Deployment"}
{"severity":"INFO","timestamp":"2025-07-21T22:02:48.711690625Z","logger":"knative-operator.manifestival","caller":"manifestival@v0.7.2/manifestival.go:145","message":"Merging","commit":"aaf737c-dirty","knative.dev/pod":"knative-operator-7c6b8f9cc4-llj49","diff":"{\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"autoscaler\"}],\"containers\":[{\"name\":\"autoscaler\",\"resources\":{\"requests\":{\"memory\":\"100Mi\"}}}]}}}}"}

[phmcder]

I have an issue with tolerations for these jobs. Can't see a way to apply tolerations when all my nodes have taints applied