GCPManagedControlPlane does not come up and requires "Subnet"

[bubbaksmith]

/kind bug

What steps did you take and what happened:

I created a quickstart GKE cluster and modified the GCPManagedCluster.spec.network to include explicit subnets and not create default subnets. Ultimately no cluster comes online and the GCPManagedControlPlane throws errors about needing a subnet defined for VPCs in custom subnet mode.

Full Cluster Spec

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: vanilla
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  controlPlaneRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: GCPManagedControlPlane
    name: vanilla-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: GCPManagedCluster
    name: vanilla
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: GCPManagedCluster
metadata:
  name: vanilla
  namespace: default
spec:
  network:
    autoCreateSubnetworks: false
    name: vanilla-vpc
    subnets:
    - cidrBlock: 10.20.96.0/20
      name: nodes-primary
      region: us-central1
    - cidrBlock: 10.21.96.0/20
      name: secondary
      region: us-central1
  project: internal-development-123456
  region: us-central1
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: GCPManagedControlPlane
metadata:
  name: vanilla-control-plane
  namespace: default
spec:
  location: us-central1
  project: internal-development-123456
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachinePool
metadata:
  name: vanilla-mp-0
  namespace: default
spec:
  clusterName: vanilla
  replicas: 3
  template:
    spec:
      bootstrap:
        dataSecretName: ""
      clusterName: vanilla
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: GCPManagedMachinePool
        name: vanilla-mp-0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: GCPManagedMachinePool
metadata:
  name: vanilla-mp-0
  namespace: default
spec: {}

Steps to recreate local setup

kind create cluster -n local

export EXP_CAPG_GKE=true
export EXP_MACHINE_POOL=true
clusterctl init --infrastructure gcp

GCP_B64ENCODED_CREDENTIALS=$(cat /Users/bubba/dev/capi/service-account.json | base64 | tr -d '\n') \
  GCP_REGION=us-central1 \
  GCP_PROJECT=internal-development-123456 \
  GCP_NETWORK_NAME=vanilla-vpc \
  clusterctl generate cluster vanilla --flavor gke > gke-vanilla.yaml
  
# Edit GCPManagedCluster.spec.network 
vim gke-vanilla.yaml

$ kubectl apply -f gke-vanilla.yaml
cluster.cluster.x-k8s.io/vanilla created
gcpmanagedcluster.infrastructure.cluster.x-k8s.io/vanilla created
gcpmanagedcontrolplane.infrastructure.cluster.x-k8s.io/vanilla-control-plane created
machinepool.cluster.x-k8s.io/vanilla-mp-0 created
gcpmanagedmachinepool.infrastructure.cluster.x-k8s.io/vanilla-mp-0 created

$ kubectl get GCPManagedControlPlane vanilla-control-plane -o yaml
...
status:
  conditions:
  - lastTransitionTime: "2023-04-13T18:51:26Z"
    message: 'rpc error: code = InvalidArgument desc = Network "vanilla-vpc" uses
      manual subnet mode and requires specifying a subnetwork.'
    reason: GKEControlPlaneReconciliationFailed
    severity: Error
    status: "False"
    type: Ready
  - lastTransitionTime: "2023-04-13T18:51:26Z"
    message: 'rpc error: code = InvalidArgument desc = Network "vanilla-vpc" uses
      manual subnet mode and requires specifying a subnetwork.'
    reason: GKEControlPlaneReconciliationFailed
    severity: Error
    status: "False"
    type: GKEControlPlaneCreating
  - lastTransitionTime: "2023-04-13T18:51:26Z"
    message: 'rpc error: code = InvalidArgument desc = Network "vanilla-vpc" uses
      manual subnet mode and requires specifying a subnetwork.'
    reason: GKEControlPlaneReconciliationFailed
    severity: Error
    status: "False"
    type: GKEControlPlaneReady
  ready: false

These are the errors found in capg-controller

I0413 20:25:56.414928 1 reconcile.go:51] "msg"="Cluster not found, creating" "GCPManagedControlPlane"={"name":"vanilla-control-plane","namespace":"default"} "controller"="gcpmanagedcontrolplane" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="GCPManagedControlPlane" "name"="vanilla-control-plane" "namespace"="default" "reconcileID"="97315a96-9174-42e1-8d24-1f2847341b02" "service"="container.clusters"
E0413 20:25:56.951325 1 reconcile.go:268] "msg"="Error creating GKE cluster" "error"="rpc error: code = InvalidArgument desc = Network "vanilla-vpc" uses manual subnet mode and requires specifying a subnetwork." "GCPManagedControlPlane"={"name":"vanilla-control-plane","namespace":"default"} "controller"="gcpmanagedcontrolplane" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="GCPManagedControlPlane" "name"="default-vanilla-control-plane" "namespace"="default" "reconcileID"="97315a96-9174-42e1-8d24-1f2847341b02" "service"="container.clusters"
E0413 20:25:56.951419 1 reconcile.go:80] "msg"="failed creating cluster" "error"="rpc error: code = InvalidArgument desc = Network "vanilla-vpc" uses manual subnet mode and requires specifying a subnetwork." "GCPManagedControlPlane"={"name":"vanilla-control-plane","namespace":"default"} "controller"="gcpmanagedcontrolplane" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="GCPManagedControlPlane" "name"="vanilla-control-plane" "namespace"="default" "reconcileID"="97315a96-9174-42e1-8d24-1f2847341b02" "service"="container.clusters"
E0413 20:25:56.951489 1 gcpmanagedcontrolplane_controller.go:176] "msg"="Reconcile error" "error"="rpc error: code = InvalidArgument desc = Network "vanilla-vpc" uses manual subnet mode and requires specifying a subnetwork." "GCPManagedControlPlane"={"name":"vanilla-control-plane","namespace":"default"} "controller"="gcpmanagedcontrolplane" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="GCPManagedControlPlane" "name"="vanilla-control-plane" "namespace"="default" "reconcileID"="97315a96-9174-42e1-8d24-1f2847341b02" "reconciler"="container_clusters"
E0413 20:25:56.952598 1 controller.go:326] "msg"="Reconciler error" "error"="rpc error: code = InvalidArgument desc = Network "vanilla-vpc" uses manual subnet mode and requires specifying a subnetwork." "GCPManagedControlPlane"={"name":"vanilla-control-plane","namespace":"default"} "controller"="gcpmanagedcontrolplane" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="GCPManagedControlPlane" "name"="vanilla-control-plane" "namespace"="default" "reconcileID"="97315a96-9174-42e1-8d24-1f2847341b02"

What did you expect to happen:

This part is kind of fuzzy, because this feature is new and I didn't really know what to expect. Ultimately I think the issue is that I can define subnets that will get created in my VPC, but not set a subnet as the Primary for my nodes. Therefore the GCPManagedControlPlane does not come online with a healthy status.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api version: v1.4.1
• CAPG version: v1.3.0
• Minikube/KIND version: kind v0.16.0 go1.19.1 darwin/arm64
• Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:25:45Z", GoVersion:"go1.19.1", Compiler:"gc", - Platform:"darwin/arm64"}
• OS (e.g. from /etc/os-release): MacOS 13.2.1 (22D68) - Darwin Kernel Version 22.3.0

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[abhijitchaudhari1]

/reopen

[k8s-ci-robot]

@abhijitchaudhari1: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.