InferencePool: Finalize ExtensionRef Type

[danehans]

#1324 simplified the InferencePool EPP config by removing the type inlining. Since ExtensionRef is required, it should not be a pointer or include the omitempty tag:

	// Extension configures an endpoint picker as an extension service.
	//
	// +kubebuilder:validation:Required
	ExtensionRef *Extension `json:"extensionRef,omitempty"`

When the API was first created, we discussed the idea of supporting the EPP extension as a sidecar container, e.g. UDS socket. If we still plan to support this use case, we should determine the configuration details. Here are a few initial thoughts:

1. We can keep ExtensionRef optional, which would indicate using a UDS socket for the Gateway<>EPP connection. However, the socket path, e.g., /tmp/uds/my_socket.sock, between the two would not be configurable, and FailureMode would need to reside elsewhere.
2. We make ExtensionRef required and make ExtensionRef.Name optional, which a "" value would indicate no Service reference and instead use a UDS socket for the Gateway<>EPP connection.
3. Add a union type to ExtensionRef to indicate the connection/reference type and defaults to a GKN reference.

If we do not intend to support the sidecar use case, then update ExtensionRef as follows:

	// ExtensionRef is a reference to the Endpoint Picker (EPP) Extension.
	//
	// +kubebuilder:validation:Required
	ExtensionRef Extension `json:"extensionRef"`

Note that ExtensionRef implies the only extension supported by an InferencePool is a single EPP extension.

cc: @robscott @smarterclayton @ahg-g @nirrozenbaum @kfswain

[capri-xiyue]

/assign

[robscott]

@danehans would a UDS socket be helpful for Kgateway? Of course we don't really have a way to support it on GKE, but I can see the usefulness. In that case, is that really something you'd expect the user to configure, or would it be more of a "batteries included" mode for the Gateway where it came bundled with an EPP? I'd kinda assumed that any config at that level would be implementation specific, but that we should leave ExtensionRef optional to allow implementations (likely in-cluster ones) to bundle an EPP with their Gateway deployment.

[danehans]

@danehans would a UDS socket be helpful for Kgateway?

@robscott not now, but potentially in the future. Kgateway uses an ext-proc sidecar for the AI extension, e.g., AI Gateway, but not for Inference Extension.

would it be more of a "batteries included" mode for the Gateway where it came bundled with an EPP?

Yes, I can see the potential of bundling the EPP with the gateway deployment, if the gateway includes an HTTPRoute that refs an InferencePool.

but that we should leave ExtensionRef optional to allow implementations (likely in-cluster ones) to bundle an EPP with their Gateway deployment.

Right, this would be option 1 above.

[capri-xiyue]

I sent #1365 to make extensionRef optional.

I accidentally made it required in #1324

[danehans]

Closing due to #1324 and #1365 fixing this issue. Thanks @capri-xiyue!!!