Skip to content

Commit 3872780

Browse files
dharaneeshvrddharaneeshvrd
committed
Add Install Gateway section in Getting Started guide
- Move instructions from the Deploy an Inference Gateway section describing installation of Gateway API CRDs and provider specific GWs Signed-off-by: Dharaneeshwaran Ravichandran <[email protected]>
1 parent f95cf36 commit 3872780

1 file changed

Lines changed: 83 additions & 70 deletions

File tree

‎site-src/guides/index.md‎

Lines changed: 83 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,76 @@
3737
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.0.1/v1-manifests.yaml
3838
```
3939

40+
### Install the Gateway
41+
42+
Choose one of the following options to install Gateway.
43+
44+
=== "GKE"
45+
46+
Nothing to install here, you can move to the next [section](#deploy-the-inferencepool-and-endpoint-picker-extension)
47+
48+
=== "Istio"
49+
50+
1. Requirements
51+
- Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
52+
53+
2. Install Istio
54+
55+
```
56+
TAG=$(curl https://storage.googleapis.com/istio-build/dev/1.28-dev)
57+
# on Linux
58+
wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-linux-amd64.tar.gz
59+
tar -xvf istioctl-$TAG-linux-amd64.tar.gz
60+
# on macOS
61+
wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-osx.tar.gz
62+
tar -xvf istioctl-$TAG-osx.tar.gz
63+
# on Windows
64+
wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-win.zip
65+
unzip istioctl-$TAG-win.zip
66+
67+
./istioctl install --set tag=$TAG --set hub=gcr.io/istio-testing --set values.pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true
68+
```
69+
70+
=== "Kgateway"
71+
72+
1. Requirements
73+
74+
- Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
75+
- [Helm](https://helm.sh/docs/intro/install/) installed.
76+
77+
2. Set the Kgateway version and install the Kgateway CRDs.
78+
79+
```bash
80+
KGTW_VERSION=v2.1.0-main
81+
helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
82+
```
83+
84+
3. Install Kgateway
85+
86+
```bash
87+
helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=true
88+
```
89+
90+
=== "Agentgateway"
91+
92+
1. Requirements
93+
94+
- Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
95+
- [Helm](https://helm.sh/docs/intro/install/) installed.
96+
97+
2. Set the Kgateway version and install the Kgateway CRDs.
98+
99+
```bash
100+
KGTW_VERSION=v2.1.0-main
101+
helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
102+
```
103+
104+
3. Install Kgateway
105+
106+
```bash
107+
helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=true --set agentGateway.enabled=true
108+
```
109+
40110
### Deploy the InferencePool and Endpoint Picker Extension
41111

42112
Install an InferencePool named `vllm-llama3-8b-instruct` that selects from endpoints with label `app: vllm-llama3-8b-instruct` and listening on port 8000. The Helm install command automatically installs the endpoint-picker, InferencePool along with provider specific resources.
@@ -59,7 +129,7 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
59129
See [Deploy Inference Gateways](https://cloud.google.com/kubernetes-engine/docs/how-to/deploy-gke-inference-gateway)
60130
for detailed instructions.
61131

62-
2. Deploy Inference Gateway:
132+
2. Deploy the Inference Gateway:
63133

64134
```bash
65135
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/v1.0.1/config/manifests/gateway/gke/gateway.yaml
@@ -89,34 +159,13 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
89159
Please note that this feature is currently in an experimental phase and is not intended for production use.
90160
The implementation and user experience are subject to changes as we continue to iterate on this project.
91161

92-
1. Requirements
93-
94-
- Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
95-
96-
2. Install Istio
97-
98-
```
99-
TAG=$(curl https://storage.googleapis.com/istio-build/dev/1.28-dev)
100-
# on Linux
101-
wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-linux-amd64.tar.gz
102-
tar -xvf istioctl-$TAG-linux-amd64.tar.gz
103-
# on macOS
104-
wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-osx.tar.gz
105-
tar -xvf istioctl-$TAG-osx.tar.gz
106-
# on Windows
107-
wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-win.zip
108-
unzip istioctl-$TAG-win.zip
109-
110-
./istioctl install --set tag=$TAG --set hub=gcr.io/istio-testing --set values.pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true
111-
```
112-
113-
3. If your EPP uses secure serving with self-signed certs (default), temporarily bypass TLS verification:
162+
1. If your EPP uses secure serving with self-signed certs (default), temporarily bypass TLS verification:
114163

115164
```bash
116165
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/destination-rule.yaml
117166
```
118167

119-
4. Deploy Gateway
168+
2. Deploy the Inference Gateway
120169

121170
```bash
122171
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/gateway.yaml
@@ -129,13 +178,13 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
129178
inference-gateway inference-gateway <MY_ADDRESS> True 22s
130179
```
131180

132-
5. Deploy the HTTPRoute
181+
3. Deploy the HTTPRoute
133182

134183
```bash
135184
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/httproute.yaml
136185
```
137186

138-
6. Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
187+
4. Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
139188

140189
```bash
141190
kubectl get httproute llm-route -o yaml
@@ -144,27 +193,9 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
144193
=== "Kgateway"
145194

146195
[Kgateway](https://kgateway.dev/) added Inference Gateway support as a **technical preview** in the
147-
[v2.0.0 release](https://github.com/kgateway-dev/kgateway/releases/tag/v2.0.0). InferencePool v1.0.1 is currently supported in the latest [rolling release](https://github.com/kgateway-dev/kgateway/releases/tag/v2.1.0-main), which includes the latest changes but may be unstable until the [v2.1.0 release](https://github.com/kgateway-dev/kgateway/milestone/58) is published.
148-
149-
1. Requirements
150-
151-
- [Helm](https://helm.sh/docs/intro/install/) installed.
152-
- Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
196+
[v2.0.0 release](https://github.com/kgateway-dev/kgateway/releases/tag/v2.0.0). InferencePool v1.0.0 is currently supported in the latest [rolling release](https://github.com/kgateway-dev/kgateway/releases/tag/v2.1.0-main), which includes the latest changes but may be unstable until the [v2.1.0 release](https://github.com/kgateway-dev/kgateway/milestone/58) is published.
153197

154-
2. Set the Kgateway version and install the Kgateway CRDs.
155-
156-
```bash
157-
KGTW_VERSION=v2.1.0-main
158-
helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
159-
```
160-
161-
3. Install Kgateway
162-
163-
```bash
164-
helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=true
165-
```
166-
167-
4. Deploy the Gateway
198+
1. Deploy the Inference Gateway
168199

169200
```bash
170201
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/v1.0.1/config/manifests/gateway/kgateway/gateway.yaml
@@ -177,41 +208,23 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
177208
inference-gateway kgateway <MY_ADDRESS> True 22s
178209
```
179210

180-
5. Deploy the HTTPRoute
211+
2. Deploy the HTTPRoute
181212

182213
```bash
183214
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/v1.0.1/config/manifests/gateway/kgateway/httproute.yaml
184215
```
185216

186-
6. Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
217+
3. Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
187218

188219
```bash
189220
kubectl get httproute llm-route -o yaml
190221
```
191222

192223
=== "Agentgateway"
193224

194-
[Agentgateway](https://agentgateway.dev/) is a purpose-built proxy designed for AI workloads, and comes with native support for Inference Gateway. Agentgateway integrates with [Kgateway](https://kgateway.dev/) as it's control plane. InferencePool v1.0.1 is currently supported in the latest [rolling release](https://github.com/kgateway-dev/kgateway/releases/tag/v2.1.0-main), which includes the latest changes but may be unstable until the [v2.1.0 release](https://github.com/kgateway-dev/kgateway/milestone/58) is published.
195-
196-
1. Requirements
197-
198-
- [Helm](https://helm.sh/docs/intro/install/) installed.
199-
- Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
200-
201-
2. Set the Kgateway version and install the Kgateway CRDs.
202-
203-
```bash
204-
KGTW_VERSION=v2.1.0-main
205-
helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
206-
```
207-
208-
3. Install Kgateway
209-
210-
```bash
211-
helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=true --set agentGateway.enabled=true
212-
```
225+
[Agentgateway](https://agentgateway.dev/) is a purpose-built proxy designed for AI workloads, and comes with native support for Inference Gateway. Agentgateway integrates with [Kgateway](https://kgateway.dev/) as it's control plane. InferencePool v1.0.0 is currently supported in the latest [rolling release](https://github.com/kgateway-dev/kgateway/releases/tag/v2.1.0-main), which includes the latest changes but may be unstable until the [v2.1.0 release](https://github.com/kgateway-dev/kgateway/milestone/58) is published.
213226

214-
4. Deploy the Gateway
227+
1. Deploy the Inference Gateway
215228

216229
```bash
217230
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/v1.0.1/config/manifests/gateway/agentgateway/gateway.yaml
@@ -224,13 +237,13 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
224237
inference-gateway agentgateway <MY_ADDRESS> True 22s
225238
```
226239

227-
5. Deploy the HTTPRoute
240+
2. Deploy the HTTPRoute
228241

229242
```bash
230243
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/v1.0.1/config/manifests/gateway/agentgateway/httproute.yaml
231244
```
232245

233-
6. Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
246+
3. Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
234247

235248
```bash
236249
kubectl get httproute llm-route -o yaml

0 commit comments

Comments
 (0)