Skip to content

Add PVC and update resource limits in k8s config #8489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zhyncs merged 2 commits into from
Jul 29, 2025
Merged

Add PVC and update resource limits in k8s config #8489

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
58456a8
Add PVC and update resource limits in k8s config
haitwang-cloud Jul 29, 2025
75414a1
Update readinessProbe path to /health_generate
haitwang-cloud Jul 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 41 additions & 10 deletions docker/k8s-sglang-service.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,16 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: llama-31-8b-sglang
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 30Gi
storageClassName: default # change this to your preferred storage class
volumeMode: Filesystem
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
Expand Down Expand Up @@ -27,41 +40,59 @@ spec:
containers:
- name: meta-llama-31-8b-instruct-sglang
image: docker.io/lmsysorg/sglang:latest
imagePullPolicy: Always # IfNotPresent or Never
imagePullPolicy: Always # IfNotPresent or Never
ports:
- containerPort: 30000
command: ["python3", "-m", "sglang.launch_server"]
args: ["--model-path", "meta-llama/Llama-3.1-8B-Instruct", "--host", "0.0.0.0", "--port", "30000"]
args:
[
"--model-path",
"meta-llama/Llama-3.1-8B-Instruct",
"--host",
"0.0.0.0",
"--port",
"30000",
]
env:
- name: HF_TOKEN
value: <secret>
resources:
limits:
nvidia.com/gpu: 1
cpu: 8
memory: 40Gi
requests:
cpu: 2
memory: 16Gi
nvidia.com/gpu: 1
volumeMounts:
- name: shm
mountPath: /dev/shm
- name: hf-cache
mountPath: /root/.cache/huggingface
readOnly: true
- name: localtime
mountPath: /etc/localtime
readOnly: true
livenessProbe:
httpGet:
path: /health
port: 30000
initialDelaySeconds: 30
initialDelaySeconds: 120
periodSeconds: 10
readinessProbe:
httpGet:
path: /health
Copy link
Copy Markdown
Collaborator

@slin1237 slin1237 Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be health_generate

Copy link
Copy Markdown
Contributor Author

@haitwang-cloud haitwang-cloud Jul 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@slin1237 Very good point, I dig more details about the /health & /health_generate and find that

  • /health : was designed to check the health of the http server and
  • /health_generate : was designed to check the health of the inference server by generating one token.

So I update the PR with

  • Liveness Probe -> /health
  • Readiness Probe -> /health_generate

And that the logs for the above change

[2025-07-29 05:28:55] The server is fired up and ready to roll!
[2025-07-29 05:30:18] INFO:     127.0.0.6:42359 - "GET /health HTTP/1.1" 200 OK
[2025-07-29 05:30:18] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-07-29 05:30:28] INFO:     127.0.0.6:58251 - "GET /health HTTP/1.1" 200 OK
[2025-07-29 05:30:28] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-07-29 05:30:33] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-07-29 05:30:38] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-07-29 05:30:38] INFO:     127.0.0.6:33001 - "GET /health HTTP/1.1" 200 OK

port: 30000
initialDelaySeconds: 120
periodSeconds: 10
volumes:
- name: shm
emptyDir:
medium: Memory
sizeLimit: 10Gi
- name: hf-cache
hostPath:
path: /root/.cache/huggingface
type: Directory
persistentVolumeClaim:
claimName: llama-31-8b-sglang
- name: localtime
hostPath:
path: /etc/localtime
Expand All @@ -76,6 +107,6 @@ spec:
app: meta-llama-31-8b-instruct-sglang
ports:
- protocol: TCP
port: 30000 # port on host
targetPort: 30000 # port in container
type: LoadBalancer
port: 80 # port on host
targetPort: 30000 # port in container
type: LoadBalancer # change to ClusterIP if needed