This repository was archived by the owner on Sep 5, 2019. It is now read-only.
This repository was archived by the owner on Sep 5, 2019. It is now read-only.
very slow concurrent builds #613
Description
/kind question
We are trying to do a lot of concurrent builds. What we are seeing is that pod remain pending for several minutes then the builds start to complete very slowly, despite the cluster having enough resources.
No errors appear in the controller or the pods, nowhere. So everything seems fine.
But we would expect the builds to start right away and complete quickly.
A basic test with the following build.yaml:
apiVersion: build.knative.dev/v1alpha1
kind: Build
metadata:
name: REPLACE
spec:
serviceAccountName: default
source:
git:
revision: master
url: https://github.com/mchmarny/simple-app
template:
arguments:
- name: IMAGE
value: knative.registry.svc.cluster.local/tzununbekov/foo-0-bar
kind: BuildTemplate
name: kaniko
Using the kaniko template:
kubectl apply -f https://raw.githubusercontent.com/triggermesh/build-templates/master/kaniko/kaniko.yaml
And running 50 concurrent builds with:
for i in {0..50}; do sed -e "s/REPLACE/build-${i}-foo/g" /tmp/build.yaml | kubectl apply -f - & done
A few builds complete ~10, but ultimately ~40 of them timeout after 10 minutes.
$ kubectl -n sebgoa get pods
NAME READY STATUS RESTARTS AGE
build-0-foo-pod-1f289a 0/1 Completed 0 11m
build-1-foo-pod-1eca70 0/1 Completed 0 10m
build-10-foo-pod-7cf35d 0/1 Terminating 0 10m
build-11-foo-pod-2c53de 0/1 Completed 0 10m
build-12-foo-pod-21e92a 0/1 Completed 0 10m
build-14-foo-pod-dc76e7 0/1 Terminating 0 10m
build-15-foo-pod-23ffbd 0/1 Terminating 0 10m
build-16-foo-pod-b36814 0/1 Completed 0 10m
build-18-foo-pod-75dd8b 0/1 Completed 0 10m
build-19-foo-pod-7a14f8 0/1 Terminating 0 10m
build-2-foo-pod-c72e9a 0/1 Completed 0 10m
build-20-foo-pod-e2d9da 0/1 Completed 0 10m
build-21-foo-pod-afc242 0/1 Completed 0 10m
build-22-foo-pod-19601e 0/1 Completed 0 10m
build-23-foo-pod-8b614e 0/1 Completed 0 10m
build-24-foo-pod-7a2d4e 0/1 Terminating 0 10m
build-25-foo-pod-dccd5b 0/1 Terminating 0 10m
build-26-foo-pod-8e2d91 0/1 Terminating 0 10m
build-28-foo-pod-7fa41a 0/1 Terminating 0 10m
build-29-foo-pod-8e796e 0/1 Init:2/3 0 9m56s
build-3-foo-pod-6c5fe4 0/1 Terminating 0 10m
build-30-foo-pod-3e4a0b 0/1 Terminating 0 10m
build-32-foo-pod-b1fe60 0/1 Completed 0 10m
build-33-foo-pod-477816 0/1 Terminating 0 10m
build-34-foo-pod-caa536 0/1 Terminating 0 10m
build-35-foo-pod-a7209e 0/1 Terminating 0 10m
build-36-foo-pod-cf09c4 0/1 Terminating 0 10m
build-37-foo-pod-d23ead 0/1 Completed 0 10m
build-38-foo-pod-970900 0/1 Init:2/3 0 9m59s
build-39-foo-pod-c45c77 0/1 Init:2/3 0 9m51s
build-4-foo-pod-e76e78 0/1 Completed 0 10m
build-40-foo-pod-dc112a 0/1 Terminating 0 10m
build-41-foo-pod-bddec0 0/1 Terminating 0 10m
build-42-foo-pod-335cc4 0/1 Terminating 0 10m
build-43-foo-pod-5c6cf9 0/1 Completed 0 10m
build-44-foo-pod-e31788 0/1 Terminating 0 10m
build-45-foo-pod-5b5d67 0/1 Init:2/3 0 9m55s
build-46-foo-pod-7a3e4b 0/1 Terminating 0 10m
build-47-foo-pod-043e93 0/1 Init:2/3 0 9m58s
build-48-foo-pod-ff557a 0/1 Completed 0 9m52s
build-49-foo-pod-1a436b 0/1 Terminating 0 10m
build-5-foo-pod-4ba973 0/1 Completed 0 10m
build-50-foo-pod-683a52 0/1 Init:2/3 0 9m53s
build-6-foo-pod-da90e6 0/1 Completed 0 10m
build-7-foo-pod-fdb93b 0/1 Completed 0 10m
build-8-foo-pod-0b917b 0/1 Completed 0 10m
build-9-foo-pod-ff9a07 0/1 Terminating 0 10m
Is the github cloning possible being throttled which slows down the Build steps ?