-
-
Notifications
You must be signed in to change notification settings - Fork 687
Closed
Labels
Description
We have a test suite running ginkgo tests in parallel on Jenkins. I'm currently trying to track down a hanging test in kubernetes/kubernetes#13485, but it's been very hard to figure out what's wrong, since the log output doesn't seem to be including the hanging test.
The log for one of our failing runs shows nothing for over 30m:
01:30:04 STEP: Destroying namespace "e2e-tests-pod-disks-z910m" for this suite.
01:30:04
01:30:04
01:30:04 • [SLOW TEST:376.248 seconds]
01:30:04 Pod Disks
01:30:04 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:267
01:30:04 should schedule a pod w/two RW PDs both mounted to one container, write to PD, verify contents, delete pod, recreate pod, verify contents, and repeat in rapid succession
01:30:04 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:266
01:30:04 ------------------------------
02:03:34 Build timed out (after 60 minutes). Marking the build as aborted.
02:03:34 Build timed out (after 60 minutes). Marking the build as failed.
02:03:34 Build was aborted
02:03:34 Recording test results
02:03:34
02:03:34 Ran 119 of 212 Specs in 2486.623 seconds
02:03:34 FAIL! -- 116 Passed | 0 Failed | 2 Pending | 91 Skipped
02:03:34
02:03:34 Ginkgo ran 1 suite in 41m27.528523653s
02:03:34 Test Suite Failed
I'm guessing this is intentional, since Ginkgo only prints out the log when a test completes. However, it doesn't seem to be handling the interrupt here properly - I'd expect it to dump any in-progress tests so that you could see what is stuck.
(I know about the Ginkgo parallel streaming mode, and I've been trying to use it, but this particular test failure seems to be very difficult to reproduce on demand.)