Skip to content

in parallel mode, no output for hanging test on timeout-induced interrupt #206

@ixdy

Description

@ixdy

We have a test suite running ginkgo tests in parallel on Jenkins. I'm currently trying to track down a hanging test in kubernetes/kubernetes#13485, but it's been very hard to figure out what's wrong, since the log output doesn't seem to be including the hanging test.

The log for one of our failing runs shows nothing for over 30m:

01:30:04 STEP: Destroying namespace "e2e-tests-pod-disks-z910m" for this suite.
01:30:04 
01:30:04 
01:30:04 • [SLOW TEST:376.248 seconds]
01:30:04 Pod Disks
01:30:04 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:267
01:30:04   should schedule a pod w/two RW PDs both mounted to one container, write to PD, verify contents, delete pod, recreate pod, verify contents, and repeat in rapid succession
01:30:04   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:266
01:30:04 ------------------------------
02:03:34 Build timed out (after 60 minutes). Marking the build as aborted.
02:03:34 Build timed out (after 60 minutes). Marking the build as failed.
02:03:34 Build was aborted
02:03:34 Recording test results
02:03:34 
02:03:34 Ran 119 of 212 Specs in 2486.623 seconds
02:03:34 FAIL! -- 116 Passed | 0 Failed | 2 Pending | 91 Skipped 
02:03:34 
02:03:34 Ginkgo ran 1 suite in 41m27.528523653s
02:03:34 Test Suite Failed

I'm guessing this is intentional, since Ginkgo only prints out the log when a test completes. However, it doesn't seem to be handling the interrupt here properly - I'd expect it to dump any in-progress tests so that you could see what is stuck.

(I know about the Ginkgo parallel streaming mode, and I've been trying to use it, but this particular test failure seems to be very difficult to reproduce on demand.)

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions