Job "spark-integration-test-minikube-k8s" testing failed

[kiwik]

Thank you for reporting bug!

Describe the bug

Periodic job spark-integration-test-minikube-k8s failed in past week, build Spark distribution and deploy minikube sucessfully, there are 5 test cases failing when run dev-run-integration-tests.sh

Related Job information

If the bug is related a CI job, put the job information below:

• job name: spark-integration-test-minikube-k8s
• job build log link: https://logs.openlabtesting.org/logs/periodic-0/12/github.com/apache/spark/master/spark-integration-test-minikube-k8s-v1.10.0/0029b00/job-output.txt.gz
• error log piece:

2019-06-16 12:33:20.116195 | ubuntu-xenial | Executing glib-2.58.1-r2.trigger
2019-06-16 12:33:20.124678 | ubuntu-xenial | OK: 483 MiB in 122 packages
2019-06-16 12:33:25.142872 | ubuntu-xenial | Removing intermediate container a75806be6899
2019-06-16 12:33:25.143036 | ubuntu-xenial |  ---> d7f954fbbf42
2019-06-16 12:33:25.143076 | ubuntu-xenial | Step 6/9 : COPY R ${SPARK_HOME}/R
2019-06-16 12:33:25.200425 | ubuntu-xenial | COPY failed: stat /var/lib/docker/tmp/docker-builder361656972/R: no such file or directory
2019-06-16 12:33:25.204437 | ubuntu-xenial | /home/zuul/src/github.com/apache/spark/resource-managers/kubernetes/integration-tests
2019-06-16 12:33:25.226038 | ubuntu-xenial | [INFO]
2019-06-29 00:17:34.156115 | ubuntu-xenial | [INFO] --- scalatest-maven-plugin:1.0:test (integration-test) @ spark-kubernetes-integration-tests_2.11 ---
2019-06-29 00:17:34.594668 | ubuntu-xenial | Discovery starting.
2019-06-29 00:17:34.686327 | ubuntu-xenial | Discovery completed in 141 milliseconds.
2019-06-29 00:17:34.698317 | ubuntu-xenial | Run starting. Expected test count is: 14
2019-06-29 00:17:34.705681 | ubuntu-xenial | KubernetesSuite:
2019-06-29 00:17:46.092157 | ubuntu-xenial | - Run SparkPi with no resources
2019-06-29 00:18:08.447092 | ubuntu-xenial | - Run SparkPi with a very long application name.
2019-06-29 00:18:28.729423 | ubuntu-xenial | - Use SparkLauncher.NO_RESOURCE
2019-06-29 00:18:40.795711 | ubuntu-xenial | - Run SparkPi with a master URL without a scheme.
2019-06-29 00:18:59.022259 | ubuntu-xenial | - Run SparkPi with an argument.
2019-06-29 00:19:19.357045 | ubuntu-xenial | - Run SparkPi with custom labels, annotations, and environment variables.
2019-06-29 00:19:35.745308 | ubuntu-xenial | - Run extraJVMOptions check on driver
2019-06-29 00:19:50.235399 | ubuntu-xenial | - Run SparkRemoteFileTest using a remote data file
2019-06-29 00:21:52.855038 | ubuntu-xenial | - Run SparkPi with env and mount secrets. *** FAILED ***
2019-06-29 00:21:52.855166 | ubuntu-xenial |   The code passed to eventually never returned normally. Attempted 70 times over 2.0070965230333333 minutes. Last failure message: 404 page not found
2019-06-29 00:21:52.855192 | ubuntu-xenial |   . (SecretsTestsSuite.scala:84)
2019-06-29 00:21:54.985754 | ubuntu-xenial | Exception in thread "OkHttp Dispatcher" java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@a155b00 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@47272ef[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2019-06-29 00:21:54.985863 | ubuntu-xenial | 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
2019-06-29 00:21:54.985925 | ubuntu-xenial | 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
2019-06-29 00:21:54.986059 | ubuntu-xenial | 	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
2019-06-29 00:21:54.986182 | ubuntu-xenial | 	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
2019-06-29 00:21:54.986304 | ubuntu-xenial | 	at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:632)
2019-06-29 00:21:54.986414 | ubuntu-xenial | 	at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
2019-06-29 00:21:54.986566 | ubuntu-xenial | 	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.scheduleReconnect(WatchConnectionManager.java:303)
2019-06-29 00:21:54.986711 | ubuntu-xenial | 	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.access$800(WatchConnectionManager.java:48)
2019-06-29 00:21:54.986851 | ubuntu-xenial | 	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onFailure(WatchConnectionManager.java:216)
2019-06-29 00:21:54.986942 | ubuntu-xenial | 	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
2019-06-29 00:21:54.987033 | ubuntu-xenial | 	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
2019-06-29 00:21:54.987104 | ubuntu-xenial | 	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
2019-06-29 00:21:54.987178 | ubuntu-xenial | 	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
2019-06-29 00:21:54.987285 | ubuntu-xenial | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-06-29 00:21:54.987391 | ubuntu-xenial | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-06-29 00:21:54.987440 | ubuntu-xenial | 	at java.lang.Thread.run(Thread.java:748)
2019-06-29 00:21:54.989008 | ubuntu-xenial | - Run PySpark on simple pi.py example *** FAILED ***
2019-06-29 00:21:54.989098 | ubuntu-xenial |   io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found
2019-06-29 00:21:54.989165 | ubuntu-xenial |   at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onFailure(WatchConnectionManager.java:201)
2019-06-29 00:21:54.989247 | ubuntu-xenial |   at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
2019-06-29 00:21:54.989310 | ubuntu-xenial |   at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
2019-06-29 00:21:54.989342 | ubuntu-xenial |   at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
2019-06-29 00:21:54.989376 | ubuntu-xenial |   at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
2019-06-29 00:21:54.989415 | ubuntu-xenial |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-06-29 00:21:54.989520 | ubuntu-xenial |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-06-29 00:21:54.991544 | ubuntu-xenial |   at java.lang.Thread.run(Thread.java:748)
2019-06-29 00:21:54.991573 | ubuntu-xenial |   ...
2019-06-29 00:22:01.150886 | ubuntu-xenial | - Run PySpark with Python2 to test a pyfiles example *** FAILED ***
2019-06-29 00:22:01.151502 | ubuntu-xenial |   io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found
2019-06-29 00:22:01.151851 | ubuntu-xenial |   at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onFailure(WatchConnectionManager.java:201)
2019-06-29 00:22:01.152317 | ubuntu-xenial |   at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
2019-06-29 00:22:01.152645 | ubuntu-xenial |   at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
2019-06-29 00:22:01.152825 | ubuntu-xenial |   at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
2019-06-29 00:22:01.153160 | ubuntu-xenial |   at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
2019-06-29 00:22:01.153618 | ubuntu-xenial |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-06-29 00:22:01.155242 | ubuntu-xenial |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-06-29 00:22:01.155511 | ubuntu-xenial |   at java.lang.Thread.run(Thread.java:748)
2019-06-29 00:22:01.155581 | ubuntu-xenial |   ...
2019-06-29 00:22:11.263900 | ubuntu-xenial | - Run PySpark with Python3 to test a pyfiles example *** FAILED ***
2019-06-29 00:22:11.264034 | ubuntu-xenial |   io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found
2019-06-29 00:22:11.264147 | ubuntu-xenial |   at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onFailure(WatchConnectionManager.java:201)
2019-06-29 00:22:11.264224 | ubuntu-xenial |   at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
2019-06-29 00:22:11.264321 | ubuntu-xenial |   at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
2019-06-29 00:22:11.264385 | ubuntu-xenial |   at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
2019-06-29 00:22:11.264448 | ubuntu-xenial |   at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
2019-06-29 00:22:11.264515 | ubuntu-xenial |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-06-29 00:22:11.264584 | ubuntu-xenial |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-06-29 00:22:11.264624 | ubuntu-xenial |   at java.lang.Thread.run(Thread.java:748)
2019-06-29 00:22:11.264638 | ubuntu-xenial |   ...
2019-06-29 00:22:21.436118 | ubuntu-xenial | - Run PySpark with memory customization *** FAILED ***
2019-06-29 00:22:21.436314 | ubuntu-xenial |   io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found
2019-06-29 00:22:21.436534 | ubuntu-xenial |   at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onFailure(WatchConnectionManager.java:201)
2019-06-29 00:22:21.437649 | ubuntu-xenial |   at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
2019-06-29 00:22:21.437810 | ubuntu-xenial |   at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
2019-06-29 00:22:21.437996 | ubuntu-xenial |   at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
2019-06-29 00:22:21.438241 | ubuntu-xenial |   at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
2019-06-29 00:22:21.438362 | ubuntu-xenial |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-06-29 00:22:21.438517 | ubuntu-xenial |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-06-29 00:22:21.438627 | ubuntu-xenial |   at java.lang.Thread.run(Thread.java:748)
2019-06-29 00:22:21.438664 | ubuntu-xenial |   ...
2019-06-29 00:22:37.712860 | ubuntu-xenial | - Run in client mode.
2019-06-29 00:22:47.895756 | ubuntu-xenial | Run completed in 5 minutes, 13 seconds.
2019-06-29 00:22:47.895821 | ubuntu-xenial | Total number of tests run: 14
2019-06-29 00:22:47.895842 | ubuntu-xenial | Suites: completed 2, aborted 0
2019-06-29 00:22:47.895872 | ubuntu-xenial | Tests: succeeded 9, failed 5, canceled 0, ignored 0, pending 0
2019-06-29 00:22:47.895887 | ubuntu-xenial | *** 5 TESTS FAILED ***

Steps to reproduce the issue

Describe how to reproduce the issue:
Check Spark periodic jobs everyday, http://status.openlabtesting.org/builds?project=apache/spark

Additional context

Add any other context about the problem here.
NA