Merge pull request apache#122 from palantir/branch-2.2.0-palantir4-k8s-release

Resync with k8s

Author: ash211

docs/running-on-kubernetes.md

@@ -3,9 +3,15 @@ layout: global
 title: Running Spark on Kubernetes
 ---
 
-Support for running on [Kubernetes](https://kubernetes.io/) is available in experimental status. The feature set is
+Support for running on [Kubernetes](https://kubernetes.io/docs/whatisk8s/) is available in experimental status. The feature set is
 currently limited and not well-tested. This should not be used in production environments.
 
+## Prerequisites
+
+* You must have a running Kubernetes cluster with access configured to it using [kubectl](https://kubernetes.io/docs/user-guide/prereqs/). If you do not already have a working Kubernetes cluster, you may setup a test cluster on your local machine using [minikube](https://kubernetes.io/docs/getting-started-guides/minikube/).
+* You must have appropriate permissions to create and list [pods](https://kubernetes.io/docs/user-guide/pods/), [nodes](https://kubernetes.io/docs/admin/node/) and [services](https://kubernetes.io/docs/user-guide/services/) in your cluster. You can verify that you can list these resources by running `kubectl get nodes`, `kubectl get pods` and `kubectl get svc` which should give you a list of nodes, pods and services (if any) respectively.
+* You must have an extracted spark distribution with Kubernetes support, or build one from [source](https://github.com/apache-spark-on-k8s/spark).
+
 ## Setting Up Docker Images
 
 Kubernetes requires users to supply images that can be deployed into containers within pods. The images are built to
@@ -49,14 +55,23 @@ being contacted at `api_server_url`. If no HTTP protocol is specified in the URL
 setting the master to `k8s://example.com:443` is equivalent to setting it to `k8s://https://example.com:443`, but to
 connect without SSL on a different port, the master would be set to `k8s://http://example.com:8443`.
 
+
+If you have a Kubernetes cluster setup, one way to discover the apiserver URL is by executing `kubectl cluster-info`.
+
+    > kubectl cluster-info
+    Kubernetes master is running at http://127.0.0.1:8080
+
+In the above example, the specific Kubernetes cluster can be used with spark submit by specifying
+`--master k8s://http://127.0.0.1:8080` as an argument to spark-submit.
+
 Note that applications can currently only be executed in cluster mode, where the driver and its executors are running on
 the cluster.
 
 ### Dependency Management and Docker Containers
 
 Spark supports specifying JAR paths that are either on the submitting host's disk, or are located on the disk of the
 driver and executors. Refer to the [application submission](submitting-applications.html#advanced-dependency-management)
-section for details. Note that files specified with the `local` scheme should be added to the container image of both
+section for details. Note that files specified with the `local://` scheme should be added to the container image of both
 the driver and the executors. Files without a scheme or with the scheme `file://` are treated as being on the disk of
 the submitting machine, and are uploaded to the driver running in Kubernetes before launching the application.
 
@@ -81,7 +96,7 @@ the driver container as a [secret volume](https://kubernetes.io/docs/user-guide/
 ### Kubernetes Clusters and the authenticated proxy endpoint
 
 Spark-submit also supports submission through the
-[local kubectl proxy](https://kubernetes.io/docs/user-guide/connecting-to-applications-proxy/). One can use the
+[local kubectl proxy](https://kubernetes.io/docs/user-guide/accessing-the-cluster/#using-kubectl-proxy). One can use the
 authenticating proxy to communicate with the api server directly without passing credentials to spark-submit.
 
 The local proxy can be started by running:

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/kubernetes/Client.scala

@@ -201,6 +201,7 @@ private[spark] class Client(
         } catch {
           case e: Throwable =>
             driverServiceManager.handleSubmissionError(e)
+            throw e
         } finally {
           Utils.tryLogNonFatalError {
             kubernetesResourceCleaner.deleteAllRegisteredResourcesFromKubernetes(kubernetesClient)

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/kubernetes/KubernetesResourceCleaner.scala

@@ -39,13 +39,15 @@ private[spark] class KubernetesResourceCleaner extends Logging {
 
   def deleteAllRegisteredResourcesFromKubernetes(kubernetesClient: KubernetesClient): Unit = {
     synchronized {
-      logInfo(s"Deleting ${resources.size} registered Kubernetes resources:")
+      val resourceCount = resources.size
+      logInfo(s"Deleting ${resourceCount} registered Kubernetes resources...")
       resources.values.foreach { resource =>
         Utils.tryLogNonFatalError {
           kubernetesClient.resource(resource).delete()
         }
       }
       resources.clear()
+      logInfo(s"Deleted ${resourceCount} registered Kubernetes resources.")
     }
   }
 }