[SPARK-16742] Mesos Kerberos Support #17665

mgummelt · 2017-04-18T00:37:48Z

What changes were proposed in this pull request?

Mesos Kerberos Support. This does not yet include support for cluster mode, or for delegation token renewal. That will come in a future PR.

The delegation tokens are passed to the executors via RPC. The driver sets the tokens as a configuration property, which the executors then read by issuing the RetrieveSparkAppConfig RPC call (I did not add this call. It already existed).

ConfigurableCredentialProvider and all of the CredentialProvider classes have been moved from the yarn to core module so that they may be shared by Mesos.

How was this patch tested?

Manually against a kerberized HDFS cluster. Verified that the job failed w/o kinit and succeeded w/ kinit

SparkQA · 2017-04-18T00:44:05Z

Test build #75872 has finished for PR 17665 at commit 72bbe55.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2017-04-18T00:49:03Z

Test build #75873 has finished for PR 17665 at commit 4014b99.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

…e generic SparkHadoopUtil class

SparkQA · 2017-04-18T00:54:06Z

Test build #75874 has finished for PR 17665 at commit a47c9c0.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

mridulm · 2017-04-18T07:30:02Z

Skimmed through the PR - the way it has been currently designed, I dont think renewal can be added in future. I am hoping I missed something; can you point me to how it is going to be approached ? Some doc/proposal or notes ?

mridulm

Added a few queries I had at a high level.

mridulm · 2017-04-18T07:55:21Z

core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala

+    val tokensBuf = new java.io.ByteArrayInputStream(tokens)
+    creds.readTokenStorageStream(new java.io.DataInputStream(tokensBuf))
+    UserGroupInformation.getCurrentUser.addCredentials(creds)
+  }


This is relying on an immutable initial credential populated in the driver conf.
How is renewal scenario expected to be handled here ?

In my pending renewal PR, I've implemented it to use a new UpdateDelegationToken RPC call. This is just used for the initial transmission. I guess it makes sense to consolidate.

That would be great, thanks - I am trying to get a sense of how renewal works in this case.
There some ongoing work for having different ways to update credentials; and I was hoping this (from what I understand, not using hdfs but direct rpc) would be another way to do it - allowing for multiple common implementations which can be leveraged across all schedulers depending on requirements.

To elaborate; one potential use of it is to do token acquisition and token distribution without needing to provide principal/keytab to spark application (other than keeping track of last credential update to account for AM failures).
This is WIP though, but if mesos approach is different from yarn - that would be a great way to iterate on the latter aspect (token distribution); and ensure it is extensible enough for future requirements/implementations.

Yes, I believe @jerryshao is working on a standard method of delegating tokens. We can plug that in once it exists.

mridulm · 2017-04-18T07:59:27Z

core/src/main/scala/org/apache/spark/deploy/security/ServiceCredentialProvider.scala

+package org.apache.spark.deploy.security

 import org.apache.hadoop.conf.Configuration
 import org.apache.hadoop.security.{Credentials, UserGroupInformation}


Will this move not break all existing ServiceCredentialProvider's ? (since the service discovery via META-INF/services will get affected now).
@jerryshao and @vanzin can comment more since they wrote/reviewed this IIRC.

The code in the "yarn" module is sort of considered "private", so anyone extending those classes is walking on thin ice... the internal services files need to be moved / updated, but I don't think we need to worry about external implementations.

This also means we need to be more careful here when making classes / traits public. Maybe add @InterfaceStability.Evolving annotations if they haven't been added, or something.

(Haven't looked at the rest of the code yet, kinda waiting for the PR builder to be happy first.)

I've moved all the ServiceCredentialProvider subclasses to core, and I've updated the references in yarn's META-INF.

@vanzin Since this is documented in running-on-yarn.md, I dont think it can be considered internal implementation detail anymore; but it is part of our exposed contract.

@mgummelt I did see that all spark impl have been moved - but since this is an exposed contract, there can be (and are) external implementations which rely on it.
If we are breaking api compatibility, we should be very explicit about it : I wanted to make sure it is called out, and evaluated accordingly (and not an internal implementation detail within yarn/core).

Hmm... that's unfortunate, not just because of this change, but because yarn/ is not included in mima checks, so if the class changed in incompatible ways we wouldn't notice.

I really doubt anyone has custom implementations of that interface at this point. And making it a proper API in core is better going forward, IMO. If you really think that it's important to keep backwards compatibility, we could have the old interface present, extending the new one, and some compatibility code to keep both working. But that sounds like a lot of work for something I doubt anyone is currently using...

@vanzin Would be good to keep backward compatibility since there are users of the interface.
I did not realize yarn was excluded from MIMA - that is unfortunate.

Cursory look at ServiceLoader seemed to indicate it might not be very involved to provide backward compatibility (yarn interface extends from core, load for both interfaces; and use both as core impl will do ? or did I miss something here ?)

For backwards compat you'd have to call ServiceLoader twice, to load implementations of the two interfaces. That should be most of it, really, assuming the interface itself is not changing.

OK I'll try to unbreak backwards compatibility.

OK, I:

Readded deploy.yarn.security.ServiceCredentialProvider, extended from deploy.security.ServiceCredentialProvider, and deprecated it.

Added a new YARNConfigurableCredentialProvider, which extends from ConfigurableCredentialProvider, with the added behavior that it loads providers from deploy.yarn.security.ServiceCredentialProvider as well.

I'm not quite sure this is correct though. As it is, if a user has a service listed in the resources entry for both deploy.security.ServiceCredentialProvider and deploy.yarn.security.ServiceCredentialProvider, then they will fetch tokens twice for each service. This should only occur if the user's workflow is to drop a duplicated, but modified version of the deploy.yarn.security.ServiceCredentialProvider resources file into META-INF. If they have a fork of Spark that they merge, or if they have an entry in some other jar for just their customized service, there should be no duplication problem.

Thoughts?

mridulm · 2017-04-18T08:07:47Z

...ers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSecurityManager.scala

+    val numTokensBefore = userCreds.numberOfTokens
+    val hadoopConf = SparkHadoopUtil.get.newConfiguration(conf)
+    val credentialManager = new ConfigurableCredentialManager(conf, hadoopConf)
+    credentialManager.obtainCredentials(hadoopConf, userCreds)


This is potentially buggy - and will work only the first time for some credential providers - hbase (for example).
Token acquisition will fail for hbase when (prev) TOKEN is already present in the credentials - irrespective of whether kerberos credentials are present or not.
Acquiring credentials in context of ugi returned via UserGroupInformation.loginUserFromKeytabAndReturnUGI avoids this issue.

@jerryshao might have a better reference for the hbase behavior - since IIRC he worked a bit on it.

Fetching tokens multiple times isn't supported yet. That will come in the renewal PR.

When you say it will fail, do you mean it will throw an exception, or just fail to fetch a token? Looking at the code, I don't see how either would happen: https://github.com/mesosphere/spark/blob/SPARK-16742-kerberos/core/src/main/scala/org/apache/spark/deploy/security/HBaseCredentialProvider.scala#L46 It looks like it fetches a token indepedent of the current credentials, then overwrites any current HBase token via creds.addToken

@mgummelt It might be a bit involved, and has to do with how hbase issues tokens at its side (not in client library, but at the server) - if it sees a token in the incoming request, it will not issue new tokens.
@jerryshao is the right person to detail it if required, but bottomline is, if we add acquired tokens to ugi.currentUser, then subsequently we cannot use ugi.currentUser to acquire tokens for hbase. This actually comes from semantics of token, and might be applicable to other token providers as well (hbase is one case I know where it does this for sure).

Regardless, it shouldn't be an issue until we add renewal, right? Without renewal, we only ever fetch a single token.

Yes, that is why I said this is potentially buggy :-)
As it currently stands in the current happy path - since there is only a single invocation of the method, it will work.
But as soon as another invocation of the method happens in the same jvm (through any means); it will fail to acquire tokens.

…r for backwards compatibility

SparkQA · 2017-04-18T23:45:43Z

Test build #75916 has finished for PR 17665 at commit 954eeff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-04-19T00:00:15Z

Test build #75918 has finished for PR 17665 at commit 2d76928.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

mgummelt · 2017-04-19T00:45:10Z

.../resources/META-INF/services/org.apache.spark.deploy.yarn.security.ServiceCredentialProvider

@@ -1 +0,0 @@
-org.apache.spark.deploy.yarn.security.TestCredentialProvider


@vanzin Do you know the purpose of this test service provider? I don't see this class anywhere in the code base.

This is for unit test.

Which unit test uses it? Since the class doesn't exist anywhere, what effect does it have?

ConfigurableCredentialManagerSuite. It is loaded by ServiceLoader implicitly.

What do you mean "it does not exist anywhere"?

It's declared in ConfigurableCredentialManagerSuite.scala in my tree.

Ah, whoops. I was searching for the file with the same name. Thanks.

jerryshao · 2017-04-19T08:29:12Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

+    // must trick it into thinking we're YARN.
+    if (clusterManager == MESOS && UserGroupInformation.isSecurityEnabled) {
+      val shortUserName = UserGroupInformation.getCurrentUser.getShortUserName
+      val key = s"spark.hadoop.${YarnConfiguration.RM_PRINCIPAL}"


Does it work in user impersonation scenario? Here shortUserName is a real user, while HadoopRDD may execute as a proxy user.

Is it? It looks like getCurrentUser returns the subject from the current AccessControllerContext, which is set by the doAs when --proxy-user is set.

Regardless, the renewer specified here actually has no effect, since we aren't renewing yet. Once I add renewal, I'll need to revisit this to make sure it's consistent with the renewal we're going to do in MesosSecurity. I'm only setting this now to avoid an Exception that gets thrown in the hadoop library if the master principal is not configured. See the JIRA in the above code comment for details.

The renewer for delegation tokens should be the user that created them (if the service is not doing it for the user, like in the YARN case); only the logged in user can create tokens, so this has to be the current user (not the proxy user, which happens later).

The tokens should be created in the proxy user's name (so user = "proxy" renewer = "real user"), when you care to support that.

Once I add renewal

So do you mean that currently this only works until the delegation tokens need renewal (which is 1 day by default - a lot shorter than the max life time)?

So do you mean that currently this only works until the delegation tokens need renewal (which is 1 day by default - a lot shorter than the max life time)?

Yes

jerryshao · 2017-04-19T08:36:03Z

core/src/main/scala/org/apache/spark/deploy/security/ConfigurableCredentialManager.scala

-private[yarn] final class ConfigurableCredentialManager(
+private[spark] class ConfigurableCredentialManager(
    sparkConf: SparkConf, hadoopConf: Configuration) extends Logging {
  private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"


Shall we also rename the configurations, currently it is "spark.yarn.security.*"?

No, these are deprecated configs which need to keep the old names for compatibility.

SparkQA · 2017-04-19T17:45:11Z

Test build #75945 has finished for PR 17665 at commit d8a968d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

mgummelt · 2017-04-19T17:48:03Z

retest this please

SparkQA · 2017-04-19T18:00:26Z

Test build #75946 has finished for PR 17665 at commit d8a968d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

vanzin

I think this PR would benefit from being broken in two: one to deal with the refactoring of the existing code, without adding anything that is required by Mesos, and a second one to integrate the refactored code with Mesos. The refactoring has some parts in it that are a bit tricky to handle, and it would benefit from having a more focused PR that didn't have new features added into the mix.

vanzin · 2017-04-19T18:28:32Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

 import org.apache.commons.lang3.StringUtils
 import org.apache.hadoop.fs.Path
 import org.apache.hadoop.security.UserGroupInformation
+import org.apache.hadoop.yarn.conf.YarnConfiguration


Hmm... you're adding a YARN dependency in core, which should be able to build without YARN... but you're not actually adding a new dependency to the POM, so I guess the dependency is already there indirectly.

Yea it looks like this should require hadoop-yarn-api. I'll add this dependency to core unless you object, in which case I suppose I could just use the raw string instead of YarnConfiguration.RM_PRINCIPAL, but that seems hacky.

FYI, we originally talked about placing the code below into the Mesos scheduler, but that seems to be too late. The SparkContext constructor creates a copy of the configuration, so we need to set the required YARN property before SparkContext is created, which means before user code gets run, which probably means somewhere in SparkSubmit.

I don't think you need the explicit dependency (otherwise this would not be compiling). This is probably being brought by hadoop-client already.

Better to explicitly declare a dependency rather than rely on transitivity, right?

vanzin · 2017-04-19T18:38:54Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

+    // must trick it into thinking we're YARN.
+    if (clusterManager == MESOS && UserGroupInformation.isSecurityEnabled) {
+      val shortUserName = UserGroupInformation.getCurrentUser.getShortUserName
+      val key = s"spark.hadoop.${YarnConfiguration.RM_PRINCIPAL}"


The renewer for delegation tokens should be the user that created them (if the service is not doing it for the user, like in the YARN case); only the logged in user can create tokens, so this has to be the current user (not the proxy user, which happens later).

The tokens should be created in the proxy user's name (so user = "proxy" renewer = "real user"), when you care to support that.

Once I add renewal

So do you mean that currently this only works until the delegation tokens need renewal (which is 1 day by default - a lot shorter than the max life time)?

vanzin · 2017-04-19T18:39:57Z

core/src/main/scala/org/apache/spark/deploy/security/ConfigurableCredentialManager.scala

-private[yarn] final class ConfigurableCredentialManager(
+private[spark] class ConfigurableCredentialManager(
    sparkConf: SparkConf, hadoopConf: Configuration) extends Logging {
  private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"


No, these are deprecated configs which need to keep the old names for compatibility.

vanzin · 2017-04-19T18:40:22Z

core/src/main/scala/org/apache/spark/deploy/security/ConfigurableCredentialManager.scala

+private[spark] class ConfigurableCredentialManager(
    sparkConf: SparkConf, hadoopConf: Configuration) extends Logging {
  private val deprecatedProviderEnabledConfig = "spark.yarn.security.tokens.%s.enabled"
  private val providerEnabledConfig = "spark.yarn.security.credentials.%s.enabled"


Now these configs probably should be updated to the new namespace, with code added for backwards compatibility...

vanzin · 2017-04-19T18:49:21Z

core/src/main/scala/org/apache/spark/deploy/security/ServiceCredentialProvider.scala

+ * A credential provider for a service. User must implement this if they need to access a
+ * secure service from Spark.
+ */
+trait ServiceCredentialProvider {


No annotation means "Unstable" IIRC, but might as well be explicit here and add a proper InterfaceStability annotation.

How does InterfaceStability relate to scala language level access control (public, private, etc.)? Every public interface must maintain backwards compatibility during a major version, right? So does it make sense to have a public trait that's unstable?

It's not a Scala thing, it's a Spark API thing. When adding a public API that might change, we annotate it appropriately so that people can know that it's not really stable. Flip side, if you change an API that is tagged with a "Stable" annotation, you'll be yelled at.

It also shows up in docs:
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.util.QueryExecutionListener

(Basically this is the new version of @DeveloperApi and @Experimental.)

When adding a public API that might change

Ah, thanks. I didn't know this was allowed at all.

vanzin · 2017-04-19T19:44:23Z

core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala

 private[spark] object CoarseGrainedExecutorBackend extends Logging {

+  private def addMesosDelegationTokens(driverConf: SparkConf) {
+    val value = driverConf.get("spark.mesos.kerberos.userCredentials")


You really should be using the config constant you create here.

But a bigger issue is that this value will be written to event logs and shown in the UI. Instead of using the config for this, would it be too hard to add this as a field in SparkAppConfig (see CoarseGrainedClusterMessage.scala)?

Yea, I'll go ahead and change this to an RPC.

vanzin · 2017-04-19T19:46:24Z

dev/.rat-excludes

 .*\.sql
 .Rbuildignore
-org.apache.spark.deploy.yarn.security.ServiceCredentialProvider
+org.apache.spark.deploy.security.ServiceCredentialProvider


In a change I have internally I changed this to:

META-INF/services/*

If that works (haven't tested yet) I think that's better than having to keep changing / adding these files here...

vanzin · 2017-04-19T19:48:32Z

...ers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSecurityManager.scala

+  def setUGITokens(conf: SparkConf): Unit = {
+    val userCreds = getDelegationTokens(conf)
+
+    val byteStream = new java.io.ByteArrayOutputStream(1024 * 1024)


Is it necessary to create a large buffer up front? Credentials will be much smaller than that in most situations.

Also, why not just import these types?

No good reason. Fixed both.

vanzin · 2017-04-19T19:50:06Z

...scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala

    sparkConfVars: Map[String, String] = null,
-    home: String = "/path"): Unit = {
-    sparkConf = (new SparkConf)
+    home: String = "/path"): SparkConf = {


nit: parameters need an extra level of indent.

vanzin · 2017-04-19T19:55:56Z

.../main/resources/META-INF/services/org.apache.spark.deploy.security.ServiceCredentialProvider

@@ -0,0 +1,3 @@
+org.apache.spark.deploy.yarn.security.YARNHadoopFSCredentialProvider


This worried me a little bit. Because I'm pretty sure ServiceLoader will look at both files (this one and the one in core), and when running in YARN you'll end up with way too many providers.

So you should verify that behavior, and the best way to do it is to write a unit test.

Ah, yes. This is a problem. I'll write the unit test, and move the core META-INF to Mesos. I was hoping for a general set of registered providers, but I guess each scheduler will need to register their own.

move the core META-INF to Mesos

That's not enough. Because in a real app (not unit tests), the mesos and yarn jars will both still exist, so both files will still be loaded.

Oh yea, I guess the standard Spark distributions are compiled with both profiles. I created a separate YARNHadoopFSCredentialProvider to handle YARN's specific functionality (renewer, extra filesystems, etc.), but I guess I could do that through parameterization rather than a separate class, since it seems we can't have a YARN specific class through service loading. How does that sound?

but I guess I could do that through parameterization

That would probably be better. You can see if using SparkHadoopUtil / YarnSparkHadoopUtil to handle the differences can help too.

mgummelt · 2017-04-19T20:35:34Z

@vanzin How important is it to you for this to be two PRs? I already factored out the renewal logic to simplify this PR, and I'd rather not have to decouple again. But I can if it's a blocker.

vanzin · 2017-04-19T20:39:01Z

The thing that triggered my comment is the fact that I'm pretty sure you're breaking the YARN side with this change (loading too many credential providers, including one that probably won't work with YARN).

Having the refactoring separately would make it easier to make sure YARN keeps working as before without having to include all the new code that deals with Mesos in the mix.

mgummelt · 2017-04-19T23:35:30Z

core/pom.xml

+      -->
+    <dependency>
+      <groupId>${hive.group}</groupId>
+      <artifactId>hive-exec</artifactId>


I have a problem here that I could use some help on.

hive-exec has a dependency on pentaho that doesn't exist in maven central anymore. This is why we have an entry for it in spark-parent's dependencyManagement section that excludes pentaho via calcite-core.

But the scope in spark-parent's dependencyManagement entry for hive-exec is compile, so if I make it test here, which is what it was in YARN, then it fails to match that entry anymore, and thus doesn't inherit the excludes. This shouldn't be a problem, because dependencies in the test scope aren't transitive, so we should never be trying to fetch the missing pentaho dependency. But sbt builds don't seem to recognize this transitivity, so if I add the test scope, I get this error:

[warn] Note: Unresolved dependencies path: [warn] org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde [warn] +- org.apache.calcite:calcite-core:1.2.0-incubating ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76) [warn] +- org.spark-project.hive:hive-exec:1.2.1.spark2 ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76) [warn] +- org.apache.spark:spark-core_2.11:2.2.0-SNAPSHOT ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76) [warn] +- org.apache.spark:spark-catalyst_2.11:2.2.0-SNAPSHOT ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76) [warn] +- org.apache.spark:spark-sql_2.11:2.2.0-SNAPSHOT ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76) [warn] +- org.apache.spark:spark-hive_2.11:2.2.0-SNAPSHOT

What I can't understand is how this wasn't a problem when these dependencies were in YARN. For some reason sbt dependencyTree on master shows that calcite-core is excluded from the hive-exec dependency tree, but the same command on this PR tries to resolve calcite-core.

Let me know if you have any ideas here. My best idea right now is to just try to duplicate the excludes in spark-parent.

SparkQA · 2017-04-20T01:44:15Z

Test build #75956 has finished for PR 17665 at commit 25d5088.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mgummelt · 2017-04-20T17:01:18Z

@vanzin OK, I'll split it up.

…eCredentialProvider from core

mgummelt · 2017-04-20T22:19:28Z

I've addressed all review comments, and as per @vanzin's request, I'll now close this out and resubmit a PR with just the YARN refactoring.

SparkQA · 2017-04-21T00:34:08Z

Test build #76008 has finished for PR 17665 at commit 4c387eb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-05-19T17:52:57Z

...rc/test/scala/org/apache/spark/deploy/yarn/security/ConfigurableCredentialManagerSuite.scala

-    timeOfNextTokenRenewal = (currTime - currTime % tokenRenewalInterval) + tokenRenewalInterval
-
-    Some(timeOfNextTokenRenewal)
-  }


Could you leave comments for each deleted test case to explain what are the new location?

Sure. I'll come back to all of your requests once #17723 gets merged.

gatorsmile · 2017-05-19T17:58:22Z

core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala

      sparkProperties: Seq[(String, String)],
-      ioEncryptionKey: Option[Array[Byte]])
+      ioEncryptionKey: Option[Array[Byte]],
+      ugiTokens: Option[Array[Byte]])


Could you add descriptions what is ugiTokens?

gatorsmile · 2017-05-19T17:59:40Z

core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala

+  private def addDelegationTokens(tokenBytes: Array[Byte], driverConf: SparkConf) {
+    val creds = new CredentialsSerializer().deserializeTokens(tokenBytes)
+
+    logInfo(s"Adding ${creds.numberOfTokens()} tokens and ${creds.numberOfSecretKeys()} secret" +


Nit: secret" -> secret "

Dr. Stefan Schimanski and others added 7 commits April 17, 2017 17:50

[Mesosphere SPARK-126] Move YarnSparkHadoopUtil token helpers into th…

ce63a9b

…e generic SparkHadoopUtil class

[Mesosphere SPARK-126] Add Mesos Kerberos support

75d849a

Par down kerberos support

35002f2

cleanup

13981c8

style

af4a3e4

Add MesosSecurityManager

5cc66dc

info logs

a47c9c0

mgummelt force-pushed the SPARK-16742-kerberos branch from 4014b99 to a47c9c0 Compare April 18, 2017 00:51

mridulm reviewed Apr 18, 2017

View reviewed changes

Michael Gummelt added 3 commits April 18, 2017 13:24

style

c8ec049

Re-add org.apache.spark.deploy.yarn.security.ServiceCredentialProvide…

954eeff

…r for backwards compatibility

move YARNHadoopFSCredentialProviderSuite

2d76928

mgummelt commented Apr 19, 2017

View reviewed changes

jerryshao reviewed Apr 19, 2017

View reviewed changes

Move hive test deps to the core module

d8a968d

vanzin reviewed Apr 19, 2017

View reviewed changes

Michael Gummelt added 2 commits April 19, 2017 15:10

remove test scope

b8093c8

remove test scope

25d5088

mgummelt commented Apr 19, 2017

View reviewed changes

Michael Gummelt added 2 commits April 20, 2017 15:15

Removed MesosSecurityManager, added RPC call, removed META-INF Servic…

4c387eb

…eCredentialProvider from core

add InterfaceStability annotation to ServiceCredentialProvider

e32afee

mgummelt closed this Apr 20, 2017

mgummelt mentioned this pull request May 4, 2017

[SPARK-20434][YARN][CORE] Move Hadoop delegation token code from yarn to core #17723

Closed

gatorsmile reviewed May 19, 2017

View reviewed changes

		@@ -1 +0,0 @@
		org.apache.spark.deploy.yarn.security.TestCredentialProvider

		@@ -0,0 +1,3 @@
		org.apache.spark.deploy.yarn.security.YARNHadoopFSCredentialProvider

[SPARK-16742] Mesos Kerberos Support #17665

[SPARK-16742] Mesos Kerberos Support #17665

Uh oh!

Conversation

mgummelt commented Apr 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Apr 18, 2017

Uh oh!

SparkQA commented Apr 18, 2017

Uh oh!

SparkQA commented Apr 18, 2017

Uh oh!

mridulm commented Apr 18, 2017

Uh oh!

mridulm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mridulm Apr 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgummelt Apr 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 18, 2017

Uh oh!

SparkQA commented Apr 19, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgummelt Apr 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgummelt commented Apr 18, 2017 •

edited

Loading

mridulm Apr 18, 2017 •

edited

Loading

mgummelt Apr 18, 2017 •

edited

Loading

mgummelt Apr 19, 2017 •

edited

Loading