[SPARK-26482][K8S][TEST][FOLLOWUP] Fix compile failure

[dongjoon-hyun]

What changes were proposed in this pull request?

This fixes K8S integration test compilation failure introduced by #23423 .

$ build/sbt -Pkubernetes-integration-tests test:package
...
[error] /Users/dongjoon/APACHE/spark/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesTestComponents.scala:71: type mismatch;
[error]  found   : org.apache.spark.internal.config.OptionalConfigEntry[Boolean]
[error]  required: String
[error]       .set(IS_TESTING, false)
[error]            ^
[error] /Users/dongjoon/APACHE/spark/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesTestComponents.scala:71: type mismatch;
[error]  found   : Boolean(false)
[error]  required: String
[error]       .set(IS_TESTING, false)
[error]                        ^
[error] two errors found

How was this patch tested?

Pass the K8S integration test.

[dongjoon-hyun]

cc @ifilonenko , @vanzin , @HeartSaVioR

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/6973/

[SparkQA]

Test build #101112 has finished for PR 23527 at commit 5617794.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/6973/

[felixcheung]

I'm confused - how did this go in originally that breaks the build?

[dongjoon-hyun]

Thank you for review, @felixcheung . It seems that K8S integration test triggers only when the PR title has [K8S] instead of the code change. Recently, two configuration refactoring PRs (including this) breaks K8S integration test suite because they don't have [K8S] tag.

[dongjoon-hyun]

Merged to master.

[vanzin]

It seems that K8S integration test triggers only when the PR title has [K8S] instead of the code change

That seems to be a problem.

[felixcheung]

looks like integration test is not in the module list https://github.com/apache/spark/blob/master/dev/sparktestsupport/modules.py#L542

so it doesn't get build unless the profile kubernetes-integration-tests is enabled explicitly
https://github.com/apache/spark/blob/master/pom.xml#L2726

[felixcheung]

I think we need it added to module and then only "execute the test" when the profile is set

[HeartSaVioR]

Thanks @dongjoon-hyun for fixing missing spot! Looks like I confused between SparkConf and SparkAppConf.

I couldn't run all tests in local (since it takes hours) but was trying to ensure compilation passes, so ran below command:

mvn -Phadoop-2.7 -Pkubernetes -Phive-thriftserver -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos clean compile test-compile

Maybe I needed to add -Pkubernetes-integration-tests as well.