Skip to content

Commit e892004

Browse files
committed
[SPARK-17418] Remove Kinesis artifacts from Spark release scripts
Kinesis license (Amazon Software License) is classified as category-x and is not allowed in Apache software projects releases.
1 parent a425a37 commit e892004

2 files changed

Lines changed: 4 additions & 2 deletions

File tree

dev/create-release/release-build.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ BASE_DIR=$(pwd)
8181

8282
MVN="build/mvn --force"
8383
PUBLISH_PROFILES="-Pmesos -Pyarn -Phive -Phive-thriftserver -Phadoop-2.2"
84-
PUBLISH_PROFILES="$PUBLISH_PROFILES -Pspark-ganglia-lgpl -Pkinesis-asl"
84+
PUBLISH_PROFILES="$PUBLISH_PROFILES -Pspark-ganglia-lgpl"
8585

8686
rm -rf spark
8787
git clone https://git-wip-us.apache.org/repos/asf/spark.git

docs/streaming-kinesis-integration.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,9 @@ A Kinesis stream can be set up at one of the valid Kinesis endpoints with 1 or m
119119

120120
3. **Deploying:** As with any Spark applications, `spark-submit` is used to launch your application. However, the details are slightly different for Scala/Java applications and Python applications.
121121

122-
For Scala and Java applications, if you are using SBT or Maven for project management, then package `spark-streaming-kinesis-asl_{{site.SCALA_BINARY_VERSION}}` and its dependencies into the application JAR. Make sure `spark-core_{{site.SCALA_BINARY_VERSION}}` and `spark-streaming_{{site.SCALA_BINARY_VERSION}}` are marked as `provided` dependencies as those are already present in a Spark installation. Then use `spark-submit` to launch your application (see [Deploying section](streaming-programming-guide.html#deploying-applications) in the main programming guide).
122+
You'll first need to perform a custom build of Spark. **Note that by linking to this library, you will include [ASL](https://aws.amazon.com/asl/)-licensed code in your application.** Download Spark source and follow the [instructions](building-spark.html) to build Spark with profile *-Pkinesis-asl*.
123+
124+
For Scala and Java applications, if you are using SBT or Maven for project management, you'll need to package `spark-streaming-kinesis-asl_{{site.SCALA_BINARY_VERSION}}` and its dependencies into the application JAR. Make sure `spark-core_{{site.SCALA_BINARY_VERSION}}` and `spark-streaming_{{site.SCALA_BINARY_VERSION}}` are marked as `provided` dependencies as those are already present in a Spark installation. Then use `spark-submit` to launch your application (see [Deploying section](streaming-programming-guide.html#deploying-applications) in the main programming guide).
123125

124126
For Python applications which lack SBT/Maven project management, `spark-streaming-kinesis-asl_{{site.SCALA_BINARY_VERSION}}` and its dependencies can be directly added to `spark-submit` using `--packages` (see [Application Submission Guide](submitting-applications.html)). That is,
125127

0 commit comments

Comments
 (0)