Skip to content

Commit e173e3f

Browse files
committed
SPARK-7481: artifact is spark-hadoop-cloud while project is hadoop-cloud. Added an extra callout to the docs "without some form of consistency layer, Amazon S3 cannot
be safely used as the direct destination of work with the normal rename-based committer" That holds for all Spark-in-EC deployments; EMR has a consistency option, as do/will others; different committers can work reliably. Change-Id: Ibbf8b1b9de10b5bb83b647cc663ceb970f70ff2d Testing: SBT and maven, with & without the hadoop-2.7 option.
1 parent b788494 commit e173e3f

File tree

3 files changed

+5
-2
lines changed

3 files changed

+5
-2
lines changed

assembly/pom.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -235,7 +235,7 @@
235235
<dependencies>
236236
<dependency>
237237
<groupId>org.apache.spark</groupId>
238-
<artifactId>hadoop-cloud_${scala.binary.version}</artifactId>
238+
<artifactId>spark-hadoop-cloud_${scala.binary.version}</artifactId>
239239
<version>${project.version}</version>
240240
</dependency>
241241
</dependencies>

docs/cloud-integration.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,9 @@ For these reasons, it is not always safe to use an object store as a direct dest
6464
an intermediate store in a chain of queries. Consult the documentation of the object store and its
6565
connector to determine which uses are considered safe.
6666

67+
In particular: *without some form of consistency layer, Amazon S3 cannot
68+
be safely used as the direct destination of work with the normal rename-based committer.*
69+
6770
### Installation
6871

6972
With the relevant libraries on the classpath and Spark configured with valid credentials,

hadoop-cloud/pom.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626
<relativePath>../pom.xml</relativePath>
2727
</parent>
2828

29-
<artifactId>hadoop-cloud_2.11</artifactId>
29+
<artifactId>spark-hadoop-cloud_2.11</artifactId>
3030
<packaging>jar</packaging>
3131
<name>Spark Project Cloud Integration</name>
3232
<description>

0 commit comments

Comments
 (0)