Skip to content

[SPARK-1681] Include datanucleus jars in Spark Hive distribution #610

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
andrewor14 wants to merge 7 commits into from
Closed

[SPARK-1681] Include datanucleus jars in Spark Hive distribution #610

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
58357cc
Include datanucleus jars in Spark distribution built with Hive support
andrewor14 May 1, 2014
940a1bb
Add back 2>/dev/null
andrewor14 May 1, 2014
32f6826
Leave the double colons
andrewor14 May 1, 2014
c16bbfd
Merge branch 'master' of github.com:apache/spark into hive-distribution
andrewor14 May 5, 2014
7855f58
Have jar command respect JAVA_HOME + check for jar errors both cases
andrewor14 May 5, 2014
fa205e1
Merge branch 'master' of github.com:apache/spark into hive-distribution
andrewor14 May 5, 2014
a4bc96f
Rename search path in jar error check
andrewor14 May 5, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 20 additions & 12 deletions bin/compute-classpath.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ FWDIR="$(cd `dirname $0`/..; pwd)"

# Build up classpath
CLASSPATH="$SPARK_CLASSPATH:$SPARK_SUBMIT_CLASSPATH:$FWDIR/conf"
CLASSPATH=$(echo "$CLASSPATH" | sed s/::/:/g)
Copy link
Contributor

@pwendell pwendell May 1, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this just for aesthetics or is there a correctness issue here?

Copy link
Contributor Author

@andrewor14 andrewor14 May 1, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aesthetics... @aarondav also pointed out that this is a little confusing. Maybe I should remove it.


ASSEMBLY_DIR="$FWDIR/assembly/target/scala-$SCALA_VERSION"

Expand All @@ -45,14 +46,14 @@ if [ -f "$ASSEMBLY_DIR"/spark-assembly*hadoop*-deps.jar ]; then
CLASSPATH="$CLASSPATH:$FWDIR/sql/core/target/scala-$SCALA_VERSION/classes"
CLASSPATH="$CLASSPATH:$FWDIR/sql/hive/target/scala-$SCALA_VERSION/classes"

DEPS_ASSEMBLY_JAR=`ls "$ASSEMBLY_DIR"/spark-assembly*hadoop*-deps.jar`
DEPS_ASSEMBLY_JAR=$(ls "$ASSEMBLY_DIR"/spark-assembly*hadoop*-deps.jar 2>/dev/null)
CLASSPATH="$CLASSPATH:$DEPS_ASSEMBLY_JAR"
else
# Else use spark-assembly jar from either RELEASE or assembly directory
if [ -f "$FWDIR/RELEASE" ]; then
ASSEMBLY_JAR=`ls "$FWDIR"/lib/spark-assembly*hadoop*.jar`
ASSEMBLY_JAR=$(ls "$FWDIR"/lib/spark-assembly*hadoop*.jar 2>/dev/null)
else
ASSEMBLY_JAR=`ls "$ASSEMBLY_DIR"/spark-assembly*hadoop*.jar`
ASSEMBLY_JAR=$(ls "$ASSEMBLY_DIR"/spark-assembly*hadoop*.jar 2>/dev/null)
fi
CLASSPATH="$CLASSPATH:$ASSEMBLY_JAR"
fi
Expand All @@ -63,14 +64,21 @@ fi
# built with Hive, so first check if the datanucleus jars exist, and then ensure the current Spark
# assembly is built for Hive, before actually populating the CLASSPATH with the jars.
# Note that this check order is faster (by up to half a second) in the case where Hive is not used.
num_datanucleus_jars=$(ls "$FWDIR"/lib_managed/jars/ 2>/dev/null | grep "datanucleus-.*\\.jar" | wc -l)
if [ $num_datanucleus_jars -gt 0 ]; then
AN_ASSEMBLY_JAR=${ASSEMBLY_JAR:-$DEPS_ASSEMBLY_JAR}
num_hive_files=$(jar tvf "$AN_ASSEMBLY_JAR" org/apache/hadoop/hive/ql/exec 2>/dev/null | wc -l)
if [ $num_hive_files -gt 0 ]; then
if [ -f "$FWDIR/RELEASE" ]; then
datanucleus_dir="$FWDIR"/lib
else
datanucleus_dir="$FWDIR"/lib_managed/jars
fi

datanucleus_jars=$(find "$datanucleus_dir" 2>/dev/null | grep "datanucleus-.*\\.jar")
Copy link
Contributor

@pwendell pwendell May 1, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to \ characters before the period? Don't you just need to escape it once?

Copy link
Contributor Author

@andrewor14 andrewor14 May 1, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first \ escapes the second one, so that only one \ is observed by grep

Copy link
Contributor

@pwendell pwendell May 1, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, seems to work for me without the slash:

$> touch datanucleus-hello.jar
$> datanucleus_jars=$(find . 2>/dev/null | grep "datanucleus-.*\.jar")
$> echo $datanucleus_jars
./datanucleus-hello.jar

Copy link
Contributor

@pwendell pwendell May 1, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oddly, works for me with either one or two slashes!

Copy link
Contributor

@pwendell pwendell May 3, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LMAO

$ echo "\."
\.
$ echo "\\."
\.

datanucleus_jars=$(echo "$datanucleus_jars" | tr "\n" : | sed s/:$//g)

if [ -n "$datanucleus_jars" ]; then
an_assembly_jar=${ASSEMBLY_JAR:-$DEPS_ASSEMBLY_JAR}
hive_files=$(jar tvf "$an_assembly_jar" org/apache/hadoop/hive/ql/exec)
Copy link
Contributor

@aarondav aarondav May 1, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should probably still pipe 2>/dev/null here as before, I believe this throws an error if it doesn't exist

Copy link
Contributor Author

@andrewor14 andrewor14 May 1, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forgot, thanks

if [ -n "$hive_files" ]; then
echo "Spark assembly has been built with Hive, including Datanucleus jars on classpath" 1>&2
DATANUCLEUSJARS=$(echo "$FWDIR/lib_managed/jars"/datanucleus-*.jar | tr " " :)
CLASSPATH=$CLASSPATH:$DATANUCLEUSJARS
CLASSPATH=$CLASSPATH:$datanucleus_jars
fi
fi

Expand All @@ -90,10 +98,10 @@ fi
# Add hadoop conf dir if given -- otherwise FileSystem.*, etc fail !
# Note, this assumes that there is either a HADOOP_CONF_DIR or YARN_CONF_DIR which hosts
# the configurtion files.
if [ "x" != "x$HADOOP_CONF_DIR" ]; then
if [ -n "$HADOOP_CONF_DIR" ]; then
CLASSPATH="$CLASSPATH:$HADOOP_CONF_DIR"
fi
if [ "x" != "x$YARN_CONF_DIR" ]; then
if [ -n "$YARN_CONF_DIR" ]; then
CLASSPATH="$CLASSPATH:$YARN_CONF_DIR"
fi

Expand Down
5 changes: 4 additions & 1 deletion make-distribution.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,10 @@ echo "Spark $VERSION built for Hadoop $SPARK_HADOOP_VERSION" > "$DISTDIR/RELEASE
cp $FWDIR/assembly/target/scala*/*assembly*hadoop*.jar "$DISTDIR/lib/"
cp $FWDIR/examples/target/scala*/spark-examples*.jar "$DISTDIR/lib/"

if [ "$SPARK_HIVE" == "true" ]; then
cp $FWDIR/lib_managed/jars/datanucleus*.jar "$DISTDIR/lib/"
fi

# Copy other things
mkdir "$DISTDIR"/conf
cp "$FWDIR"/conf/*.template "$DISTDIR"/conf
Expand All @@ -155,7 +159,6 @@ cp -r "$FWDIR/bin" "$DISTDIR"
cp -r "$FWDIR/python" "$DISTDIR"
cp -r "$FWDIR/sbin" "$DISTDIR"


# Download and copy in tachyon, if requested
if [ "$SPARK_TACHYON" == "true" ]; then
TACHYON_VERSION="0.4.1"
Expand Down