Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
21b6d7b
Initial commit for clojure package - src code
gigasquid Jun 8, 2018
d3961ff
examples for clojure package
gigasquid Jun 9, 2018
d4645b1
Add cloverage code coverage plugin.
gigasquid Jun 11, 2018
8d50031
add bug fix from Kamil Hryniewicz
gigasquid Jun 14, 2018
7b4b496
add basic optimizer test
gigasquid Jun 14, 2018
e7e1d27
add Kamil to thanks
gigasquid Jun 14, 2018
838330e
add Christian Weilbach
gigasquid Jun 14, 2018
8de88ed
add set -e to scripts
gigasquid Jun 18, 2018
2f876ba
added documentation about the current process of building and deploying
gigasquid Jun 18, 2018
948e24a
unit tests for util
gigasquid Jun 19, 2018
d39cd66
add tests for generator
gigasquid Jun 21, 2018
ecd065b
make the test-validate use a spec in util-test
gigasquid Jun 21, 2018
4e6f003
update to 1.2.1 scala version
gigasquid Jun 21, 2018
7f2a936
add generator test fix & enhancements from Burin
gigasquid Jun 22, 2018
b3fa023
add test for eval-metric
gigasquid Jun 22, 2018
0bf3961
add callback unit test
gigasquid Jun 22, 2018
2894adc
switch project default to linux-cpu for CI
gigasquid Jun 22, 2018
84cfd7f
Merge branch 'master' into clojure-package
gigasquid Jun 23, 2018
3c48fa3
remove reference to the clojars snapshot
gigasquid Jun 23, 2018
07add1b
@kurman feedback newline/scripts
gigasquid Jun 26, 2018
5662b98
feedback from @kurman - use scala instead of $
gigasquid Jun 26, 2018
ba7b5a3
move gitignore back into clojure-package
gigasquid Jun 26, 2018
6d7b128
feedback from @kurman - lowercase testing.md
gigasquid Jun 26, 2018
310693f
feedback from @kurman use generate code alias for better clarity
gigasquid Jun 26, 2018
97b5057
feedback from @kurman - cnn text example
gigasquid Jun 26, 2018
ad97924
Add clojure package testing to CI
gigasquid Jun 26, 2018
0358d95
bug fix from jimdunn - skip validation in fit when eval-data is nil
gigasquid Jun 26, 2018
edf795a
Change CI to use locally installed scala jars instead of nexus
gigasquid Jun 27, 2018
29fbe28
clean ci & bump clojure-package version to be inline with scala pacak…
gigasquid Jun 27, 2018
acfec15
update cat image link since the other one is broken
gigasquid Jun 27, 2018
69f9eb6
change cat link to one in the repo
gigasquid Jun 27, 2018
0863846
Merge branch 'master' into clojure-package
gigasquid Jun 27, 2018
29485b8
Update README
gigasquid Jun 28, 2018
6567766
missing minus sign in example - contributed by jimdunn
gigasquid Jun 28, 2018
3669345
feedback from @lanking520
gigasquid Jun 28, 2018
fa9f23b
feedback from @lanking520 - specify that branch checkout is optional
gigasquid Jun 28, 2018
6760c90
feedback from @yzhliu - update README ml.dmlc -> org.apache
gigasquid Jun 29, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -697,6 +697,18 @@ try {
}
}
},
'Clojure: CPU': {
node('mxnetlinux-cpu') {
ws('workspace/ut-clojure-cpu') {
timeout(time: max_time, unit: 'MINUTES') {
init_git()
unpack_lib('cpu', mx_dist_lib)
docker_run('ubuntu_cpu', 'unittest_ubuntu_cpu_clojure', false)
publish_test_coverage()
}
}
}
},
'Perl: CPU': {
node('mxnetlinux-cpu') {
ws('workspace/ut-perl-cpu') {
Expand Down
2 changes: 2 additions & 0 deletions ci/docker/Dockerfile.build.ubuntu_cpu
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ COPY install/ubuntu_python.sh /work/
RUN /work/ubuntu_python.sh
COPY install/ubuntu_scala.sh /work/
RUN /work/ubuntu_scala.sh
COPY install/ubuntu_clojure.sh /work/
RUN /work/ubuntu_clojure.sh
COPY install/ubuntu_r.sh /work/
RUN /work/ubuntu_r.sh
COPY install/ubuntu_perl.sh /work/
Expand Down
33 changes: 33 additions & 0 deletions ci/docker/install/ubuntu_clojure.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/bin/bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# build and install are separated so changes to build don't invalidate
# the whole docker cache for the image

set -ex
# install libraries for mxnet's clojure package on ubuntu
echo 'Installing Clojure...'

wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein
chmod 777 lein
sudo cp lein /usr/local/bin
sudo apt install libcurl3
sudo add-apt-repository ppa:timsc/opencv-3.4
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the opencv version that is in ubuntu too old? Is not clear why we are adding this to me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@gigasquid gigasquid Jun 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes - that is the issue I'm looking into. It appears that the version of the opencv that was used to build the scala jars for nexus is different than the opencv required by caffe. I'm looking into changing the unit test from the clojure package to run off of the ones generated by make scalapkg and make scalainstall instead, (which is most likely a better way to go anyway).

Using the scala jars from nexus on ubuntu uses this version
sudo apt install -y libopencv-imgcodecs3.4
which clashes with the one required in caffe build libopencv-dev

sudo apt-get update
sudo apt install -y libopencv-imgcodecs3.4
5 changes: 5 additions & 0 deletions ci/docker/runtime_functions.sh
Original file line number Diff line number Diff line change
Expand Up @@ -652,6 +652,11 @@ unittest_ubuntu_gpu_scala() {
make scalatest USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 SCALA_TEST_ON_GPU=1 USE_DIST_KVSTORE=1
}

unittest_ubuntu_cpu_clojure() {
set -ex
./contrib/clojure-package/test.sh
}

unittest_ubuntu_cpugpu_perl() {
set -ex
./perl-package/test.sh
Expand Down
4 changes: 2 additions & 2 deletions contrib/clojure-package/project.clj
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@
:dependencies [[org.clojure/clojure "1.9.0"]
[t6/from-scala "0.3.0"]
;; Choose the right dependency for your system
[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.2.1"]
;[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu "1.2.1"]
;[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.2.1"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering if Clojure always depended on Maven jars, would the clojure-jars be always 1 release behind Scala? or are you consuming from the source build from the master?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI build is taking it from master when the jars are installed locally with make scalainstall. I kept the maven jars lines in there as well commented out so that it is easy for a user to switch back and forth.

[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu "1.2.1"]
;[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu "1.2.1"]
[org.clojure/tools.logging "0.4.0"]
[org.apache.logging.log4j/log4j-core "2.8.1"]
Expand Down
23 changes: 23 additions & 0 deletions contrib/clojure-package/test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -evx

MXNET_HOME=${PWD}
cd ${MXNET_HOME}/contrib/clojure-package
lein test