From 85c33949706e8ba69c33ce73e4f1634d5015a094 Mon Sep 17 00:00:00 2001 From: "Wu, Xiaochang" Date: Thu, 16 Jul 2020 14:54:24 +0800 Subject: [PATCH 1/9] Add Spark MLlib Linear Algebra Acceleration Guide --- docs/ml-guide.md | 22 ++++--------- docs/ml-linalg-guide.md | 73 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+), 16 deletions(-) create mode 100644 docs/ml-linalg-guide.md diff --git a/docs/ml-guide.md b/docs/ml-guide.md index ddce98b32f94..e4c11da582b2 100644 --- a/docs/ml-guide.md +++ b/docs/ml-guide.md @@ -62,23 +62,13 @@ The primary Machine Learning API for Spark is now the [DataFrame](sql-programmin # Dependencies -MLlib uses the linear algebra package [Breeze](http://www.scalanlp.org/), which depends on -[netlib-java](https://github.com/fommil/netlib-java) for optimised numerical processing. -If native libraries[^1] are not available at runtime, you will see a warning message and a pure JVM -implementation will be used instead. +MLlib uses linear algebra packages [Breeze](http://www.scalanlp.org/) and [netlib-java](https://github.com/fommil/netlib-java) for optimised numerical processing[^1]. Those packages may call native acceleration libraries such as [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) or [OpenBLAS](http://www.openblas.net) if they are available as system libraries or in runtime library paths. -Due to licensing issues with runtime proprietary binaries, we do not include `netlib-java`'s native -proxies by default. -To configure `netlib-java` / Breeze to use system optimised binaries, include -`com.github.fommil.netlib:all:1.1.2` (or build Spark with `-Pnetlib-lgpl`) as a dependency of your -project and read the [netlib-java](https://github.com/fommil/netlib-java) documentation for your -platform's additional installation instructions. - -The most popular native BLAS such as [Intel MKL](https://software.intel.com/en-us/mkl), [OpenBLAS](http://www.openblas.net), can use multiple threads in a single operation, which can conflict with Spark's execution model. - -Configuring these BLAS implementations to use a single thread for operations may actually improve performance (see [SPARK-21305](https://issues.apache.org/jira/browse/SPARK-21305)). It is usually optimal to match this to the number of cores each Spark task is configured to use, which is 1 by default and typically left at 1. - -Please refer to resources like the following to understand how to configure the number of threads these BLAS implementations use: [Intel MKL](https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications) or [Intel oneMKL](https://software.intel.com/en-us/onemkl-linux-developer-guide-improving-performance-with-threading) and [OpenBLAS](https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded). Note that if nativeBLAS is not properly configured in system, java implementation(f2jBLAS) will be used as fallback option. +Due to licensing issues with runtime proprietary binaries, we do not include `netlib-java`'s native proxies by default. See [MLlib Linear Algebra Acceleration Guide](ml-linalg-guide.md) for how to enable accelerated linear algebra processing. If accelerated native libraries are not enabled, you will see a warning message below and a pure JVM implementation will be used instead. +``` +WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeSystemBLAS +WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeRefBLAS +``` To use MLlib in Python, you will need [NumPy](http://www.numpy.org) version 1.4 or newer. diff --git a/docs/ml-linalg-guide.md b/docs/ml-linalg-guide.md new file mode 100644 index 000000000000..6b65e9298035 --- /dev/null +++ b/docs/ml-linalg-guide.md @@ -0,0 +1,73 @@ +# Spark MLlib Linear Algebra Acceleration Guide + +## Introduction + +This guide provides necessary information to enable accelerated linear algebra processing for Spark MLlib. + +Spark MLlib defined Vector and Matrix basic data types for machine learning algorithms. On top of them, [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) and [LAPACK](https://en.wikipedia.org/wiki/LAPACK) operations are implemented and supported by [netlib-java](https://github.com/fommil/netlib-Java).[^1] `netlib-java` can use optimized native libraries for faster numerical processing. [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) and [OpenBLAS](http://www.openblas.net) are two most popular native linear algebra libraries (refered to as "native libraries" hereafter). + +However due to license restrictions, the official released Spark binaries by default doesn't contain native libraries enabled for `netlib-java`. + +The following sections describe how to enable `netlib-java` with native libraries support, to install the native libraries and configure them properly. + +[^1]: The algorithms may call Breeze and it will in turn call `netlib-java`. + +## Enable `netlib-java` + +To build Spark with `netlib-java` native library proxies, you need to add `-Pnetlib-lgpl` profile to Maven build command line. For example: +``` +$SPARK_SOURCE_HOME/build/mvn -Pnetlib-lgpl -DskipTests -Pyarn -Phadoop-2.7 clean package +``` + +If you only want to enable it in your project, include `com.github.fommil.netlib:all:1.1.2` as a dependency of your project. + + +and read the [netlib-java](https://github.com/fommil/netlib-java) documentation for your platform's additional installation instructions. + +## Install Native System Libraries + +Intel MKL and OpenBLAS are two most popular native linear algebra libraries, you can choose one of them based on your preference. We will describe how to install them for `netlib-java`. + +### Intel MKL + +1. Download and install Intel MKL. The installation should be done on all nodes of the cluster. We assume the installation location is $MKLROOT. +2. Make sure `/usr/local/lib` is in system library search path and run the following commands: +``` +$ ln -sf $MKLROOT/lib/intel64/libmkl_rt.so /usr/local/lib/libblas.so.3 +$ ln -sf $MKLROOT/lib/intel64/libmkl_rt.so /usr/local/lib/liblapack.so.3 +``` + +### OpenBLAS + +TBD + +## Check if native libraries are enabled for MLlib + +To verify native libraries are properly loaded, start `spark-shell` and run the following code +``` +scala> import com.github.fommil.netlib.BLAS; +scala> System.out.println(BLAS.getInstance().getClass().getName()); +``` + +If they arecorrectly loaded, it should print `com.github.fommil.netlib.NativeSystemBLAS`. Otherwise the warnings should be printed: +``` +WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeSystemBLAS +WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeRefBLAS +``` + +if native libraries are not properly configured in the system, Java implementation(f2jBLAS) will be used as fallback option. + +## Spark Configuration + +The use of multiple-threading in either Intel MKL or OpenBLAS can conflict with Spark's execution model.[^2] + +Therefore configuring these native libraries to use a single thread for operations may actually improve performance (see [SPARK-21305](https://issues.apache.org/jira/browse/SPARK-21305)). It is usually optimal to match this to the number of `spark.task.cpus`, which is 1 by default and typically left at 1. + +Use the options in `config/spark-env.sh` to disable multi-threading by setting thread number to 1. +``` +# You might get better performance to enable these options if using native BLAS (see SPARK-21305). +# - MKL_NUM_THREADS=1 Disable multi-threading of Intel MKL +# - OPENBLAS_NUM_THREADS=1 Disable multi-threading of OpenBLAS +``` + +[^2]: Please refer to resources like the following to understand how to configure the number of threads these BLAS implementations use: [Intel MKL](https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications) or [Intel oneMKL](https://software.intel.com/en-us/onemkl-linux-developer-guide-improving-performance-with-threading) and [OpenBLAS](https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded). From ffacb498f0a37904538ac11b953805d265d2e460 Mon Sep 17 00:00:00 2001 From: "Wu, Xiaochang" Date: Thu, 16 Jul 2020 18:08:27 +0800 Subject: [PATCH 2/9] Improve doc --- docs/ml-guide.md | 2 +- docs/ml-linalg-guide.md | 48 +++++++++++++++++++++++++---------------- 2 files changed, 31 insertions(+), 19 deletions(-) diff --git a/docs/ml-guide.md b/docs/ml-guide.md index e4c11da582b2..806176734c8c 100644 --- a/docs/ml-guide.md +++ b/docs/ml-guide.md @@ -64,7 +64,7 @@ The primary Machine Learning API for Spark is now the [DataFrame](sql-programmin MLlib uses linear algebra packages [Breeze](http://www.scalanlp.org/) and [netlib-java](https://github.com/fommil/netlib-java) for optimised numerical processing[^1]. Those packages may call native acceleration libraries such as [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) or [OpenBLAS](http://www.openblas.net) if they are available as system libraries or in runtime library paths. -Due to licensing issues with runtime proprietary binaries, we do not include `netlib-java`'s native proxies by default. See [MLlib Linear Algebra Acceleration Guide](ml-linalg-guide.md) for how to enable accelerated linear algebra processing. If accelerated native libraries are not enabled, you will see a warning message below and a pure JVM implementation will be used instead. +Due to licensing issues with runtime proprietary binaries, we do not include `netlib-java`'s native proxies by default. See [MLlib Linear Algebra Acceleration Guide](ml-linalg-guide.md) for how to enable accelerated linear algebra processing. If accelerated native libraries are not enabled, you will see a warning message below and a pure JVM implementation will be used instead: ``` WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeSystemBLAS WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeRefBLAS diff --git a/docs/ml-linalg-guide.md b/docs/ml-linalg-guide.md index 6b65e9298035..ae083cc6f0a9 100644 --- a/docs/ml-linalg-guide.md +++ b/docs/ml-linalg-guide.md @@ -4,34 +4,36 @@ This guide provides necessary information to enable accelerated linear algebra processing for Spark MLlib. -Spark MLlib defined Vector and Matrix basic data types for machine learning algorithms. On top of them, [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) and [LAPACK](https://en.wikipedia.org/wiki/LAPACK) operations are implemented and supported by [netlib-java](https://github.com/fommil/netlib-Java).[^1] `netlib-java` can use optimized native libraries for faster numerical processing. [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) and [OpenBLAS](http://www.openblas.net) are two most popular native linear algebra libraries (refered to as "native libraries" hereafter). +Spark MLlib defines Vector and Matrix as basic data types for machine learning algorithms. On top of them, [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) and [LAPACK](https://en.wikipedia.org/wiki/LAPACK) operations are implemented and supported by [netlib-java](https://github.com/fommil/netlib-Java).[^1] `netlib-java` can use optimized native linear algebra libraries (refered to as "native libraries" or "BLAS libraries" hereafter) for faster numerical processing. [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) and [OpenBLAS](http://www.openblas.net) are two most popular ones. -However due to license restrictions, the official released Spark binaries by default doesn't contain native libraries enabled for `netlib-java`. +However due to license restrictions, the official released Spark binaries by default doesn't contain native libraries support for `netlib-java`. -The following sections describe how to enable `netlib-java` with native libraries support, to install the native libraries and configure them properly. +The following sections describe how to enable `netlib-java` with native libraries support for Spark MLlib and how to install native libraries and configure them properly. [^1]: The algorithms may call Breeze and it will in turn call `netlib-java`. -## Enable `netlib-java` +## Enable `netlib-java` with native library proxies -To build Spark with `netlib-java` native library proxies, you need to add `-Pnetlib-lgpl` profile to Maven build command line. For example: +`netlib-java` native libraries has a dependency on `libgfortran`. It requires GFORTRAN 1.4 or above. This can be obtained by installing `libgfortran` package. After installation, the following command can be used to verify if it is installed properly. +``` +strings /path/to/libgfortran.so.3.0.0 | grep GFORTRAN_1.4 +``` + +To build Spark with `netlib-java` native library proxies, you need to add `-Pnetlib-lgpl` to Maven build command line. For example: ``` $SPARK_SOURCE_HOME/build/mvn -Pnetlib-lgpl -DskipTests -Pyarn -Phadoop-2.7 clean package ``` If you only want to enable it in your project, include `com.github.fommil.netlib:all:1.1.2` as a dependency of your project. +## Install Native Linear Algebra Libraries -and read the [netlib-java](https://github.com/fommil/netlib-java) documentation for your platform's additional installation instructions. - -## Install Native System Libraries - -Intel MKL and OpenBLAS are two most popular native linear algebra libraries, you can choose one of them based on your preference. We will describe how to install them for `netlib-java`. +Intel MKL and OpenBLAS are two most popular native linear algebra libraries, you can choose one of them based on your preference. We described basic instructions as below. You can refer to [netlib-java documentation](https://github.com/fommil/netlib-java) for more advanced installation instructions. ### Intel MKL -1. Download and install Intel MKL. The installation should be done on all nodes of the cluster. We assume the installation location is $MKLROOT. -2. Make sure `/usr/local/lib` is in system library search path and run the following commands: +- Download and install Intel MKL. The installation should be done on all nodes of the cluster. We assume the installation location is $MKLROOT. +- Make sure `/usr/local/lib` is in system library search path and run the following commands: ``` $ ln -sf $MKLROOT/lib/intel64/libmkl_rt.so /usr/local/lib/libblas.so.3 $ ln -sf $MKLROOT/lib/intel64/libmkl_rt.so /usr/local/lib/liblapack.so.3 @@ -39,7 +41,17 @@ $ ln -sf $MKLROOT/lib/intel64/libmkl_rt.so /usr/local/lib/liblapack.so.3 ### OpenBLAS -TBD +The installation should be done on all nodes of the cluster. Generic version of OpenBLAS are available with most distributions. You can install it with a distribution Package Manager (APT or YUM). + +For Debian / Ubuntu: +``` +sudo apt-get install libopenblas-base +sudo update-alternatives --config libblas.so.3 +``` +For CentOS / RHEL: +``` +sudo yum install openblas +``` ## Check if native libraries are enabled for MLlib @@ -49,25 +61,25 @@ scala> import com.github.fommil.netlib.BLAS; scala> System.out.println(BLAS.getInstance().getClass().getName()); ``` -If they arecorrectly loaded, it should print `com.github.fommil.netlib.NativeSystemBLAS`. Otherwise the warnings should be printed: +If they are correctly loaded, it should print `com.github.fommil.netlib.NativeSystemBLAS`. Otherwise the warnings should be printed: ``` WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeSystemBLAS WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeRefBLAS ``` -if native libraries are not properly configured in the system, Java implementation(f2jBLAS) will be used as fallback option. +If native libraries are not properly configured in the system, Java BLAS implementation(f2jBLAS) will be used as fallback option. ## Spark Configuration The use of multiple-threading in either Intel MKL or OpenBLAS can conflict with Spark's execution model.[^2] -Therefore configuring these native libraries to use a single thread for operations may actually improve performance (see [SPARK-21305](https://issues.apache.org/jira/browse/SPARK-21305)). It is usually optimal to match this to the number of `spark.task.cpus`, which is 1 by default and typically left at 1. +Therefore configuring these native libraries to use a single thread for operations may actually improve performance (see [SPARK-21305](https://issues.apache.org/jira/browse/SPARK-21305)). It is usually optimal to match this to the number of `spark.task.cpus`, which is `1` by default and typically left at `1`. -Use the options in `config/spark-env.sh` to disable multi-threading by setting thread number to 1. +You can use the options in `config/spark-env.sh` to disable multi-threading by setting thread number to 1. ``` # You might get better performance to enable these options if using native BLAS (see SPARK-21305). # - MKL_NUM_THREADS=1 Disable multi-threading of Intel MKL # - OPENBLAS_NUM_THREADS=1 Disable multi-threading of OpenBLAS ``` -[^2]: Please refer to resources like the following to understand how to configure the number of threads these BLAS implementations use: [Intel MKL](https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications) or [Intel oneMKL](https://software.intel.com/en-us/onemkl-linux-developer-guide-improving-performance-with-threading) and [OpenBLAS](https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded). +[^2]: Please refer to the following resources to understand how to configure the number of threads for these BLAS implementations: [Intel MKL](https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications) or [Intel oneMKL](https://software.intel.com/en-us/onemkl-linux-developer-guide-improving-performance-with-threading) and [OpenBLAS](https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded). From deba6508b90b99675615be473a8470077de4c735 Mon Sep 17 00:00:00 2001 From: "Wu, Xiaochang" Date: Mon, 20 Jul 2020 09:36:05 +0800 Subject: [PATCH 3/9] Update to resolve review comments --- docs/ml-guide.md | 2 +- docs/ml-linalg-guide.md | 48 ++++++++++++++++++++++++++++++----------- 2 files changed, 36 insertions(+), 14 deletions(-) diff --git a/docs/ml-guide.md b/docs/ml-guide.md index 806176734c8c..374fead8f2b0 100644 --- a/docs/ml-guide.md +++ b/docs/ml-guide.md @@ -64,7 +64,7 @@ The primary Machine Learning API for Spark is now the [DataFrame](sql-programmin MLlib uses linear algebra packages [Breeze](http://www.scalanlp.org/) and [netlib-java](https://github.com/fommil/netlib-java) for optimised numerical processing[^1]. Those packages may call native acceleration libraries such as [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) or [OpenBLAS](http://www.openblas.net) if they are available as system libraries or in runtime library paths. -Due to licensing issues with runtime proprietary binaries, we do not include `netlib-java`'s native proxies by default. See [MLlib Linear Algebra Acceleration Guide](ml-linalg-guide.md) for how to enable accelerated linear algebra processing. If accelerated native libraries are not enabled, you will see a warning message below and a pure JVM implementation will be used instead: +Due to differing OSS licenses, `netlib-java`'s native proxies can't be distributed with Spark. See [MLlib Linear Algebra Acceleration Guide](ml-linalg-guide.md) for how to enable accelerated linear algebra processing. If accelerated native libraries are not enabled, you will see a warning message below and a pure JVM implementation will be used instead: ``` WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeSystemBLAS WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeRefBLAS diff --git a/docs/ml-linalg-guide.md b/docs/ml-linalg-guide.md index ae083cc6f0a9..d9e167f5ea72 100644 --- a/docs/ml-linalg-guide.md +++ b/docs/ml-linalg-guide.md @@ -1,10 +1,29 @@ -# Spark MLlib Linear Algebra Acceleration Guide +--- +layout: global +title: MLlib Linear Algebra Acceleration Guide +displayTitle: MLlib Linear Algebra Acceleration Guide +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +# MLlib Linear Algebra Acceleration Guide ## Introduction This guide provides necessary information to enable accelerated linear algebra processing for Spark MLlib. -Spark MLlib defines Vector and Matrix as basic data types for machine learning algorithms. On top of them, [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) and [LAPACK](https://en.wikipedia.org/wiki/LAPACK) operations are implemented and supported by [netlib-java](https://github.com/fommil/netlib-Java).[^1] `netlib-java` can use optimized native linear algebra libraries (refered to as "native libraries" or "BLAS libraries" hereafter) for faster numerical processing. [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) and [OpenBLAS](http://www.openblas.net) are two most popular ones. +Spark MLlib defines Vector and Matrix as basic data types for machine learning algorithms. On top of them, [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) and [LAPACK](https://en.wikipedia.org/wiki/LAPACK) operations are implemented and supported by [netlib-java](https://github.com/fommil/netlib-Java) [^1]. `netlib-java` can use optimized native linear algebra libraries (refered to as "native libraries" or "BLAS libraries" hereafter) for faster numerical processing. [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) and [OpenBLAS](http://www.openblas.net) are two most popular ones. However due to license restrictions, the official released Spark binaries by default doesn't contain native libraries support for `netlib-java`. @@ -14,7 +33,7 @@ The following sections describe how to enable `netlib-java` with native librarie ## Enable `netlib-java` with native library proxies -`netlib-java` native libraries has a dependency on `libgfortran`. It requires GFORTRAN 1.4 or above. This can be obtained by installing `libgfortran` package. After installation, the following command can be used to verify if it is installed properly. +`netlib-java` depends on `libgfortran`. It requires GFORTRAN 1.4 or above. This can be obtained by installing `libgfortran` package. After installation, the following command can be used to verify if it is installed properly. ``` strings /path/to/libgfortran.so.3.0.0 | grep GFORTRAN_1.4 ``` @@ -26,14 +45,14 @@ $SPARK_SOURCE_HOME/build/mvn -Pnetlib-lgpl -DskipTests -Pyarn -Phadoop-2.7 clean If you only want to enable it in your project, include `com.github.fommil.netlib:all:1.1.2` as a dependency of your project. -## Install Native Linear Algebra Libraries +## Install native linear algebra libraries -Intel MKL and OpenBLAS are two most popular native linear algebra libraries, you can choose one of them based on your preference. We described basic instructions as below. You can refer to [netlib-java documentation](https://github.com/fommil/netlib-java) for more advanced installation instructions. +Intel MKL and OpenBLAS are two most popular native linear algebra libraries. You can choose one of them based on your preference. We provide basic instructions as below. You can refer to [netlib-java documentation](https://github.com/fommil/netlib-java) for more advanced installation instructions. ### Intel MKL -- Download and install Intel MKL. The installation should be done on all nodes of the cluster. We assume the installation location is $MKLROOT. -- Make sure `/usr/local/lib` is in system library search path and run the following commands: +- Download and install Intel MKL. The installation should be done on all nodes of the cluster. We assume the installation location is $MKLROOT (Eg. /opt/intel/mkl). +- Create soft links to `libmkl_rt.so` with specific names in system library search paths. For instance, make sure `/usr/local/lib` is in system library search paths and run the following commands: ``` $ ln -sf $MKLROOT/lib/intel64/libmkl_rt.so /usr/local/lib/libblas.so.3 $ ln -sf $MKLROOT/lib/intel64/libmkl_rt.so /usr/local/lib/liblapack.so.3 @@ -67,19 +86,22 @@ WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeSys WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeRefBLAS ``` -If native libraries are not properly configured in the system, Java BLAS implementation(f2jBLAS) will be used as fallback option. +If native libraries are not properly configured in the system, Java BLAS implementation (f2jBLAS) will be used as fallback option. ## Spark Configuration -The use of multiple-threading in either Intel MKL or OpenBLAS can conflict with Spark's execution model.[^2] +The default behavior of multi-threading in either Intel MKL or OpenBLAS may not be optimal with Spark's execution model [^2]. Therefore configuring these native libraries to use a single thread for operations may actually improve performance (see [SPARK-21305](https://issues.apache.org/jira/browse/SPARK-21305)). It is usually optimal to match this to the number of `spark.task.cpus`, which is `1` by default and typically left at `1`. -You can use the options in `config/spark-env.sh` to disable multi-threading by setting thread number to 1. +You can use the options in `config/spark-env.sh` to set thread number for Intel MKL or OpenBLAS: +* For Intel MKL: ``` -# You might get better performance to enable these options if using native BLAS (see SPARK-21305). -# - MKL_NUM_THREADS=1 Disable multi-threading of Intel MKL -# - OPENBLAS_NUM_THREADS=1 Disable multi-threading of OpenBLAS +MKL_NUM_THREADS=1 +``` +* For OpenBLAS: +``` +OPENBLAS_NUM_THREADS=1 ``` [^2]: Please refer to the following resources to understand how to configure the number of threads for these BLAS implementations: [Intel MKL](https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications) or [Intel oneMKL](https://software.intel.com/en-us/onemkl-linux-developer-guide-improving-performance-with-threading) and [OpenBLAS](https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded). From 4ccd7998bcb355981f476c5d4829180d5700bd6c Mon Sep 17 00:00:00 2001 From: Xiaochang Wu Date: Mon, 20 Jul 2020 09:41:10 +0800 Subject: [PATCH 4/9] Update ml-linalg-guide.md --- docs/ml-linalg-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ml-linalg-guide.md b/docs/ml-linalg-guide.md index d9e167f5ea72..46f03428e001 100644 --- a/docs/ml-linalg-guide.md +++ b/docs/ml-linalg-guide.md @@ -25,7 +25,7 @@ This guide provides necessary information to enable accelerated linear algebra p Spark MLlib defines Vector and Matrix as basic data types for machine learning algorithms. On top of them, [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) and [LAPACK](https://en.wikipedia.org/wiki/LAPACK) operations are implemented and supported by [netlib-java](https://github.com/fommil/netlib-Java) [^1]. `netlib-java` can use optimized native linear algebra libraries (refered to as "native libraries" or "BLAS libraries" hereafter) for faster numerical processing. [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) and [OpenBLAS](http://www.openblas.net) are two most popular ones. -However due to license restrictions, the official released Spark binaries by default doesn't contain native libraries support for `netlib-java`. +However due to license difference, the official released Spark binaries by default doesn't contain native libraries support for `netlib-java`. The following sections describe how to enable `netlib-java` with native libraries support for Spark MLlib and how to install native libraries and configure them properly. From 5bc118e9e3b4dd884db018e92e14afec3acd9895 Mon Sep 17 00:00:00 2001 From: Xiaochang Wu Date: Mon, 20 Jul 2020 14:14:59 +0800 Subject: [PATCH 5/9] Update ml-guide.md --- docs/ml-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ml-guide.md b/docs/ml-guide.md index 374fead8f2b0..41e9fd8c86ed 100644 --- a/docs/ml-guide.md +++ b/docs/ml-guide.md @@ -64,7 +64,7 @@ The primary Machine Learning API for Spark is now the [DataFrame](sql-programmin MLlib uses linear algebra packages [Breeze](http://www.scalanlp.org/) and [netlib-java](https://github.com/fommil/netlib-java) for optimised numerical processing[^1]. Those packages may call native acceleration libraries such as [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) or [OpenBLAS](http://www.openblas.net) if they are available as system libraries or in runtime library paths. -Due to differing OSS licenses, `netlib-java`'s native proxies can't be distributed with Spark. See [MLlib Linear Algebra Acceleration Guide](ml-linalg-guide.md) for how to enable accelerated linear algebra processing. If accelerated native libraries are not enabled, you will see a warning message below and a pure JVM implementation will be used instead: +Due to differing OSS licenses, `netlib-java`'s native proxies can't be distributed with Spark. See [MLlib Linear Algebra Acceleration Guide](ml-linalg-guide.html) for how to enable accelerated linear algebra processing. If accelerated native libraries are not enabled, you will see a warning message below and a pure JVM implementation will be used instead: ``` WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeSystemBLAS WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeRefBLAS From 8fd29c92162de7fa34767dafabee52575494f439 Mon Sep 17 00:00:00 2001 From: "Wu, Xiaochang" Date: Tue, 21 Jul 2020 09:27:14 +0800 Subject: [PATCH 6/9] Update ml-linalg-guide --- docs/ml-linalg-guide.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/ml-linalg-guide.md b/docs/ml-linalg-guide.md index 46f03428e001..4f5a5956bb2e 100644 --- a/docs/ml-linalg-guide.md +++ b/docs/ml-linalg-guide.md @@ -23,14 +23,12 @@ license: | This guide provides necessary information to enable accelerated linear algebra processing for Spark MLlib. -Spark MLlib defines Vector and Matrix as basic data types for machine learning algorithms. On top of them, [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) and [LAPACK](https://en.wikipedia.org/wiki/LAPACK) operations are implemented and supported by [netlib-java](https://github.com/fommil/netlib-Java) [^1]. `netlib-java` can use optimized native linear algebra libraries (refered to as "native libraries" or "BLAS libraries" hereafter) for faster numerical processing. [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) and [OpenBLAS](http://www.openblas.net) are two most popular ones. +Spark MLlib defines Vector and Matrix as basic data types for machine learning algorithms. On top of them, [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) and [LAPACK](https://en.wikipedia.org/wiki/LAPACK) operations are implemented and supported by [netlib-java](https://github.com/fommil/netlib-Java) (the algorithms may call [Breeze](https://github.com/scalanlp/breeze) and it will in turn call `netlib-java`). `netlib-java` can use optimized native linear algebra libraries (refered to as "native libraries" or "BLAS libraries" hereafter) for faster numerical processing. [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) and [OpenBLAS](http://www.openblas.net) are two most popular ones. -However due to license difference, the official released Spark binaries by default doesn't contain native libraries support for `netlib-java`. +However due to license differences, the official released Spark binaries by default don't contain native libraries support for `netlib-java`. The following sections describe how to enable `netlib-java` with native libraries support for Spark MLlib and how to install native libraries and configure them properly. -[^1]: The algorithms may call Breeze and it will in turn call `netlib-java`. - ## Enable `netlib-java` with native library proxies `netlib-java` depends on `libgfortran`. It requires GFORTRAN 1.4 or above. This can be obtained by installing `libgfortran` package. After installation, the following command can be used to verify if it is installed properly. @@ -47,11 +45,11 @@ If you only want to enable it in your project, include `com.github.fommil.netlib ## Install native linear algebra libraries -Intel MKL and OpenBLAS are two most popular native linear algebra libraries. You can choose one of them based on your preference. We provide basic instructions as below. You can refer to [netlib-java documentation](https://github.com/fommil/netlib-java) for more advanced installation instructions. +Intel MKL and OpenBLAS are two popular native linear algebra libraries. You can choose one of them based on your preference. We provide basic instructions as below. You can refer to [netlib-java documentation](https://github.com/fommil/netlib-java) for more advanced installation instructions. ### Intel MKL -- Download and install Intel MKL. The installation should be done on all nodes of the cluster. We assume the installation location is $MKLROOT (Eg. /opt/intel/mkl). +- Download and install Intel MKL. The installation should be done on all nodes of the cluster. We assume the installation location is $MKLROOT (e.g. /opt/intel/mkl). - Create soft links to `libmkl_rt.so` with specific names in system library search paths. For instance, make sure `/usr/local/lib` is in system library search paths and run the following commands: ``` $ ln -sf $MKLROOT/lib/intel64/libmkl_rt.so /usr/local/lib/libblas.so.3 @@ -60,7 +58,7 @@ $ ln -sf $MKLROOT/lib/intel64/libmkl_rt.so /usr/local/lib/liblapack.so.3 ### OpenBLAS -The installation should be done on all nodes of the cluster. Generic version of OpenBLAS are available with most distributions. You can install it with a distribution Package Manager (APT or YUM). +The installation should be done on all nodes of the cluster. Generic version of OpenBLAS are available with most distributions. You can install it with a distribution package manager `apt` or `yum`. For Debian / Ubuntu: ``` @@ -86,11 +84,11 @@ WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeSys WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeRefBLAS ``` -If native libraries are not properly configured in the system, Java BLAS implementation (f2jBLAS) will be used as fallback option. +If native libraries are not properly configured in the system, the Java implementation (f2jBLAS) will be used as fallback option. ## Spark Configuration -The default behavior of multi-threading in either Intel MKL or OpenBLAS may not be optimal with Spark's execution model [^2]. +The default behavior of multi-threading in either Intel MKL or OpenBLAS may not be optimal with Spark's execution model [^1]. Therefore configuring these native libraries to use a single thread for operations may actually improve performance (see [SPARK-21305](https://issues.apache.org/jira/browse/SPARK-21305)). It is usually optimal to match this to the number of `spark.task.cpus`, which is `1` by default and typically left at `1`. @@ -104,4 +102,4 @@ MKL_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 ``` -[^2]: Please refer to the following resources to understand how to configure the number of threads for these BLAS implementations: [Intel MKL](https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications) or [Intel oneMKL](https://software.intel.com/en-us/onemkl-linux-developer-guide-improving-performance-with-threading) and [OpenBLAS](https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded). +[^1]: Please refer to the following resources to understand how to configure the number of threads for these BLAS implementations: [Intel MKL](https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications) or [Intel oneMKL](https://software.intel.com/en-us/onemkl-linux-developer-guide-improving-performance-with-threading) and [OpenBLAS](https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded). From ca09f89972f738e5b36171cd0de44f4b56a172aa Mon Sep 17 00:00:00 2001 From: Xiaochang Wu Date: Wed, 22 Jul 2020 22:21:09 +0800 Subject: [PATCH 7/9] Update ml-guide.md --- docs/ml-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ml-guide.md b/docs/ml-guide.md index 41e9fd8c86ed..1b4a3e4eee55 100644 --- a/docs/ml-guide.md +++ b/docs/ml-guide.md @@ -64,7 +64,7 @@ The primary Machine Learning API for Spark is now the [DataFrame](sql-programmin MLlib uses linear algebra packages [Breeze](http://www.scalanlp.org/) and [netlib-java](https://github.com/fommil/netlib-java) for optimised numerical processing[^1]. Those packages may call native acceleration libraries such as [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) or [OpenBLAS](http://www.openblas.net) if they are available as system libraries or in runtime library paths. -Due to differing OSS licenses, `netlib-java`'s native proxies can't be distributed with Spark. See [MLlib Linear Algebra Acceleration Guide](ml-linalg-guide.html) for how to enable accelerated linear algebra processing. If accelerated native libraries are not enabled, you will see a warning message below and a pure JVM implementation will be used instead: +Due to differing OSS licenses, `netlib-java`'s native proxies can't be distributed with Spark. See [MLlib Linear Algebra Acceleration Guide](ml-linalg-guide.html) for how to enable accelerated linear algebra processing. If accelerated native libraries are not enabled, you will see a warning message like below and a pure JVM implementation will be used instead: ``` WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeSystemBLAS WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeRefBLAS From 50ada7deb21905a72e660f871fd4cb34dacce03b Mon Sep 17 00:00:00 2001 From: Xiaochang Wu Date: Wed, 22 Jul 2020 22:22:19 +0800 Subject: [PATCH 8/9] Update ml-linalg-guide.md --- docs/ml-linalg-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ml-linalg-guide.md b/docs/ml-linalg-guide.md index 4f5a5956bb2e..7438485c86cc 100644 --- a/docs/ml-linalg-guide.md +++ b/docs/ml-linalg-guide.md @@ -58,7 +58,7 @@ $ ln -sf $MKLROOT/lib/intel64/libmkl_rt.so /usr/local/lib/liblapack.so.3 ### OpenBLAS -The installation should be done on all nodes of the cluster. Generic version of OpenBLAS are available with most distributions. You can install it with a distribution package manager `apt` or `yum`. +The installation should be done on all nodes of the cluster. Generic version of OpenBLAS are available with most distributions. You can install it with a distribution package manager like `apt` or `yum`. For Debian / Ubuntu: ``` From 432c3c197df7feba172520d6cc03bf40dd1ef405 Mon Sep 17 00:00:00 2001 From: Xiaochang Wu Date: Mon, 27 Jul 2020 09:19:46 +0800 Subject: [PATCH 9/9] Update ml-linalg-guide.md --- docs/ml-linalg-guide.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/ml-linalg-guide.md b/docs/ml-linalg-guide.md index 7438485c86cc..739091363473 100644 --- a/docs/ml-linalg-guide.md +++ b/docs/ml-linalg-guide.md @@ -17,13 +17,11 @@ license: | limitations under the License. --- -# MLlib Linear Algebra Acceleration Guide - ## Introduction This guide provides necessary information to enable accelerated linear algebra processing for Spark MLlib. -Spark MLlib defines Vector and Matrix as basic data types for machine learning algorithms. On top of them, [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) and [LAPACK](https://en.wikipedia.org/wiki/LAPACK) operations are implemented and supported by [netlib-java](https://github.com/fommil/netlib-Java) (the algorithms may call [Breeze](https://github.com/scalanlp/breeze) and it will in turn call `netlib-java`). `netlib-java` can use optimized native linear algebra libraries (refered to as "native libraries" or "BLAS libraries" hereafter) for faster numerical processing. [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) and [OpenBLAS](http://www.openblas.net) are two most popular ones. +Spark MLlib defines Vector and Matrix as basic data types for machine learning algorithms. On top of them, [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) and [LAPACK](https://en.wikipedia.org/wiki/LAPACK) operations are implemented and supported by [netlib-java](https://github.com/fommil/netlib-Java) (the algorithms may call [Breeze](https://github.com/scalanlp/breeze) and it will in turn call `netlib-java`). `netlib-java` can use optimized native linear algebra libraries (refered to as "native libraries" or "BLAS libraries" hereafter) for faster numerical processing. [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) and [OpenBLAS](http://www.openblas.net) are two popular ones. However due to license differences, the official released Spark binaries by default don't contain native libraries support for `netlib-java`. @@ -72,7 +70,7 @@ sudo yum install openblas ## Check if native libraries are enabled for MLlib -To verify native libraries are properly loaded, start `spark-shell` and run the following code +To verify native libraries are properly loaded, start `spark-shell` and run the following code: ``` scala> import com.github.fommil.netlib.BLAS; scala> System.out.println(BLAS.getInstance().getClass().getName());