Merge pull request #614 from JohnSnowLabs/221-release-candidate

2.2.1 Release Candidate

Author: saif-ellafi

CHANGELOG

@@ -1,3 +1,24 @@
+========
+2.2.1
+========
+---------------
+Overview
+---------------
+This short release is to address a few uncovered issues in the previous 2.2.0 release. Thank you all for quick feedback.
+
+---------------
+Enhancements
+---------------
+* NerDLApproach new param includeValidationProp allows partitioning the training set and exclude a fraction
+* NerDLApproach trainValidationProp now randomly samples the data as opposed to head first
+
+---------------
+Bugfixes
+---------------
+* Fixed a bug in ResourceHelper causing folder resources to fail when a folder is empty (affects various annotators)
+* Fixed a bug in python embeddings format not parsed to upper case
+* Fixed a bug in python causing an incapability to load PipelineModels after loading embeddings
+
 ========
 2.2.0
 ========

README.md

@@ -42,7 +42,7 @@ Take a look at our official Spark NLP page: [http://nlp.johnsnowlabs.com/](http:
 
 ## Apache Spark Support
 
-Spark NLP *2.2.0* has been built on top of Apache Spark 2.4.3
+Spark NLP *2.2.1* has been built on top of Apache Spark 2.4.3
 
 | Spark NLP   |   Spark 2.3.x         | Spark 2.4    |
 |-------------|-------------------------------------|--------------|
@@ -68,18 +68,18 @@ This library has been uploaded to the [spark-packages repository](https://spark-
 
 Benefit of spark-packages is that makes it available for both Scala-Java and Python
 
-To use the most recent version just add the `--packages JohnSnowLabs:spark-nlp:2.2.0` to you spark command
+To use the most recent version just add the `--packages JohnSnowLabs:spark-nlp:2.2.1` to you spark command
 
 ```sh
-spark-shell --packages JohnSnowLabs:spark-nlp:2.2.0
+spark-shell --packages JohnSnowLabs:spark-nlp:2.2.1
 ```
 
 ```sh
-pyspark --packages JohnSnowLabs:spark-nlp:2.2.0
+pyspark --packages JohnSnowLabs:spark-nlp:2.2.1
 ```
 
 ```sh
-spark-submit --packages JohnSnowLabs:spark-nlp:2.2.0
+spark-submit --packages JohnSnowLabs:spark-nlp:2.2.1
 ```
 
 This can also be used to create a SparkSession manually by using the `spark.jars.packages` option in both Python and Scala
@@ -147,7 +147,7 @@ Our package is deployed to maven central. In order to add this package as a depe
 <dependency>
     <groupId>com.johnsnowlabs.nlp</groupId>
     <artifactId>spark-nlp_2.11</artifactId>
-    <version>2.2.0</version>
+    <version>2.2.1</version>
 </dependency>
 ```
 
@@ -158,22 +158,22 @@ and
 <dependency>
     <groupId>com.johnsnowlabs.nlp</groupId>
     <artifactId>spark-nlp-ocr_2.11</artifactId>
-    <version>2.2.0</version>
+    <version>2.2.1</version>
 </dependency>
 ```
 
 ### SBT
 
 ```sbtshell
 // https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp
-libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp" % "2.2.0"
+libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp" % "2.2.1"
 ```
 
 and
 
 ```sbtshell
 // https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-ocr
-libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-ocr" % "2.2.0"
+libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-ocr" % "2.2.1"
 ```
 
 Maven Central: [https://mvnrepository.com/artifact/com.johnsnowlabs.nlp](https://mvnrepository.com/artifact/com.johnsnowlabs.nlp)
@@ -189,7 +189,7 @@ If you installed pyspark through pip/conda, you can install `spark-nlp` through
 Pip:
 
 ```bash
-pip install spark-nlp==2.2.0
+pip install spark-nlp==2.2.1
 ```
 
 Conda:
@@ -216,7 +216,7 @@ spark = SparkSession.builder \
     .master("local[4]")\
     .config("spark.driver.memory","8G")\
     .config("spark.driver.maxResultSize", "2G") \
-    .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.0")\
+    .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.1")\
     .config("spark.kryoserializer.buffer.max", "500m")\
     .getOrCreate()
 ```
@@ -249,7 +249,7 @@ Use either one of the following options
 * Add the following Maven Coordinates to the interpreter's library list
 
 ```bash
-com.johnsnowlabs.nlp:spark-nlp_2.11:2.2.0
+com.johnsnowlabs.nlp:spark-nlp_2.11:2.2.1
 ```
 
 * Add path to pre-built jar from [here](#pre-compiled-spark-nlp-and-spark-nlp-ocr) in the interpreter's library list making sure the jar is available to driver path
@@ -259,7 +259,7 @@ com.johnsnowlabs.nlp:spark-nlp_2.11:2.2.0
 Apart from previous step, install python module through pip
 
 ```bash
-pip install spark-nlp==2.2.0
+pip install spark-nlp==2.2.1
 ```
 
 Or you can install `spark-nlp` from inside Zeppelin by using Conda:
@@ -284,7 +284,7 @@ export PYSPARK_PYTHON=python3
 export PYSPARK_DRIVER_PYTHON=jupyter
 export PYSPARK_DRIVER_PYTHON_OPTS=notebook
 
-pyspark --packages JohnSnowLabs:spark-nlp:2.2.0
+pyspark --packages JohnSnowLabs:spark-nlp:2.2.1
 ```
 
 Alternatively, you can mix in using `--jars` option for pyspark + `pip install spark-nlp`
@@ -346,7 +346,7 @@ To include the OCR submodule in Spark NLP, you will need to add the following to
 
 ```basg
 --repositories http://repo.spring.io/plugins-release
---packages JohnSnowLabs:spark-nlp:2.2.0,com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:2.2.0,javax.media.jai:com.springsource.javax.media.jai.core:1.1.3
+--packages JohnSnowLabs:spark-nlp:2.2.1,com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:2.2.1,javax.media.jai:com.springsource.javax.media.jai.core:1.1.3
 ```
 
 This way you will download the extra dependencies needed by our OCR submodule. The Python SparkSession equivalent is
@@ -358,7 +358,7 @@ spark = SparkSession.builder \
     .config("spark.driver.memory", "6g") \
     .config("spark.executor.memory", "6g") \
     .config("spark.jars.repositories", "http://repo.spring.io/plugins-release") \
-    .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.0,com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:2.2.0,javax.media.jai:com.springsource.javax.media.jai.core:1.1.3") \
+    .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.1,com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:2.2.1,javax.media.jai:com.springsource.javax.media.jai.core:1.1.3") \
     .getOrCreate()
 ```
 

build.sbt

@@ -16,7 +16,7 @@ if(is_gpu.equals("false")){
 
 organization:= "com.johnsnowlabs.nlp"
 
-version := "2.2.0"
+version := "2.2.1"
 
 scalaVersion in ThisBuild := scalaVer
 
@@ -200,7 +200,7 @@ assemblyMergeStrategy in assembly := {
 lazy val evaluation = (project in file("eval"))
   .settings(
     name := "spark-nlp-eval",
-    version := "2.2.0",
+    version := "2.2.1",
 
     assemblyMergeStrategy in assembly := evalMergeRules,
 
@@ -241,7 +241,7 @@ lazy val evaluation = (project in file("eval"))
 lazy val ocr = (project in file("ocr"))
   .settings(
     name := "spark-nlp-ocr",
-    version := "2.2.0",
+    version := "2.2.1",
 
     test in assembly := {},
 

docs/_layouts/landing.html

@@ -49,22 +49,22 @@ <h1>{{ _section.title }}</h1>
           <div class="cell cell--12 cell--lg-12" style="text-align: left; background-color: #2d2d2d; padding: 10px">
             {% highlight bash %}
 # Install Spark NLP from PyPI
-$ pip install spark-nlp==2.2.0
+$ pip install spark-nlp==2.2.1
 
 # Install Spark NLP from Anacodna/Conda
 $ conda install -c johnsnowlabs spark-nlp
 
 # Load Spark NLP with Spark Shell
-$ spark-shell --packages JohnSnowLabs:spark-nlp:2.2.0
+$ spark-shell --packages JohnSnowLabs:spark-nlp:2.2.1
 
 # Load Spark NLP with PySpark
-$ pyspark --packages JohnSnowLabs:spark-nlp:2.2.0
+$ pyspark --packages JohnSnowLabs:spark-nlp:2.2.1
 
 # Load Spark NLP with Spark Submit
-$ spark-submit --packages JohnSnowLabs:spark-nlp:2.2.0
+$ spark-submit --packages JohnSnowLabs:spark-nlp:2.2.1
 
 # Load Spark NLP as external JAR after comiling and bulding Spark NLP by `sbt assembly`
-$ spark-shell --jar spark-nlp-assembly-2.2.0
+$ spark-shell --jar spark-nlp-assembly-2.2.1
             {% endhighlight %}
           </div>
         </div>

docs/en/install.md

@@ -13,7 +13,7 @@ modify_date: "2019-05-16"
 If you installed pyspark through pip, you can install `spark-nlp` through pip as well.
 
 ```bash
-pip install spark-nlp==2.2.0
+pip install spark-nlp==2.2.1
 ```
 
 PyPI [spark-nlp package](https://pypi.org/project/spark-nlp/)
@@ -36,7 +36,7 @@ spark = SparkSession.builder \
     .master("local[*]")\
     .config("spark.driver.memory","8G")\
     .config("spark.driver.maxResultSize", "2G") \
-    .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.0")\
+    .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.1")\
     .config("spark.kryoserializer.buffer.max", "500m")\
     .getOrCreate()
 ```
@@ -97,7 +97,7 @@ Our package is deployed to maven central. In order to add this package as a depe
 <dependency>
     <groupId>com.johnsnowlabs.nlp</groupId>
     <artifactId>spark-nlp_2.11</artifactId>
-    <version>2.2.0</version>
+    <version>2.2.1</version>
 </dependency>
 ```
 
@@ -108,22 +108,22 @@ and
 <dependency>
     <groupId>com.johnsnowlabs.nlp</groupId>
     <artifactId>spark-nlp-ocr_2.11</artifactId>
-    <version>2.2.0</version>
+    <version>2.2.1</version>
 </dependency>
 ```
 
 ### SBT
 
 ```sbtshell
 // https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp
-libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp" % "2.2.0"
+libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp" % "2.2.1"
 ```
 
 and
 
 ```sbtshell
 // https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-ocr
-libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-ocr" % "2.2.0"
+libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-ocr" % "2.2.1"
 ```
 
 Maven Central: [https://mvnrepository.com/artifact/com.johnsnowlabs.nlp](https://mvnrepository.com/artifact/com.johnsnowlabs.nlp)
@@ -151,7 +151,7 @@ Note: You can import these notebooks by using their URLs.
 4- From the Source drop-down menu, select **Maven Coordinate:**
 ![Databricks](https://databricks.com/wp-content/uploads/2015/07/select-maven-1024x711.png)
 
-5- Now, all available **Spark Packages** are at your fingertips! Just search for **JohnSnowLabs:spark-nlp:version** where **version** stands for the library version such as: `1.8.4` or `2.2.0`
+5- Now, all available **Spark Packages** are at your fingertips! Just search for **JohnSnowLabs:spark-nlp:version** where **version** stands for the library version such as: `1.8.4` or `2.2.1`
 ![Databricks](https://databricks.com/wp-content/uploads/2015/07/browser-1024x548.png)
 
 6- Select **spark-nlp** package and we are good to go!

docs/en/quickstart.md

@@ -29,17 +29,17 @@ Spark NLP is built on top of **Apache Spark 2.4.0** and such is the **only** sup
 To start using the library, execute any of the following lines depending on your desired use case:
 
 ```bash
-spark-shell --packages JohnSnowLabs:spark-nlp:2.2.0
-pyspark --packages JohnSnowLabs:spark-nlp:2.2.0
-spark-submit --packages JohnSnowLabs:spark-nlp:2.2.0
+spark-shell --packages JohnSnowLabs:spark-nlp:2.2.1
+pyspark --packages JohnSnowLabs:spark-nlp:2.2.1
+spark-submit --packages JohnSnowLabs:spark-nlp:2.2.1
 ```
 
 ### **Straight forward Python on jupyter notebook**
 
 Use pip to install (after you pip installed numpy and pyspark)
 
 ```bash
-pip install spark-nlp==2.2.0
+pip install spark-nlp==2.2.1
 jupyter notebook
 ```
 
@@ -60,7 +60,7 @@ spark = SparkSession.builder \
     .appName('OCR Eval') \
     .config("spark.driver.memory", "6g") \
     .config("spark.executor.memory", "6g") \
-    .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.0") \
+    .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.1") \
     .getOrCreate()
 ```
 
@@ -69,13 +69,13 @@ spark = SparkSession.builder \
 Add the following maven coordinates in the dependency configuration page:
 
 ```bash
-com.johnsnowlabs.nlp:spark-nlp_2.11:2.2.0
+com.johnsnowlabs.nlp:spark-nlp_2.11:2.2.1
 ```
 
 For Python in **Apache Zeppelin** you may need to setup _**SPARK_SUBMIT_OPTIONS**_ utilizing --packages instruction shown above like this
 
 ```bash
-export SPARK_SUBMIT_OPTIONS="--packages JohnSnowLabs:spark-nlp:2.2.0"
+export SPARK_SUBMIT_OPTIONS="--packages JohnSnowLabs:spark-nlp:2.2.1"
 ```
 
 ### **Python Jupyter Notebook with PySpark**
@@ -85,7 +85,7 @@ export SPARK_HOME=/path/to/your/spark/folder
 export PYSPARK_DRIVER_PYTHON=jupyter
 export PYSPARK_DRIVER_PYTHON_OPTS=notebook
 
-pyspark --packages JohnSnowLabs:spark-nlp:2.2.0
+pyspark --packages JohnSnowLabs:spark-nlp:2.2.1
 ```
 
 ### S3 based standalone cluster (No Hadoop)
@@ -297,7 +297,7 @@ lightPipeline.annotate("Hello world, please annotate my text")
 Spark NLP OCR Module is not included within Spark NLP. It is not an annotator and not an extension to Spark ML. You can include it with the following coordinates for Maven:
 
 ```bash
-com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:2.2.0
+com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:2.2.1
 ```
 
 ### Creating Spark datasets from PDF (To be used with Spark NLP)

python/setup.py

@@ -40,7 +40,7 @@
     # For a discussion on single-sourcing the version across setup.py and the
     # project code, see
     # https://packaging.python.org/en/latest/single_source_version.html
-    version='2.2.0',  # Required
+    version='2.2.1',  # Required
 
     # This is a one-line description or tagline of what your project does. This
     # corresponds to the "Summary" metadata field:

python/sparknlp/__init__.py

@@ -41,14 +41,14 @@ def start(include_ocr=False):
 
     if include_ocr:
         builder \
-            .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.0,com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:2.2.0,javax.media.jai:com.springsource.javax.media.jai.core:1.1.3") \
+            .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.1,com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:2.2.1,javax.media.jai:com.springsource.javax.media.jai.core:1.1.3") \
             .config("spark.jars.repositories", "http://repo.spring.io/plugins-release")
 
     else:
-        builder.config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.0") \
+        builder.config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.1") \
 
     return builder.getOrCreate()
 
 
 def version():
-    print('2.2.0')
+    print('2.2.1')

python/sparknlp/internal.py

@@ -22,7 +22,7 @@ def new_java_obj(self, java_class, *args):
 
     def new_java_array(self, pylist, java_class):
         """
-        ToDo: Inspired from spark 2.2.0. Delete if we upgrade
+        ToDo: Inspired from spark 2.0. Review if spark changes
         """
         java_array = self.sc._gateway.new_array(java_class, len(pylist))
         for i in range(len(pylist)):

src/main/scala/com/johnsnowlabs/nlp/SparkNLP.scala

@@ -4,7 +4,7 @@ import org.apache.spark.sql.SparkSession
 
 object SparkNLP {
 
-  val currentVersion = "2.2.0"
+  val currentVersion = "2.2.1"
 
   def start(includeOcr: Boolean = false): SparkSession = {
     val build = SparkSession.builder()
@@ -15,11 +15,11 @@ object SparkNLP {
 
     if (includeOcr) {
       build
-        .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.0,com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:2.2.0,javax.media.jai:com.springsource.javax.media.jai.core:1.1.3")
+        .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.1,com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:2.2.1,javax.media.jai:com.springsource.javax.media.jai.core:1.1.3")
         .config("spark.jars.repositories", "http://repo.spring.io/plugins-release")
     } else {
       build
-        .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.0")
+        .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.1")
     }
 
     build.getOrCreate()