Support dill 0.3.6 (#5166)

* Remove fix on dill >= 0.3.6

* Unpin dill

* Fix dill save_function for dill 0.3.6

* Fix dill save_code for dill 0.3.6

* Pin dill<0.3.7 to support it explicitly in their next release

* Fix CI to test latest dill

* Fix style

* Fix dill _save_regex for dill 0.3.6

* Fix _save_regex

* Uninstall apache-beam for CI latest dependencies

* Fix uninstall of apache-beam

* Remove raising error when pickling logger

* Use pickable MagicMock when patching

* Fix style

Author: albertvillanova

.github/workflows/ci.yml

@@ -64,10 +64,12 @@ jobs:
           pip install -r additional-tests-requirements.txt --no-deps
       - name: Install dependencies (latest versions)
         if: ${{ matrix.deps_versions == 'latest' }}
-        run: pip install --upgrade pyarrow huggingface-hub
-      - name: Install depencencies (minimum versions)
+        run: |
+          pip uninstall -y apache-beam
+          pip install --upgrade pyarrow huggingface-hub dill
+      - name: Install dependencies (minimum versions)
         if: ${{ matrix.deps_versions != 'latest' }}
-        run: pip install pyarrow==6.0.1 huggingface-hub==0.2.0 transformers
+        run: pip install pyarrow==6.0.1 huggingface-hub==0.2.0 transformers dill==0.3.1.1
       - name: Test with pytest
         run: |
           python -m pytest -rfExX -m ${{ matrix.test }} -n 2 --dist loadfile -sv ./tests/

setup.py

@@ -70,7 +70,7 @@
     # Minimum 6.0.0 to support wrap_array which is needed for ArrayND features
     "pyarrow>=6.0.0",
     # For smart caching dataset processing
-    "dill<0.3.6",  # tmp pin until 0.3.6 release: see https://github.com/huggingface/datasets/pull/4397
+    "dill<0.3.7",  # tmp pin until next 0.3.7 release: see https://github.com/huggingface/datasets/pull/5166
     # For performance gains with apache arrow
     "pandas",
     # for downloading datasets over HTTPS