Skip to content

Conversation

@HyukjinKwon
Copy link
Member

What changes were proposed in this pull request?

Currently, some of PySpark tests sill assume the tests could be ran in Python 2.6 by importing unittest2. For instance:

if sys.version_info[:2] <= (2, 6):
    try:
        import unittest2 as unittest
    except ImportError:
        sys.stderr.write('Please install unittest2 to test with Python 2.6 or earlier')
        sys.exit(1)
else:
    import unittest

While I am here, I removed some of unused imports and reordered imports per PEP 8.

We officially dropped Python 2.6 support a while ago and started to discuss about Python 2 drop. It's better to remove them out.

How was this patch tested?

Manually tests, and existing tests via Jenkins.

@HyukjinKwon
Copy link
Member Author

cc @BryanCutler.

BTW, Bryan, do you have some time to work on the has_numpy stuff that we talked about before? I was thinking we should specify the package version and produce similar skip messages like we do in Padnas and Arrow:

pandas_requirement_message = None
try:
from pyspark.sql.utils import require_minimum_pandas_version
require_minimum_pandas_version()
except ImportError as e:
# If Pandas version requirement is not satisfied, skip related tests.
pandas_requirement_message = _exception_message(e)
pyarrow_requirement_message = None
try:
from pyspark.sql.utils import require_minimum_pyarrow_version
require_minimum_pyarrow_version()
except ImportError as e:
# If Arrow version requirement is not satisfied, skip related tests.
pyarrow_requirement_message = _exception_message(e)
test_not_compiled_message = None
try:
from pyspark.sql.utils import require_test_compiled
require_test_compiled()
except Exception as e:
test_not_compiled_message = _exception_message(e)
have_pandas = pandas_requirement_message is None
have_pyarrow = pyarrow_requirement_message is None
test_compiled = test_not_compiled_message is None

I'll try to work on this towards the end of the next week if you're busy.

@SparkQA
Copy link

SparkQA commented Nov 18, 2018

Test build #98973 has finished for PR 23077 at commit ed834f5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

@BryanCutler BryanCutler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @HyukjinKwon !

@BryanCutler
Copy link
Member

Oops, actually I think there is one more here https://github.com/apache/spark/blob/master/python/pyspark/testing/mllibutils.py#L20

Other than that, looks good

@BryanCutler
Copy link
Member

BTW, Bryan, do you have some time to work on the has_numpy stuff

Yup, I can do that

@BryanCutler
Copy link
Member

Oh, I think the PR title should be SPARK-26105 too

@SparkQA
Copy link

SparkQA commented Nov 19, 2018

Test build #98985 has finished for PR 23077 at commit a188076.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon HyukjinKwon changed the title [SPARK-25344][PYTHON] Clean unittest2 imports up that were added for Python 2.6 before [SPARK-26105][PYTHON] Clean unittest2 imports up that were added for Python 2.6 before Nov 19, 2018
@HyukjinKwon
Copy link
Member Author

Merged to master.

Thanks for reviewing this, @BryanCutler and @srowen.

@asfgit asfgit closed this in bbbdaa8 Nov 19, 2018
jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
…Python 2.6 before

## What changes were proposed in this pull request?

Currently, some of PySpark tests sill assume the tests could be ran in Python 2.6 by importing `unittest2`. For instance:

```python
if sys.version_info[:2] <= (2, 6):
    try:
        import unittest2 as unittest
    except ImportError:
        sys.stderr.write('Please install unittest2 to test with Python 2.6 or earlier')
        sys.exit(1)
else:
    import unittest
```

While I am here, I removed some of unused imports and reordered imports per PEP 8.

We officially dropped Python 2.6 support a while ago and started to discuss about Python 2 drop. It's better to remove them out.

## How was this patch tested?

Manually tests, and existing tests via Jenkins.

Closes apache#23077 from HyukjinKwon/SPARK-26105.

Lead-authored-by: hyukjinkwon <[email protected]>
Co-authored-by: Bryan Cutler <[email protected]>
Signed-off-by: hyukjinkwon <[email protected]>
@HyukjinKwon HyukjinKwon deleted the SPARK-26105 branch March 3, 2020 01:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants