Skip to content

[SPARK-21045][PYSPARK]Fixed executor blocked because traceback.format_exc throw UnicodeDecodeError #18262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dataknocker wants to merge 52 commits into from
Closed

[SPARK-21045][PYSPARK]Fixed executor blocked because traceback.format_exc throw UnicodeDecodeError #18262

Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
81f5db5
Fixed executor blocked because traceback.format_exc throw UnicodeDeco…
Jun 10, 2017
9893efd
check by python version
Jun 13, 2017
ee793ec
[SPARK-20211][SQL] Fix the Precision and Scale of Decimal Values when…
gatorsmile Jun 10, 2017
cdb26eb
[SPARK-20620][TEST] Improve some unit tests for NullExpressionsSuite …
10110346 Jun 10, 2017
0f2da15
[SPARK-20877][SPARKR] refactor tests to basic tests only for CRAN
felixcheung Jun 11, 2017
e5940a0
[SPARK-21000][MESOS] Add Mesos labels support to the Spark Dispatcher
Jun 11, 2017
ef661f6
[SPARK-20935][STREAMING] Always close WriteAheadLog and make it idemp…
HyukjinKwon Jun 11, 2017
ebb284f
[SPARK-13933][BUILD] Update hadoop-2.7 profile's curator version to 2…
wangyum Jun 11, 2017
d1dde23
[SPARK-20877][SPARKR][FOLLOWUP] clean up after test move
felixcheung Jun 11, 2017
160cbe7
Fixed typo in sql.functions
Jun 11, 2017
9713c7c
[SPARK-21031][SQL] Add `alterTableStats` to store spark's stats and l…
wzhfy Jun 12, 2017
d262b77
[SPARK-18891][SQL] Support for Scala Map collection types
michalsenkyr Jun 12, 2017
9bbd3ca
[SPARK-18891][SQL] Support for specific Java List subtypes
michalsenkyr Jun 12, 2017
ae461e9
[SPARK-20715] Store MapStatuses only in MapOutputTracker, not Shuffle…
JoshRosen Jun 12, 2017
1b61edd
[SPARK-20665][SQL][FOLLOW-UP] Move test case to MathExpressionsSuite
10110346 Jun 12, 2017
c8c8ba8
[DOCS] Fix error: ambiguous reference to overloaded definition
ZiyueHuang Jun 12, 2017
f02667c
[SPARK-21041][SQL] SparkSession.range should be consistent with Spark…
dongjoon-hyun Jun 12, 2017
b9d80e8
[SPARK-21046][SQL] simplify the array offset and length in ColumnVector
cloud-fan Jun 12, 2017
11755f1
[SPARK-17914][SQL] Fix parsing of timestamp strings with nanoseconds
Jun 12, 2017
160fcdb
[SPARK-20345][SQL] Fix STS error handling logic on HiveSQLException
dongjoon-hyun Jun 12, 2017
26f056c
[SPARK-21059][SQL] LikeSimplification can NPE on null pattern
rxin Jun 12, 2017
3cd45c8
[SPARK-21050][ML] Word2vec persistence overflow bug fix
jkbradley Jun 12, 2017
66bd772
[SPARK-20979][SS] Add RateSource to generate values for tests and ben…
zsxwing Jun 12, 2017
cb8a343
Revert "[SPARK-21046][SQL] simplify the array offset and length in Co…
cloud-fan Jun 13, 2017
a071d75
[SPARK-19910][SQL] `stack` should not reject NULL values due to type …
dongjoon-hyun Jun 13, 2017
a01e076
[TEST][SPARKR][CORE] Fix broken SparkSubmitSuite
felixcheung Jun 13, 2017
0d248cf
[SPARK-20920][SQL] ForkJoinPool pools are leaked when writing hive ta…
srowen Jun 13, 2017
8dcca70
[SPARK-21006][TESTS][FOLLOW-UP] Some Worker's RpcEnv is leaked in Wor…
10110346 Jun 13, 2017
f163e86
[SPARK-21039][SPARK CORE] Use treeAggregate instead of aggregate in D…
rishabhbhardwaj Jun 13, 2017
5621e4a
[SPARK-21060][WEB-UI] Css style about paging function is error in the…
Jun 13, 2017
df01660
[SPARK-21064][CORE][TEST] Fix the default value bug in NettyBlockTran…
Jun 13, 2017
bf139ca
[SPARK-21051][SQL] Add hash map metrics to aggregate
viirya Jun 13, 2017
60bc9ff
[SPARK-21016][CORE] Improve code fault tolerance for converting strin…
10110346 Jun 13, 2017
fb1a870
[SPARK-12552][CORE] Correctly count the driver resource when recoveri…
jerryshao Jun 14, 2017
f71145a
[SPARK-20986][SQL] Reset table's statistics after PruneFileSourcePart…
lianhuiwang Jun 14, 2017
1f71f40
[SPARK-19753][CORE] Un-register all shuffle output on a host in case …
Jun 14, 2017
6d097f7
[SPARK-20754][SQL][FOLLOWUP] Add Function Alias For MOD/POSITION.
wangyum Jun 14, 2017
c19ca9f
[SPARK-21057][ML] Do not use a PascalDistribution in countApprox
srowen Jun 14, 2017
12e068d
[SPARK-21085][SQL] Failed to read the partitioned table created by Sp…
gatorsmile Jun 14, 2017
5a54d6b
[SPARK-21089][SQL] Fix DESC EXTENDED/FORMATTED to Show Table Properties
gatorsmile Jun 14, 2017
7686f02
Revert "[SPARK-20941][SQL] Fix SubqueryExec Reuse"
gatorsmile Jun 14, 2017
24b409d
[SPARK-21091][SQL] Move constraint code into QueryPlanConstraints
rxin Jun 14, 2017
3c1f793
[SPARK-19900][CORE] Remove driver when relaunching.
lycplus Jun 15, 2017
b897c60
[SPARK-21092][SQL] Wire SQLConf in logical plan and expressions
rxin Jun 15, 2017
fb8bcf1
[SPARK-20980][SQL] Rename `wholeFile` to `multiLine` for both CSV and…
gatorsmile Jun 15, 2017
73dae47
[SPARK-18016][SQL][CATALYST] Code Generation: Constant Pool Limit - C…
Jun 15, 2017
b89d66a
[SPARK-20980][DOCS] update doc to reflect multiLine change
felixcheung Jun 15, 2017
fea7e71
[SPARK-16251][SPARK-20200][CORE][TEST] Flaky test: org.apache.spark.r…
jiangxb1987 Jun 15, 2017
6e74008
[SPARK-20434][YARN][CORE] Move Hadoop delegation token code from yarn…
Jun 15, 2017
3c77234
[SPARK-21112][SQL] ALTER TABLE SET TBLPROPERTIES should not overwrite…
gatorsmile Jun 16, 2017
6bbd09c
[SPARK-21072][SQL] TreeNode.mapChildren should only apply to the chil…
ConeyLiu Jun 16, 2017
8120192
add test for SPARK-21045
Jun 16, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion python/pyspark/worker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,8 +177,13 @@ def process():
process()
except Exception:
try:
exc_info = traceback.format_exc()
try:
exc_info = exc_info.encode("utf-8")
Copy link
Member

@HyukjinKwon HyukjinKwon Jun 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should not just rely on the exception handling but only need to encode on unicode in Python 2 and str in Python 3.

Copy link
Author

@dataknocker dataknocker Jun 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Python 2, traceback.format_exc() is already str, don't need to encode. So can I change like this:


    exc_info = traceback.format_exc()
    if sys.version > '3':
        exc_info = exc_info.encode("utf-8")
    write_int(SpecialLengths.PYTHON_EXCEPTION_THROWN, outfile)
    write_with_length(exc_info, outfile)

Copy link
Member

@HyukjinKwon HyukjinKwon Jun 13, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather go for something like the one below:

if sys.version >= '3':
    unicode = str
...
if isinstance(exc_info, unicode):
    exc_info = exc_info.encode('utf-8'))
else:
    exc_info = exc_info.str(desc)

for sure (at least for sanity check). I believe this approach is rather more common.

Copy link
Author

@dataknocker dataknocker Jun 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benefit a lot, thx.

exc_info = exc_info.str(desc) is right? I think I can remove it.

if sys.version >= '3':
    unicode = str
if isinstance(exc_info, unicode):
    exc_info = exc_info.encode('utf-8'))

Copy link
Member

@HyukjinKwon HyukjinKwon Jun 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, we could refer https://github.com/apache/spark/pull/17267.

except:
pass
write_int(SpecialLengths.PYTHON_EXCEPTION_THROWN, outfile)
write_with_length(traceback.format_exc().encode("utf-8"), outfile)
write_with_length(exc_info, outfile)
except IOError:
# JVM close the socket
pass
Expand Down