Skip to content

Commit 4ecf841

Browse files
committed
Make proactive serializability checking optional.
SparkContext.clean uses ClosureCleaner's proactive serializability checking by default. This commit adds an overloaded clean method to SparkContext that allows clients to specify that serializability checking should not occur as part of closure cleaning.
1 parent d8df3db commit 4ecf841

1 file changed

Lines changed: 9 additions & 1 deletion

File tree

core/src/main/scala/org/apache/spark/SparkContext.scala

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1032,7 +1032,15 @@ class SparkContext(
10321032
* (removes unreferenced variables in $outer's, updates REPL variables)
10331033
*/
10341034
private[spark] def clean[F <: AnyRef](f: F): F = {
1035-
ClosureCleaner.clean(f)
1035+
clean(f, true)
1036+
}
1037+
1038+
/**
1039+
* Clean a closure to make it ready to serialized and send to tasks
1040+
* (removes unreferenced variables in $outer's, updates REPL variables)
1041+
*/
1042+
private[spark] def clean[F <: AnyRef](f: F, checkSerializable: Boolean): F = {
1043+
ClosureCleaner.clean(f, checkSerializable)
10361044
f
10371045
}
10381046

0 commit comments

Comments
 (0)