Skip to content

[SPARK-10906][MLlib] More efficient SparseMatrix.equals #8960

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rahulpalamuttam wants to merge 7 commits into from
Closed

[SPARK-10906][MLlib] More efficient SparseMatrix.equals #8960

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f3f7e65
updated SparseMatrix equal method
rahulpalamuttam Oct 2, 2015
f9c6ac0
updated SparseMatrix equal method to use iterator traversal.
rahulpalamuttam Oct 9, 2015
70016e4
Updated tests to include 3x3 matrix with more zero elements.
rahulpalamuttam Oct 10, 2015
2fff7d7
Added tests with identity matrices
rahulpalamuttam Oct 10, 2015
7f3f888
unit test which fails in MatricesSuite equals
jkbradley Oct 10, 2015
b3c29ca
Updated to handle explicit and implicit zero comparison. Added corres…
rahulpalamuttam Oct 17, 2015
e882162
Merge branch 'rahulpalamuttam-RahulP_SparseMatrixEquals' of https://g…
rahulpalamuttam Oct 17, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -550,7 +550,10 @@ class SparseMatrix @Since("1.3.0") (
values: Array[Double]) = this(numRows, numCols, colPtrs, rowIndices, values, false)

override def equals(o: Any): Boolean = o match {
case m: Matrix => toBreeze == m.toBreeze
case m: Matrix =>
val thisIteratorSet = toBreeze.activeIterator.toSet
val mIteratorSet = m.toBreeze.activeIterator.toSet.filter(p => p._2 != 0.0)
thisIteratorSet == mIteratorSet
Copy link
Member

@jkbradley jkbradley Oct 9, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than converting to sets (fairly expensive), how about comparing the iterators directly? It's a bit more complex (handling zeros) but would then be linear time, with a small constant. It would also allow early stopping as soon as a non-matching value was found.

Also, this method should compare the matrix dimensions before looking at the values.

case _ => false
}

Expand Down