Skip to content

Commit 70c1be3

Browse files
committed
[SPARK-58732][ML] Avoid MLlib vector conversion in Normalizer
### What changes were proposed in this pull request? Update ML Normalizer to calculate the norm and normalize values with ML vectors directly. The normalization implementation follows the existing MLlib Normalizer transform method, including cloning dense and sparse value arrays and reusing sparse indices. ### Why are the changes needed? ML Normalizer currently converts every input vector from ML to MLlib and converts the normalized result back to ML. Avoiding those conversions reduces allocation and transform overhead. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The existing NormalizerSuite covers dense, sparse, zero-vector, and parameterized normalization behavior. build/sbt 'mllib/testOnly org.apache.spark.ml.feature.NormalizerSuite' ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) Closes #57955 from zhengruifeng/SPARK-58732-normalizer-native-vector. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com> (cherry picked from commit cddd9ae) Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>
1 parent 99302f5 commit 70c1be3

1 file changed

Lines changed: 38 additions & 5 deletions

File tree

mllib/src/main/scala/org/apache/spark/ml/feature/Normalizer.scala

Lines changed: 38 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,10 @@ package org.apache.spark.ml.feature
2020
import org.apache.spark.annotation.Since
2121
import org.apache.spark.ml.UnaryTransformer
2222
import org.apache.spark.ml.attribute.AttributeGroup
23-
import org.apache.spark.ml.linalg.{SQLDataTypes, Vector, VectorUDT}
23+
import org.apache.spark.ml.linalg.{DenseVector, SparseVector, SQLDataTypes, Vector, Vectors,
24+
VectorUDT}
2425
import org.apache.spark.ml.param.{DoubleParam, ParamValidators}
2526
import org.apache.spark.ml.util._
26-
import org.apache.spark.mllib.feature
27-
import org.apache.spark.mllib.linalg.{Vectors => OldVectors}
2827
import org.apache.spark.sql.types._
2928

3029
/**
@@ -56,8 +55,42 @@ class Normalizer @Since("1.4.0") (@Since("1.4.0") override val uid: String)
5655
def setP(value: Double): this.type = set(p, value)
5756

5857
override protected def createTransformFunc: Vector => Vector = {
59-
val normalizer = new feature.Normalizer($(p))
60-
vector => normalizer.transform(OldVectors.fromML(vector)).asML
58+
val localP = $(p)
59+
vector => {
60+
val norm = Vectors.norm(vector, localP)
61+
if (norm != 0.0) {
62+
val scale = 1.0 / norm
63+
// For dense vector, we've to allocate new memory for new output vector.
64+
// However, for sparse vector, the `index` array will not be changed,
65+
// so we can re-use it to save memory.
66+
vector match {
67+
case DenseVector(vs) =>
68+
val values = vs.clone()
69+
val size = values.length
70+
var i = 0
71+
while (i < size) {
72+
values(i) *= scale
73+
i += 1
74+
}
75+
Vectors.dense(values)
76+
case SparseVector(size, ids, vs) =>
77+
val values = vs.clone()
78+
val nnz = values.length
79+
var i = 0
80+
while (i < nnz) {
81+
values(i) *= scale
82+
i += 1
83+
}
84+
Vectors.sparse(size, ids, values)
85+
case v => throw new IllegalArgumentException("Do not support vector type " + v.getClass)
86+
}
87+
} else {
88+
// Since the norm is zero, return the input vector object itself.
89+
// Note that it's safe since we always assume that the data in RDD
90+
// should be immutable.
91+
vector
92+
}
93+
}
6194
}
6295

6396
override protected def validateInputType(inputType: DataType): Unit = {

0 commit comments

Comments
 (0)