Skip to content

Conversation

@kshyatt
Copy link
Member

@kshyatt kshyatt commented Oct 18, 2025

@kshyatt kshyatt requested a review from maleadt October 18, 2025 13:24
@maleadt
Copy link
Member

maleadt commented Oct 20, 2025

Tests didn't run yet: UndefVarError: AbstractGPUSparseArray not defined in GPUArrays

@kshyatt
Copy link
Member Author

kshyatt commented Oct 20, 2025

They do run on 1.12, btw - I think Resolver is making everything weird on 1.11 and 1.10. Either way there are some fixes I need to do.

@kshyatt kshyatt force-pushed the ksh/gpuarrays_sparse branch 2 times, most recently from a101ecd to c473fa7 Compare October 22, 2025 06:13
@gdalle
Copy link

gdalle commented Nov 12, 2025

What kind of expectations does GPUArrays.AbstractGPUSparseMatrixCSC/CSR have on the order of indices within each column/row?

@codecov
Copy link

codecov bot commented Nov 25, 2025

Codecov Report

❌ Patch coverage is 80.39216% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.15%. Comparing base (1af91be) to head (f561395).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
lib/cusparse/array.jl 75.47% 13 Missing ⚠️
lib/cusparse/generic.jl 66.66% 13 Missing ⚠️
lib/cusparse/conversions.jl 92.00% 4 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #2942       +/-   ##
===========================================
+ Coverage   76.12%   89.15%   +13.02%     
===========================================
  Files         150      148        -2     
  Lines       13068    12885      -183     
===========================================
+ Hits         9948    11487     +1539     
+ Misses       3120     1398     -1722     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kshyatt kshyatt force-pushed the ksh/gpuarrays_sparse branch 2 times, most recently from 1a5866d to cd339b2 Compare November 26, 2025 07:44
@kshyatt kshyatt marked this pull request as ready for review November 26, 2025 07:44
@github-actions
Copy link
Contributor

github-actions bot commented Nov 26, 2025

Your PR requires formatting changes to meet the project's style guidelines.
Please consider running Runic (git runic master) to apply these changes.

Click here to view the suggested changes.
diff --git a/lib/cusparse/array.jl b/lib/cusparse/array.jl
index 5ec4c1370..094d6487d 100644
--- a/lib/cusparse/array.jl
+++ b/lib/cusparse/array.jl
@@ -45,7 +45,7 @@ mutable struct CuSparseMatrixCSC{Tv, Ti} <: GPUArrays.AbstractGPUSparseMatrixCSC
 end
 CuSparseMatrixCSC{Tv, Ti}(csc::CuSparseMatrixCSC{Tv, Ti}) where {Tv, Ti} = csc
 
-SparseArrays.rowvals(g::T) where {T<:CuSparseVector} = nonzeroinds(g)
+SparseArrays.rowvals(g::T) where {T <: CuSparseVector} = nonzeroinds(g)
 
 SparseArrays.rowvals(g::CuSparseMatrixCSC) = g.rowVal
 SparseArrays.getcolptr(S::CuSparseMatrixCSC) = S.colPtr
@@ -110,10 +110,10 @@ GPUArrays.dense_array_type(::Type{<:CuSparseMatrixCSR}) = CuArray
 
 GPUArrays.csc_type(sa::CuSparseMatrixCSR) = CuSparseMatrixCSC
 GPUArrays.csr_type(sa::CuSparseMatrixCSC) = CuSparseMatrixCSR
-GPUArrays.coo_type(sa::Union{CuSparseMatrixCSR, Transpose{<:Any,<:CuSparseMatrixCSR}, Adjoint{<:Any,<:CuSparseMatrixCSR}}) = CuSparseMatrixCOO
-GPUArrays.coo_type(sa::Union{CuSparseMatrixCSC, Transpose{<:Any,<:CuSparseMatrixCSC}, Adjoint{<:Any,<:CuSparseMatrixCSC}}) = CuSparseMatrixCOO
-GPUArrays.coo_type(::Type{T}) where {T<:Union{CuSparseMatrixCSR, Transpose{<:Any,<:CuSparseMatrixCSR}, Adjoint{<:Any,<:CuSparseMatrixCSR}}} = CuSparseMatrixCOO
-GPUArrays.coo_type(::Type{T}) where {T<:Union{CuSparseMatrixCSC, Transpose{<:Any,<:CuSparseMatrixCSC}, Adjoint{<:Any,<:CuSparseMatrixCSC}}} = CuSparseMatrixCOO
+GPUArrays.coo_type(sa::Union{CuSparseMatrixCSR, Transpose{<:Any, <:CuSparseMatrixCSR}, Adjoint{<:Any, <:CuSparseMatrixCSR}}) = CuSparseMatrixCOO
+GPUArrays.coo_type(sa::Union{CuSparseMatrixCSC, Transpose{<:Any, <:CuSparseMatrixCSC}, Adjoint{<:Any, <:CuSparseMatrixCSC}}) = CuSparseMatrixCOO
+GPUArrays.coo_type(::Type{T}) where {T <: Union{CuSparseMatrixCSR, Transpose{<:Any, <:CuSparseMatrixCSR}, Adjoint{<:Any, <:CuSparseMatrixCSR}}} = CuSparseMatrixCOO
+GPUArrays.coo_type(::Type{T}) where {T <: Union{CuSparseMatrixCSC, Transpose{<:Any, <:CuSparseMatrixCSC}, Adjoint{<:Any, <:CuSparseMatrixCSC}}} = CuSparseMatrixCOO
 
 """
 Container to hold sparse matrices in block compressed sparse row (BSR) format on
@@ -553,12 +553,12 @@ CuSparseMatrixCSC(x::Adjoint{T}) where {T} = CuSparseMatrixCSC{T}(x)
 CuSparseMatrixCOO(x::Transpose{T}) where {T} = CuSparseMatrixCOO{T}(x)
 CuSparseMatrixCOO(x::Adjoint{T}) where {T} = CuSparseMatrixCOO{T}(x)
 
-CuSparseMatrixCSR(x::Transpose{T,<:Union{CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixCOO}}) where {T} = CuSparseMatrixCSR(GPUArrays._sptranspose(parent(x)))
-CuSparseMatrixCSC(x::Transpose{T,<:Union{CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixCOO}}) where {T} = CuSparseMatrixCSC(GPUArrays._sptranspose(parent(x)))
-CuSparseMatrixCOO(x::Transpose{T,<:Union{CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixCOO}}) where {T} = CuSparseMatrixCOO(GPUArrays._sptranspose(parent(x)))
-CuSparseMatrixCSR(x::Adjoint{T,<:Union{CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixCOO}}) where {T} = CuSparseMatrixCSR(GPUArrays._spadjoint(parent(x)))
-CuSparseMatrixCSC(x::Adjoint{T,<:Union{CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixCOO}}) where {T} = CuSparseMatrixCSC(GPUArrays._spadjoint(parent(x)))
-CuSparseMatrixCOO(x::Adjoint{T,<:Union{CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixCOO}}) where {T} = CuSparseMatrixCOO(GPUArrays._spadjoint(parent(x)))
+CuSparseMatrixCSR(x::Transpose{T, <:Union{CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixCOO}}) where {T} = CuSparseMatrixCSR(GPUArrays._sptranspose(parent(x)))
+CuSparseMatrixCSC(x::Transpose{T, <:Union{CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixCOO}}) where {T} = CuSparseMatrixCSC(GPUArrays._sptranspose(parent(x)))
+CuSparseMatrixCOO(x::Transpose{T, <:Union{CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixCOO}}) where {T} = CuSparseMatrixCOO(GPUArrays._sptranspose(parent(x)))
+CuSparseMatrixCSR(x::Adjoint{T, <:Union{CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixCOO}}) where {T} = CuSparseMatrixCSR(GPUArrays._spadjoint(parent(x)))
+CuSparseMatrixCSC(x::Adjoint{T, <:Union{CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixCOO}}) where {T} = CuSparseMatrixCSC(GPUArrays._spadjoint(parent(x)))
+CuSparseMatrixCOO(x::Adjoint{T, <:Union{CuSparseMatrixCSC, CuSparseMatrixCSR, CuSparseMatrixCOO}}) where {T} = CuSparseMatrixCOO(GPUArrays._spadjoint(parent(x)))
 
 # gpu to cpu
 SparseArrays.SparseVector(x::CuSparseVector) = SparseVector(length(x), Array(SparseArrays.nonzeroinds(x)), Array(SparseArrays.nonzeros(x)))
@@ -687,11 +687,13 @@ end
 
 # interop with device arrays
 
-function GPUArrays.GPUSparseDeviceVector(iPtr::CuDeviceVector{Ti, A},
-                                         nzVal::CuDeviceVector{Tv, A},
-                                         len::Int,
-                                         nnz::Ti) where {Ti, Tv, A}
-    GPUArrays.GPUSparseDeviceVector{Tv, Ti, CuDeviceVector{Ti, A}, CuDeviceVector{Tv, A}, A}(iPtr, nzVal, len, nnz)
+function GPUArrays.GPUSparseDeviceVector(
+        iPtr::CuDeviceVector{Ti, A},
+        nzVal::CuDeviceVector{Tv, A},
+        len::Int,
+        nnz::Ti
+    ) where {Ti, Tv, A}
+    return GPUArrays.GPUSparseDeviceVector{Tv, Ti, CuDeviceVector{Ti, A}, CuDeviceVector{Tv, A}, A}(iPtr, nzVal, len, nnz)
 end
 
 function Adapt.adapt_structure(to::CUDA.KernelAdaptor, x::CuSparseVector)
@@ -702,12 +704,14 @@ function Adapt.adapt_structure(to::CUDA.KernelAdaptor, x::CuSparseVector)
     )
 end
 
-function GPUArrays.GPUSparseDeviceMatrixCSR(rowPtr::CuDeviceVector{Ti, A},
-                                            colVal::CuDeviceVector{Ti, A},
-                                            nzVal::CuDeviceVector{Tv, A},
-                                            dims::NTuple{2, Int},
-                                            nnz::Ti) where {Ti, Tv, A}
-    GPUArrays.GPUSparseDeviceMatrixCSR{Tv, Ti, CuDeviceVector{Ti, A}, CuDeviceVector{Tv, A}, A}(rowPtr, colVal, nzVal, dims, nnz)
+function GPUArrays.GPUSparseDeviceMatrixCSR(
+        rowPtr::CuDeviceVector{Ti, A},
+        colVal::CuDeviceVector{Ti, A},
+        nzVal::CuDeviceVector{Tv, A},
+        dims::NTuple{2, Int},
+        nnz::Ti
+    ) where {Ti, Tv, A}
+    return GPUArrays.GPUSparseDeviceMatrixCSR{Tv, Ti, CuDeviceVector{Ti, A}, CuDeviceVector{Tv, A}, A}(rowPtr, colVal, nzVal, dims, nnz)
 end
 
 function Adapt.adapt_structure(to::CUDA.KernelAdaptor, x::CuSparseMatrixCSR)
@@ -719,12 +723,14 @@ function Adapt.adapt_structure(to::CUDA.KernelAdaptor, x::CuSparseMatrixCSR)
     )
 end
 
-function GPUArrays.GPUSparseDeviceMatrixCSC(colPtr::CuDeviceVector{Ti, A},
-                                            rowVal::CuDeviceVector{Ti, A},
-                                            nzVal::CuDeviceVector{Tv, A},
-                                            dims::NTuple{2, Int},
-                                            nnz::Ti) where {Ti, Tv, A}
-    GPUArrays.GPUSparseDeviceMatrixCSC{Tv, Ti, CuDeviceVector{Ti, A}, CuDeviceVector{Tv, A}, A}(colPtr, rowVal, nzVal, dims, nnz)
+function GPUArrays.GPUSparseDeviceMatrixCSC(
+        colPtr::CuDeviceVector{Ti, A},
+        rowVal::CuDeviceVector{Ti, A},
+        nzVal::CuDeviceVector{Tv, A},
+        dims::NTuple{2, Int},
+        nnz::Ti
+    ) where {Ti, Tv, A}
+    return GPUArrays.GPUSparseDeviceMatrixCSC{Tv, Ti, CuDeviceVector{Ti, A}, CuDeviceVector{Tv, A}, A}(colPtr, rowVal, nzVal, dims, nnz)
 end
 
 function Adapt.adapt_structure(to::CUDA.KernelAdaptor, x::CuSparseMatrixCSC)
@@ -736,14 +742,16 @@ function Adapt.adapt_structure(to::CUDA.KernelAdaptor, x::CuSparseMatrixCSC)
     )
 end
 
-function GPUArrays.GPUSparseDeviceMatrixBSR(rowPtr::CuDeviceVector{Ti, A},
-                                            colVal::CuDeviceVector{Ti, A},
-                                            nzVal::CuDeviceVector{Tv, A},
-                                            dims::NTuple{2, Int},
-                                            blockDim::Ti,
-                                            dir::Char,
-                                            nnz::Ti) where {Ti, Tv, A}
-    GPUArrays.GPUSparseDeviceMatrixBSR{Tv, Ti, CuDeviceVector{Ti, A}, CuDeviceVector{Tv, A}, A}(rowPtr, colVal, nzVal, dims, blockDim, dir, nnz)
+function GPUArrays.GPUSparseDeviceMatrixBSR(
+        rowPtr::CuDeviceVector{Ti, A},
+        colVal::CuDeviceVector{Ti, A},
+        nzVal::CuDeviceVector{Tv, A},
+        dims::NTuple{2, Int},
+        blockDim::Ti,
+        dir::Char,
+        nnz::Ti
+    ) where {Ti, Tv, A}
+    return GPUArrays.GPUSparseDeviceMatrixBSR{Tv, Ti, CuDeviceVector{Ti, A}, CuDeviceVector{Tv, A}, A}(rowPtr, colVal, nzVal, dims, blockDim, dir, nnz)
 end
 
 function Adapt.adapt_structure(to::CUDA.KernelAdaptor, x::CuSparseMatrixBSR)
@@ -756,12 +764,14 @@ function Adapt.adapt_structure(to::CUDA.KernelAdaptor, x::CuSparseMatrixBSR)
     )
 end
 
-function GPUArrays.GPUSparseDeviceMatrixCOO(rowInd::CuDeviceVector{Ti, A},
-                                            colInd::CuDeviceVector{Ti, A},
-                                            nzVal::CuDeviceVector{Tv, A},
-                                            dims::NTuple{2, Int},
-                                            nnz::Ti) where {Ti, Tv, A}
-    GPUArrays.GPUSparseDeviceMatrixCOO{Tv, Ti, CuDeviceVector{Ti, A}, CuDeviceVector{Tv, A}, A}(rowInd, colInd, nzVal, dims, nnz)
+function GPUArrays.GPUSparseDeviceMatrixCOO(
+        rowInd::CuDeviceVector{Ti, A},
+        colInd::CuDeviceVector{Ti, A},
+        nzVal::CuDeviceVector{Tv, A},
+        dims::NTuple{2, Int},
+        nnz::Ti
+    ) where {Ti, Tv, A}
+    return GPUArrays.GPUSparseDeviceMatrixCOO{Tv, Ti, CuDeviceVector{Ti, A}, CuDeviceVector{Tv, A}, A}(rowInd, colInd, nzVal, dims, nnz)
 end
 
 function Adapt.adapt_structure(to::CUDA.KernelAdaptor, x::CuSparseMatrixCOO)
diff --git a/lib/cusparse/conversions.jl b/lib/cusparse/conversions.jl
index ad6f1b8e3..9b897cff9 100644
--- a/lib/cusparse/conversions.jl
+++ b/lib/cusparse/conversions.jl
@@ -1,8 +1,9 @@
 export sort_csc, sort_csr, sort_coo
 
 adjtrans_wrappers = ((identity, identity),
-                     (M -> :(Transpose{T, <:$M}), M -> :(GPUArrays._sptranspose(parent($M)))),
-                     (M -> :(Adjoint{T, <:$M}), M -> :(GPUArrays._spadjoint(parent($M)))))
+    (M -> :(Transpose{T, <:$M}), M -> :(GPUArrays._sptranspose(parent($M)))),
+    (M -> :(Adjoint{T, <:$M}), M -> :(GPUArrays._spadjoint(parent($M)))),
+)
 
 # conversion routines between different sparse and dense storage formats
 
@@ -330,7 +331,7 @@ end
 # by flipping rows and columns, we can use that to get CSC to CSR too
 for elty in (:Float32, :Float64, :ComplexF32, :ComplexF64)
     @eval begin
-        function CuSparseMatrixCSC{$elty, Ti}(csr::CuSparseMatrixCSR{$elty, Ti}; index::SparseChar='O', action::cusparseAction_t=CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t=CUSPARSE_CSR2CSC_ALG1) where {Ti}
+        function CuSparseMatrixCSC{$elty, Ti}(csr::CuSparseMatrixCSR{$elty, Ti}; index::SparseChar = 'O', action::cusparseAction_t = CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t = CUSPARSE_CSR2CSC_ALG1) where {Ti}
             m,n = size(csr)
             colPtr = CUDA.zeros(Cint, n+1)
             rowVal = CUDA.zeros(Cint, nnz(csr))
@@ -349,11 +350,11 @@ for elty in (:Float32, :Float64, :ComplexF32, :ComplexF64)
             end
             CuSparseMatrixCSC(colPtr,rowVal,nzVal,size(csr))
         end
-        CuSparseMatrixCSC{$elty}(csr::CuSparseMatrixCSR{$elty, Ti}; index::SparseChar='O', action::cusparseAction_t=CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t=CUSPARSE_CSR2CSC_ALG1) where {Ti} =
-            CuSparseMatrixCSC{$elty, Ti}(csr; index=index, action=action, algo=algo)
-        CuSparseMatrixCSC(csr::CuSparseMatrixCSR{$elty, Ti}; index::SparseChar='O', action::cusparseAction_t=CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t=CUSPARSE_CSR2CSC_ALG1) where {Ti} =
-            CuSparseMatrixCSC{$elty, Ti}(csr; index=index, action=action, algo=algo)
-        function CuSparseMatrixCSR{$elty, Ti}(csc::CuSparseMatrixCSC{$elty, Ti}; index::SparseChar='O', action::cusparseAction_t=CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t=CUSPARSE_CSR2CSC_ALG1) where {Ti}
+        CuSparseMatrixCSC{$elty}(csr::CuSparseMatrixCSR{$elty, Ti}; index::SparseChar = 'O', action::cusparseAction_t = CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t = CUSPARSE_CSR2CSC_ALG1) where {Ti} =
+            CuSparseMatrixCSC{$elty, Ti}(csr; index = index, action = action, algo = algo)
+        CuSparseMatrixCSC(csr::CuSparseMatrixCSR{$elty, Ti}; index::SparseChar = 'O', action::cusparseAction_t = CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t = CUSPARSE_CSR2CSC_ALG1) where {Ti} =
+            CuSparseMatrixCSC{$elty, Ti}(csr; index = index, action = action, algo = algo)
+        function CuSparseMatrixCSR{$elty, Ti}(csc::CuSparseMatrixCSC{$elty, Ti}; index::SparseChar = 'O', action::cusparseAction_t = CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t = CUSPARSE_CSR2CSC_ALG1) where {Ti}
             m,n    = size(csc)
             rowPtr = CUDA.zeros(Cint,m+1)
             colVal = CUDA.zeros(Cint,nnz(csc))
@@ -372,10 +373,10 @@ for elty in (:Float32, :Float64, :ComplexF32, :ComplexF64)
             end
             CuSparseMatrixCSR(rowPtr,colVal,nzVal,size(csc))
         end
-        CuSparseMatrixCSR(csc::CuSparseMatrixCSC{$elty, Ti}; index::SparseChar='O', action::cusparseAction_t=CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t=CUSPARSE_CSR2CSC_ALG1) where {Ti} =
-            CuSparseMatrixCSR{$elty, Ti}(csc; index=index, action=action, algo=algo)
-        CuSparseMatrixCSR{$elty}(csc::CuSparseMatrixCSC{$elty, Ti}; index::SparseChar='O', action::cusparseAction_t=CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t=CUSPARSE_CSR2CSC_ALG1) where {Ti} =
-            CuSparseMatrixCSR{$elty, Ti}(csc; index=index, action=action, algo=algo)
+        CuSparseMatrixCSR(csc::CuSparseMatrixCSC{$elty, Ti}; index::SparseChar = 'O', action::cusparseAction_t = CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t = CUSPARSE_CSR2CSC_ALG1) where {Ti} =
+            CuSparseMatrixCSR{$elty, Ti}(csc; index = index, action = action, algo = algo)
+        CuSparseMatrixCSR{$elty}(csc::CuSparseMatrixCSC{$elty, Ti}; index::SparseChar = 'O', action::cusparseAction_t = CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t = CUSPARSE_CSR2CSC_ALG1) where {Ti} =
+            CuSparseMatrixCSR{$elty, Ti}(csc; index = index, action = action, algo = algo)
     end
 end
 
@@ -384,7 +385,7 @@ end
 for (elty, welty) in ((:Float16, :Float32),
                       (:ComplexF16, :ComplexF32))
     @eval begin
-        function CuSparseMatrixCSC{$elty, Ti}(csr::CuSparseMatrixCSR{$elty, Ti}; index::SparseChar='O', action::cusparseAction_t=CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t=CUSPARSE_CSR2CSC_ALG1) where {Ti}
+        function CuSparseMatrixCSC{$elty, Ti}(csr::CuSparseMatrixCSR{$elty, Ti}; index::SparseChar = 'O', action::cusparseAction_t = CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t = CUSPARSE_CSR2CSC_ALG1) where {Ti}
             m,n = size(csr)
             colPtr = CUDA.zeros(Cint, n+1)
             rowVal = CUDA.zeros(Cint, nnz(csr))
@@ -409,11 +410,11 @@ for (elty, welty) in ((:Float16, :Float32),
                 return CuSparseMatrixCSC(wide_csc.colPtr, wide_csc.rowVal, convert(CuVector{$elty}, nonzeros(wide_csc)), size(wide_csc))
             end
         end
-        CuSparseMatrixCSC{$elty}(csr::CuSparseMatrixCSR{$elty, Ti}; index::SparseChar='O', action::cusparseAction_t=CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t=CUSPARSE_CSR2CSC_ALG1) where {Ti} =
-            CuSparseMatrixCSC{$elty, Ti}(csr; index=index, action=action, algo=algo)
-        CuSparseMatrixCSC(csr::CuSparseMatrixCSR{$elty, Ti}; index::SparseChar='O', action::cusparseAction_t=CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t=CUSPARSE_CSR2CSC_ALG1) where {Ti} =
-            CuSparseMatrixCSC{$elty, Ti}(csr; index=index, action=action, algo=algo)
-        function CuSparseMatrixCSR{$elty, Ti}(csc::CuSparseMatrixCSC{$elty, Ti}; index::SparseChar='O', action::cusparseAction_t=CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t=CUSPARSE_CSR2CSC_ALG1) where {Ti}
+        CuSparseMatrixCSC{$elty}(csr::CuSparseMatrixCSR{$elty, Ti}; index::SparseChar = 'O', action::cusparseAction_t = CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t = CUSPARSE_CSR2CSC_ALG1) where {Ti} =
+            CuSparseMatrixCSC{$elty, Ti}(csr; index = index, action = action, algo = algo)
+        CuSparseMatrixCSC(csr::CuSparseMatrixCSR{$elty, Ti}; index::SparseChar = 'O', action::cusparseAction_t = CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t = CUSPARSE_CSR2CSC_ALG1) where {Ti} =
+            CuSparseMatrixCSC{$elty, Ti}(csr; index = index, action = action, algo = algo)
+        function CuSparseMatrixCSR{$elty, Ti}(csc::CuSparseMatrixCSC{$elty, Ti}; index::SparseChar = 'O', action::cusparseAction_t = CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t = CUSPARSE_CSR2CSC_ALG1) where {Ti}
             m,n    = size(csc)
             rowPtr = CUDA.zeros(Cint,m+1)
             colVal = CUDA.zeros(Cint,nnz(csc))
@@ -438,10 +439,10 @@ for (elty, welty) in ((:Float16, :Float32),
                 return CuSparseMatrixCSR(wide_csr.rowPtr, wide_csr.colVal, convert(CuVector{$elty}, nonzeros(wide_csr)), size(wide_csr))
             end
         end
-        CuSparseMatrixCSR(csc::CuSparseMatrixCSC{$elty, Ti}; index::SparseChar='O', action::cusparseAction_t=CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t=CUSPARSE_CSR2CSC_ALG1) where {Ti} =
-            CuSparseMatrixCSR{$elty, Ti}(csc; index=index, action=action, algo=algo)
-        CuSparseMatrixCSR{$elty}(csc::CuSparseMatrixCSC{$elty, Ti}; index::SparseChar='O', action::cusparseAction_t=CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t=CUSPARSE_CSR2CSC_ALG1) where {Ti} =
-            CuSparseMatrixCSR{$elty, Ti}(csc; index=index, action=action, algo=algo)
+        CuSparseMatrixCSR(csc::CuSparseMatrixCSC{$elty, Ti}; index::SparseChar = 'O', action::cusparseAction_t = CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t = CUSPARSE_CSR2CSC_ALG1) where {Ti} =
+            CuSparseMatrixCSR{$elty, Ti}(csc; index = index, action = action, algo = algo)
+        CuSparseMatrixCSR{$elty}(csc::CuSparseMatrixCSC{$elty, Ti}; index::SparseChar = 'O', action::cusparseAction_t = CUSPARSE_ACTION_NUMERIC, algo::cusparseCsr2CscAlg_t = CUSPARSE_CSR2CSC_ALG1) where {Ti} =
+            CuSparseMatrixCSR{$elty, Ti}(csc; index = index, action = action, algo = algo)
     end
 end
 
@@ -508,9 +509,11 @@ for (fname,elty) in ((:cusparseScsr2bsr, :Float32),
                      (:cusparseCcsr2bsr, :ComplexF32),
                      (:cusparseZcsr2bsr, :ComplexF64))
     @eval begin
-        function CuSparseMatrixBSR{$elty}(csr::CuSparseMatrixCSR{$elty, Ti}, blockDim::Integer;
+        function CuSparseMatrixBSR{$elty}(
+                csr::CuSparseMatrixCSR{$elty, Ti}, blockDim::Integer;
                                           dir::SparseChar='R', index::SparseChar='O',
-                                          indc::SparseChar='O') where {Ti}
+                indc::SparseChar = 'O'
+            ) where {Ti}
             m,n = size(csr)
             nnz_ref = Ref{Cint}(1)
             mb = cld(m, blockDim)
@@ -537,8 +540,10 @@ for (fname,elty) in ((:cusparseSbsr2csr, :Float32),
                      (:cusparseCbsr2csr, :ComplexF32),
                      (:cusparseZbsr2csr, :ComplexF64))
     @eval begin
-        function CuSparseMatrixCSR{$elty}(bsr::CuSparseMatrixBSR{$elty, Ti};
-                                          index::SparseChar='O', indc::SparseChar='O') where {Ti}
+        function CuSparseMatrixCSR{$elty}(
+                bsr::CuSparseMatrixBSR{$elty, Ti};
+                index::SparseChar = 'O', indc::SparseChar = 'O'
+            ) where {Ti}
             m,n = size(bsr)
             mb = cld(m, bsr.blockDim)
             nb = cld(n, bsr.blockDim)
@@ -564,8 +569,10 @@ for (elty, felty) in ((:Int16, :Float16),
                       (:Int64, :Float64),
                       (:Int128, :ComplexF64))
     @eval begin
-        function CuSparseMatrixCSR{$elty}(bsr::CuSparseMatrixBSR{$elty, Ti};
-                                          index::SparseChar='O', indc::SparseChar='O') where {Ti}
+        function CuSparseMatrixCSR{$elty}(
+                bsr::CuSparseMatrixBSR{$elty, Ti};
+                index::SparseChar = 'O', indc::SparseChar = 'O'
+            ) where {Ti}
             bsr_compat = CuSparseMatrixBSR(
                 bsr.rowPtr,
                 bsr.colVal,
@@ -576,7 +583,7 @@ for (elty, felty) in ((:Int16, :Float16),
                 size(bsr)
             )
             csr_compat = CuSparseMatrixCSR{$felty}(bsr_compat; index, indc)
-            CuSparseMatrixCSR{$elty, Ti}(
+            return CuSparseMatrixCSR{$elty, Ti}(
                 csr_compat.rowPtr,
                 csr_compat.colVal,
                 reinterpret($elty, csr_compat.nzVal),
@@ -608,50 +615,50 @@ end
 ## CSR to COO and vice-versa
 
 # need both typevars for compatibility with GPUArrays
-function CuSparseMatrixCSR{Tv, Ti}(coo::CuSparseMatrixCOO{Tv, Ti}; index::SparseChar='O') where {Tv, Ti}
+function CuSparseMatrixCSR{Tv, Ti}(coo::CuSparseMatrixCOO{Tv, Ti}; index::SparseChar = 'O') where {Tv, Ti}
     m,n = size(coo)
-    nnz(coo) == 0 && return CuSparseMatrixCSR{Tv,Cint}(CUDA.ones(Cint, m+1), coo.colInd, nonzeros(coo), size(coo))
+    nnz(coo) == 0 && return CuSparseMatrixCSR{Tv, Cint}(CUDA.ones(Cint, m + 1), coo.colInd, nonzeros(coo), size(coo))
     coo = sort_coo(coo, 'R')
     csrRowPtr = CuVector{Cint}(undef, m+1)
     cusparseXcoo2csr(handle(), coo.rowInd, nnz(coo), m, csrRowPtr, index)
-    CuSparseMatrixCSR{Tv,Cint}(csrRowPtr, coo.colInd, nonzeros(coo), size(coo))
+    return CuSparseMatrixCSR{Tv, Cint}(csrRowPtr, coo.colInd, nonzeros(coo), size(coo))
 end
-CuSparseMatrixCSR{Tv}(coo::CuSparseMatrixCOO{Tv, Ti}; index::SparseChar='O') where {Tv, Ti} = CuSparseMatrixCSR{Tv, Ti}(coo; index)
-CuSparseMatrixCSR(coo::CuSparseMatrixCOO{Tv, Ti}; index::SparseChar='O') where {Tv, Ti} = CuSparseMatrixCSR{Tv, Ti}(coo; index)
+CuSparseMatrixCSR{Tv}(coo::CuSparseMatrixCOO{Tv, Ti}; index::SparseChar = 'O') where {Tv, Ti} = CuSparseMatrixCSR{Tv, Ti}(coo; index)
+CuSparseMatrixCSR(coo::CuSparseMatrixCOO{Tv, Ti}; index::SparseChar = 'O') where {Tv, Ti} = CuSparseMatrixCSR{Tv, Ti}(coo; index)
 
-function CuSparseMatrixCOO{Tv, Ti}(csr::CuSparseMatrixCSR{Tv}; index::SparseChar='O') where {Tv, Ti}
+function CuSparseMatrixCOO{Tv, Ti}(csr::CuSparseMatrixCSR{Tv}; index::SparseChar = 'O') where {Tv, Ti}
     m,n = size(csr)
-    nnz(csr) == 0 && return CuSparseMatrixCOO{Tv,Cint}(CUDA.zeros(Cint, 0), CUDA.zeros(Cint, 0), nonzeros(csr), size(csr))
+    nnz(csr) == 0 && return CuSparseMatrixCOO{Tv, Cint}(CUDA.zeros(Cint, 0), CUDA.zeros(Cint, 0), nonzeros(csr), size(csr))
     cooRowInd = CuVector{Cint}(undef, nnz(csr))
     cusparseXcsr2coo(handle(), csr.rowPtr, nnz(csr), m, cooRowInd, index)
-    CuSparseMatrixCOO{Tv,Cint}(cooRowInd, csr.colVal, nonzeros(csr), size(csr))
+    return CuSparseMatrixCOO{Tv, Cint}(cooRowInd, csr.colVal, nonzeros(csr), size(csr))
 end
-CuSparseMatrixCOO{Tv}(csr::CuSparseMatrixCSR{Tv, Ti}; index::SparseChar='O') where {Tv, Ti} = CuSparseMatrixCOO{Tv, Ti}(csr; index)
-CuSparseMatrixCOO(csr::CuSparseMatrixCSR{Tv, Ti}; index::SparseChar='O') where {Tv, Ti} = CuSparseMatrixCOO{Tv, Ti}(csr; index)
+CuSparseMatrixCOO{Tv}(csr::CuSparseMatrixCSR{Tv, Ti}; index::SparseChar = 'O') where {Tv, Ti} = CuSparseMatrixCOO{Tv, Ti}(csr; index)
+CuSparseMatrixCOO(csr::CuSparseMatrixCSR{Tv, Ti}; index::SparseChar = 'O') where {Tv, Ti} = CuSparseMatrixCOO{Tv, Ti}(csr; index)
 
 ### CSC to COO and viceversa
 
-function CuSparseMatrixCSC{Tv, Ti}(coo::CuSparseMatrixCOO{Tv, Ti}; index::SparseChar='O') where {Tv, Ti}
+function CuSparseMatrixCSC{Tv, Ti}(coo::CuSparseMatrixCOO{Tv, Ti}; index::SparseChar = 'O') where {Tv, Ti}
     m,n = size(coo)
     nnz(coo) == 0 && return CuSparseMatrixCSC{Tv}(CUDA.ones(Cint, n+1), coo.rowInd, nonzeros(coo), size(coo))
     coo = sort_coo(coo, 'C')
     cscColPtr = CuVector{Cint}(undef, n+1)
     cusparseXcoo2csr(handle(), coo.colInd, nnz(coo), n, cscColPtr, index)
-    CuSparseMatrixCSC{Tv,Cint}(cscColPtr, coo.rowInd, nonzeros(coo), size(coo))
+    return CuSparseMatrixCSC{Tv, Cint}(cscColPtr, coo.rowInd, nonzeros(coo), size(coo))
 end
-CuSparseMatrixCSC{Tv}(coo::CuSparseMatrixCOO{Tv, Ti}; index::SparseChar='O') where {Tv, Ti} = CuSparseMatrixCSC{Tv, Ti}(coo; index)
-CuSparseMatrixCSC(coo::CuSparseMatrixCOO{Tv, Ti}; index::SparseChar='O') where {Tv, Ti} = CuSparseMatrixCSC{Tv, Ti}(coo; index)
+CuSparseMatrixCSC{Tv}(coo::CuSparseMatrixCOO{Tv, Ti}; index::SparseChar = 'O') where {Tv, Ti} = CuSparseMatrixCSC{Tv, Ti}(coo; index)
+CuSparseMatrixCSC(coo::CuSparseMatrixCOO{Tv, Ti}; index::SparseChar = 'O') where {Tv, Ti} = CuSparseMatrixCSC{Tv, Ti}(coo; index)
 
-function CuSparseMatrixCOO{Tv, Ti}(csc::CuSparseMatrixCSC{Tv, Ti}; index::SparseChar='O') where {Tv, Ti}
+function CuSparseMatrixCOO{Tv, Ti}(csc::CuSparseMatrixCSC{Tv, Ti}; index::SparseChar = 'O') where {Tv, Ti}
     m,n = size(csc)
-    nnz(csc) == 0 && return CuSparseMatrixCOO{Tv,Cint}(CUDA.zeros(Cint, 0), CUDA.zeros(Cint, 0), nonzeros(csc), size(csc))
+    nnz(csc) == 0 && return CuSparseMatrixCOO{Tv, Cint}(CUDA.zeros(Cint, 0), CUDA.zeros(Cint, 0), nonzeros(csc), size(csc))
     cooColInd = CuVector{Cint}(undef, nnz(csc))
     cusparseXcsr2coo(handle(), csc.colPtr, nnz(csc), n, cooColInd, index)
-    coo = CuSparseMatrixCOO{Tv,Cint}(csc.rowVal, cooColInd, nonzeros(csc), size(csc))
+    coo = CuSparseMatrixCOO{Tv, Cint}(csc.rowVal, cooColInd, nonzeros(csc), size(csc))
     coo = sort_coo(coo, 'R')
 end
-CuSparseMatrixCOO{Tv}(csc::CuSparseMatrixCSC{Tv, Ti}; index::SparseChar='O') where {Tv, Ti} = CuSparseMatrixCOO{Tv, Ti}(csc; index)
-CuSparseMatrixCOO(csc::CuSparseMatrixCSC{Tv, Ti}; index::SparseChar='O') where {Tv, Ti} = CuSparseMatrixCOO{Tv, Ti}(csc; index)
+CuSparseMatrixCOO{Tv}(csc::CuSparseMatrixCSC{Tv, Ti}; index::SparseChar = 'O') where {Tv, Ti} = CuSparseMatrixCOO{Tv, Ti}(csc; index)
+CuSparseMatrixCOO(csc::CuSparseMatrixCSC{Tv, Ti}; index::SparseChar = 'O') where {Tv, Ti} = CuSparseMatrixCOO{Tv, Ti}(csc; index)
 
 ### BSR to COO and vice-versa
 
@@ -688,8 +695,10 @@ function CuSparseMatrixCSC(A::CuMatrix{T}; index::SparseChar='O', sorted::Bool=f
     return csc
 end
 
-function CUDA.CuMatrix{T}(bsr::CuSparseMatrixBSR{T, Ti}; index::SparseChar='O',
-                          indc::SparseChar='O') where {T, Ti}
+function CUDA.CuMatrix{T}(
+        bsr::CuSparseMatrixBSR{T, Ti}; index::SparseChar = 'O',
+        indc::SparseChar = 'O'
+    ) where {T, Ti}
     CuMatrix{T}(CuSparseMatrixCSR{T}(bsr; index, indc))
 end
 
diff --git a/lib/cusparse/generic.jl b/lib/cusparse/generic.jl
index 1ce4ea53a..18674f982 100644
--- a/lib/cusparse/generic.jl
+++ b/lib/cusparse/generic.jl
@@ -25,13 +25,15 @@ function mm! end
 ## API functions
 
 # implement Int conversions using reinterpreted Float
-for (elty, felty) in ((:Int16, :Float16),
-                      (:Int32, :Float32),
-                      (:Int64, :Float64),
-                      (:Int128, :ComplexF64))
+for (elty, felty) in (
+        (:Int16, :Float16),
+        (:Int32, :Float32),
+        (:Int64, :Float64),
+        (:Int128, :ComplexF64),
+    )
     @eval begin
-        function sparsetodense(coo::CuSparseMatrixCOO{$elty}, index::SparseChar, algo::cusparseSparseToDenseAlg_t=CUSPARSE_SPARSETODENSE_ALG_DEFAULT)
-            m,n = size(coo)
+        function sparsetodense(coo::CuSparseMatrixCOO{$elty}, index::SparseChar, algo::cusparseSparseToDenseAlg_t = CUSPARSE_SPARSETODENSE_ALG_DEFAULT)
+            m, n = size(coo)
             coo_compat = CuSparseMatrixCOO(
                 coo.rowInd,
                 coo.colInd,
@@ -52,8 +54,8 @@ for (elty, felty) in ((:Int16, :Float16),
             end
             return reinterpret($elty, B)
         end
-        function sparsetodense(bsr::CuSparseMatrixBSR{$elty}, index::SparseChar, algo::cusparseSparseToDenseAlg_t=CUSPARSE_SPARSETODENSE_ALG_DEFAULT)
-            m,n = size(bsr)
+        function sparsetodense(bsr::CuSparseMatrixBSR{$elty}, index::SparseChar, algo::cusparseSparseToDenseAlg_t = CUSPARSE_SPARSETODENSE_ALG_DEFAULT)
+            m, n = size(bsr)
             bsr_compat = CuSparseMatrixBSR(
                 bsr.rowPtr,
                 bsr.colVal,
@@ -77,8 +79,8 @@ for (elty, felty) in ((:Int16, :Float16),
             end
             return reinterpret($elty, B)
         end
-        function sparsetodense(csr::CuSparseMatrixCSR{$elty}, index::SparseChar, algo::cusparseSparseToDenseAlg_t=CUSPARSE_SPARSETODENSE_ALG_DEFAULT)
-            m,n = size(csr)
+        function sparsetodense(csr::CuSparseMatrixCSR{$elty}, index::SparseChar, algo::cusparseSparseToDenseAlg_t = CUSPARSE_SPARSETODENSE_ALG_DEFAULT)
+            m, n = size(csr)
             csr_compat = CuSparseMatrixCSR(
                 csr.rowPtr,
                 csr.colVal,
diff --git a/src/CUDAKernels.jl b/src/CUDAKernels.jl
index 5b4fd54fa..68b751efe 100644
--- a/src/CUDAKernels.jl
+++ b/src/CUDAKernels.jl
@@ -26,7 +26,7 @@ CUDABackend(; prefer_blocks=false, always_inline=false) = CUDABackend(prefer_blo
 @inline KA.ones(::CUDABackend, ::Type{T}, dims::Tuple; unified::Bool = false) where T = fill!(CuArray{T, length(dims), unified ? UnifiedMemory : default_memory}(undef, dims), one(T))
 
 KA.get_backend(::CuArray) = CUDABackend()
-KA.get_backend(::CUSPARSE.CuSparseVector)    = CUDABackend()
+KA.get_backend(::CUSPARSE.CuSparseVector) = CUDABackend()
 KA.get_backend(::CUSPARSE.CuSparseMatrixCSC) = CUDABackend()
 KA.get_backend(::CUSPARSE.CuSparseMatrixCSR) = CUDABackend()
 KA.synchronize(::CUDABackend) = synchronize()
@@ -36,8 +36,8 @@ KA.functional(::CUDABackend) = CUDA.functional()
 KA.supports_unified(::CUDABackend) = true
 
 Adapt.adapt_storage(::CUDABackend, a::AbstractArray) = Adapt.adapt(CuArray, a)
-Adapt.adapt_storage(::CUDABackend, a::Union{CuArray,GPUArrays.AbstractGPUSparseArray}) = a
-Adapt.adapt_storage(::KA.CPU, a::Union{CuArray,GPUArrays.AbstractGPUSparseArray}) = Adapt.adapt(Array, a)
+Adapt.adapt_storage(::CUDABackend, a::Union{CuArray, GPUArrays.AbstractGPUSparseArray}) = a
+Adapt.adapt_storage(::KA.CPU, a::Union{CuArray, GPUArrays.AbstractGPUSparseArray}) = Adapt.adapt(Array, a)
 
 ## memory operations
 
diff --git a/test/libraries/cusparse.jl b/test/libraries/cusparse.jl
index 396270fcf..ec6669e15 100644
--- a/test/libraries/cusparse.jl
+++ b/test/libraries/cusparse.jl
@@ -271,11 +271,11 @@ end
             @test similar(d_x, Float32, n, m) isa CuSparseMatrixCSR{Float32}
             @test similar(d_x, Float32, (n, m)) isa CuSparseMatrixCSR{Float32}
         end
-       
-        if elty <: Union{Float32, Float64, ComplexF32, ComplexF64} 
+
+        if elty <: Union{Float32, Float64, ComplexF32, ComplexF64}
             @testset "COO" begin
-                x = sprand(elty,m,n, 0.2)
-                d_x  = CuSparseMatrixCOO(x)
+                x = sprand(elty, m, n, 0.2)
+                d_x = CuSparseMatrixCOO(x)
                 @test collect(d_x) == collect(x)
                 @test similar(d_x) isa CuSparseMatrixCOO{elty}
                 @test similar(d_x, (3, 4)) isa CuSparseMatrixCOO{elty}
@@ -287,7 +287,7 @@ end
         end
 
         @testset "BSR" begin
-            x   = sprand(elty,m,n, 0.2)
+            x = sprand(elty, m, n, 0.2)
             d_x = CuSparseMatrixBSR(x, blockdim)
             @test collect(d_x) == collect(x)
             @test similar(d_x) isa CuSparseMatrixBSR{elty}
@@ -295,7 +295,7 @@ end
         end
 
         @testset "COO" begin
-            x   = sprand(elty,m,n, 0.2)
+            x = sprand(elty, m, n, 0.2)
             d_x = CuSparseMatrixCOO(x)
             @test collect(d_x) == collect(x)
             @test similar(d_x) isa CuSparseMatrixCOO{elty}
diff --git a/test/libraries/cusparse/device.jl b/test/libraries/cusparse/device.jl
index 164d5835b..fc1b60ca2 100644
--- a/test/libraries/cusparse/device.jl
+++ b/test/libraries/cusparse/device.jl
@@ -2,7 +2,7 @@ using CUDA.CUSPARSE
 using SparseArrays
 using SparseArrays: nonzeros, nnz, rowvals
 using CUDA.GPUArrays: GPUSparseDeviceVector, GPUSparseDeviceMatrixCSC, GPUSparseDeviceMatrixCSR,
-                      GPUSparseDeviceMatrixBSR, GPUSparseDeviceMatrixCOO
+    GPUSparseDeviceMatrixBSR, GPUSparseDeviceMatrixCOO
 
 @testset "cudaconvert" begin
     @test isbitstype(GPUSparseDeviceVector{Float32, Cint, CuDeviceVector{Cint, AS.Global}, CuDeviceVector{Float32, AS.Global}, AS.Global})

Copy link
Member

@maleadt maleadt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments.

@kshyatt kshyatt force-pushed the ksh/gpuarrays_sparse branch from cd339b2 to 6772970 Compare December 5, 2025 10:36
@kshyatt
Copy link
Member Author

kshyatt commented Dec 5, 2025

OK, I think I got everything. Some of the c-tors and conversion functions need both typevars due to the GPUArrays changes, but I kept the single parameter ones for backwards compatibility

@kshyatt
Copy link
Member Author

kshyatt commented Dec 5, 2025

Going to merge as this seems to be passing and has approval. -1k lines isn't too bad 😏

@kshyatt kshyatt merged commit 9eb1085 into master Dec 5, 2025
3 checks passed
@kshyatt kshyatt deleted the ksh/gpuarrays_sparse branch December 5, 2025 14:29
AntonOresten pushed a commit to MurrellGroup/CUDA.jl that referenced this pull request Dec 8, 2025
* Try integrating with the GPUArrays sparse migration

* Don't specify typevar unnecessarily

* comments

* Fix cusparse tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants