Too many open files

[milankl]

Despite using NetCDF.jl v0.11 I just got this error

NetCDF error code 24:
	Too many open files

Stacktrace:
 [1] check at /home/kloewer/.julia/packages/NetCDF/2DiO1/src/netcdf_helpers.jl:22 [inlined]
 [2] nc_open(::String, ::UInt16, ::Array{Int32,1}) at /home/kloewer/.julia/packages/NetCDF/2DiO1/src/netcdf_c.jl:234
 [3] nc_open at /home/kloewer/.julia/packages/NetCDF/2DiO1/src/netcdf_helpers.jl:103 [inlined]
 [4] open(::String; mode::UInt16, readdimvar::Bool, add_finalizer::Bool) at /home/kloewer/.julia/packages/NetCDF/2DiO1/src/NetCDF.jl:911
 [5] open(::String) at /home/kloewer/.julia/packages/NetCDF/2DiO1/src/NetCDF.jl:911

while looping over several hundreds of those

ncfile = NetCDF.open(joinpath(path,"member$ie","ensemble.t.member$ie.step$s.ll.nc"))
t[ie,it] = ncfile.vars["t"][x0,y0,1]

How can there be too many open files when NetCDF.close is deprecated?

[milankl]

The NetCDF.ncread syntax with start and count does not seem to cause the same issues...

[visr]

Hmm so NetCDF.open has a finalizer attached, so it will be closed after it becomes unreachable and the GC is run. This is not really deterministic however, so I can imagine an explicit close still being useful here.

But perhaps what you'd want here is to use NetCDF.open in a do block? That closes directly after the do block ends.
https://juliageo.org/NetCDF.jl/dev/intermediate/#Open-a-file

[milankl]

Ah I see, so if I actually open many files in a short time (as I only pull little data from each), I can still end up in a too many open files situation... Great yeah, I'll use the do block then instead.

[visr]

Yeah indeed, there is no guarantee that you cannot hit that error.

But I'm thinking perhaps it would be good to un-deprecate NetCDF.close (it is NetCDF._close now if you still want it). To prevent issues with closing twice, we'd need something like this: JuliaGeo/NCDatasets.jl#93