time decoding error with "days since"

[rabernat]

I am trying to use xray with some CESM POP model netCDF output, which supposedly follows CF-1.0 conventions. It is failing because the models time units are "'days since 0000-01-01 00:00:00". When calling open_dataset, I get the following error:

ValueError: unable to decode time units u'days since 0000-01-01 00:00:00' with the default calendar. Try opening your dataset with decode_times=False. Full traceback:
Traceback (most recent call last):
  File "/home/rpa/xray/xray/conventions.py", line 372, in __init__
    # Otherwise, tracebacks end up swallowed by Dataset.__repr__ when users
  File "/home/rpa/xray/xray/conventions.py", line 145, in decode_cf_datetime
    dates = _decode_datetime_with_netcdf4(flat_num_dates, units, calendar)
  File "/home/rpa/xray/xray/conventions.py", line 97, in _decode_datetime_with_netcdf4
    dates = np.asarray(nc4.num2date(num_dates, units, calendar))
  File "netCDF4/_netCDF4.pyx", line 4522, in netCDF4._netCDF4.num2date (netCDF4/_netCDF4.c:50388)
  File "netCDF4/_netCDF4.pyx", line 4337, in netCDF4._netCDF4._dateparse (netCDF4/_netCDF4.c:48234)
ValueError: year is out of range

Full metadata for the time variable:

    double time(time) ;
        time:long_name = "time" ;
        time:units = "days since 0000-01-01 00:00:00" ;
        time:bounds = "time_bound" ;
        time:calendar = "noleap" ;

I guess this is a problem with the underlying netCDF4 num2date package?

[rabernat]

In fact I just found a netCDF issue on this topic! Apparently they don't think it should be supported. Unidata/netcdf4-python#442

[jhamman]

@rabernat -

Yes - this is all coming from the netCDF4.netcdftime module.

The work around with xray is to use ds = xray.open_dataset(filename, decode_times=False) then to fix up the time variable "manually". You can use xray.decode_cf() or simply assign a new pandas time index to your time variable.

As an aside, I also work with CESM output and this is a common problem with its netCDF output.

[ocefpaf]

@jhamman consider using cf_units in xray 😉

(See Unidata/netcdf4-python#442 (comment))

[rabernat]

The PR above fixes this issue. However, since my model years are in the range 100-200, I am still getting the warning

RuntimeWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using dummy netCDF4.datetime objects instead, reason: dates out of range

and eventually when I try to access the time data, an error with a very long stack trace ending with

pandas/tslib.pyx in pandas.tslib.Timestamp.__new__ (pandas/tslib.c:7638)()
pandas/tslib.pyx in pandas.tslib.convert_to_tsobject (pandas/tslib.c:21232)()
pandas/tslib.pyx in pandas.tslib._check_dts_bounds (pandas/tslib.c:23332)()
OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 100-02-01 00:00:00

I see there is a check in conventions.py that the year has to lie between 1678 and 2226. What is the reason for this?

[jhamman]

We try to cast all the time variables to a pandas time index. This gives xray the ability to use many of the fast and fancy timeseries tools that pandas has. One consequence of that is that non-standard calendars, such as the "noleap" calendar must have dates inside the valid range of the standard calendars (1678 and 2226).

Does that make since? Ideally, numpy and pandas would support custom calendars but they don't so, at this point, we're bound to there limits.

[rabernat]

@jhamman Thanks for the clear explanation! One of the main uses for non-standard calendars would be climate model "control runs", which don't occur any any specific point in historical time but still have seasonal cycles, well defined months, etc. It would be nice to have "group by" functionality for these datasets. But I do see how this is impossible with the current numpy datetime64 datatype. Perhaps the long term fix is to implement non-standard calendars within numpy itself.

[jhamman]

Perhaps the long term fix is to implement non-standard calendars within numpy itself.

I agree, although that sounds like quite an undertaking. Maybe raise an issue over at numpy and ask if they would be interested in a multi-calendar api? If numpy could make it work, then I'm sure pandas could as well.