Nanosecond timestamps are silently truncated even when Value("timestamp[ns]") is requested

[LeSingh1]

Describe the bug

Dataset.from_dict / from_list / map silently drop sub-microsecond precision, even when the user explicitly declares a nanosecond feature type. The resulting dataset reports timestamp[ns] but the data has already been truncated to microseconds.

Steps to reproduce the bug

import pandas as pd
from datasets import Dataset, Features, Value

ts = pd.Timestamp("2024-01-01 00:00:00.123456789")
ds = Dataset.from_dict({"t": [ts]}, features=Features({"t": Value("timestamp[ns]")}))
print(ds.features)   # {'t': Value('timestamp[ns]')}
print(ds[0]["t"])    # 2024-01-01 00:00:00.123456   <-- .789 ns lost

The same applies to Value("duration[ns]") via pd.Timedelta. Dataset.from_pandas on a datetime64[ns] column preserves the nanoseconds, so the two entry points disagree.

Cause

Two compounding steps:

1. src/datasets/features/features.py converts pd.Timestamp → datetime.datetime via to_pydatetime(), which only has microsecond resolution (pandas even emits UserWarning: Discarding nonzero nanoseconds in conversion).
2. More fundamentally, src/datasets/arrow_writer.py calls pa.array(cast_to_python_objects(examples, only_1d_for_numpy=True)) without passing the target type. PyArrow infers timestamp[us] from any datetime object, so the data is truncated before cast_array_to_feature casts the already-lossy array up to timestamp[ns].

Notes for whoever picks this up

I tried the obvious narrow fixes and neither is safe, so I'm filing rather than sending a patch:

• Skipping to_pydatetime() when obj.nanosecond != 0 changes nothing user-visible. Every caller feeds the result straight into pa.array(...) with no type, so pyarrow re-infers timestamp[us] and truncates anyway. The only effect is suppressing the pandas warning.
• Returning obj.to_datetime64() does make pyarrow infer timestamp[ns] losslessly, but it is not a safe drop-in: _cast_to_python_objects decides list-wide conversion from the first element only, so a column mixing nanosecond and non-nanosecond timestamps produces a mixed [np.datetime64, datetime.datetime] list and pyarrow raises ArrowInvalid: numpy.datetime64 scalars cannot be mixed with other Python scalar values. That turns a silent truncation into a crash. Making it uniform requires scanning whole columns, which is a perf cost on the hottest write path.

The real fix is probably to thread the known pa_type into pa.array() in TypedSequence._arrow_array, which is a core-writer change the maintainers should own.

Related: #8390 covers a separate temporal-precision problem in to_json.

Environment info

• datasets 5.0.2.dev0 (main @ b7cb10b)
• pyarrow 25.0.0, Python 3.12

[sophianggan]

#self-assign