Skip to content

GeoPandas: Improve the interger overflow check using np.iinfo #3518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
seisman merged 2 commits into from
Oct 16, 2024
Merged

GeoPandas: Improve the interger overflow check using np.iinfo #3518

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0922600
GeoPandas: Improve the interger overflow check using np.iinfo
seisman Oct 15, 2024
7d7f747
Merge branch 'main' into int32-overflow
seisman Oct 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions pygmt/helpers/tempfile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,9 @@ def tempfile_from_geojson(geojson):
# 32-bit integer overflow issue. Related issues:
# https://github.com/geopandas/geopandas/issues/967#issuecomment-842877704
# https://github.com/GenericMappingTools/pygmt/issues/2497

int32_info = np.iinfo(np.int32)

if Version(gpd.__version__).major < 1: # GeoPandas v0.x
# The default engine 'fiona' supports the 'schema' parameter.
if geojson.index.name is None:
Expand All @@ -151,7 +154,10 @@ def tempfile_from_geojson(geojson):
schema = gpd.io.file.infer_schema(geojson)
for col, dtype in schema["properties"].items():
if dtype in {"int", "int64"}:
overflow = geojson[col].abs().max() > 2**31 - 1
overflow = (
geojson[col].max() > int32_info.max
or geojson[col].min() < int32_info.min
)
schema["properties"][col] = "float" if overflow else "int32"
geojson[col] = geojson[col].astype(schema["properties"][col])
ogrgmt_kwargs["schema"] = schema
Expand All @@ -160,7 +166,10 @@ def tempfile_from_geojson(geojson):
# but we can change the dtype directly.
for col in geojson.columns:
if geojson[col].dtype.name in {"int", "int64", "Int64"}:
overflow = geojson[col].abs().max() > 2**31 - 1
overflow = (
geojson[col].max() > int32_info.max
or geojson[col].min() < int32_info.min
)
dtype = "float" if overflow else "int32"
geojson[col] = geojson[col].astype(dtype)
# Using geopandas.to_file to directly export to OGR_GMT format
Expand Down