Release v0.9.3

docs/examples/05_caching.ipynb

@@ -113,11 +113,21 @@
    "outputs": [],
    "source": [
     "# read netcdf with layer model data\n",
-    "layer_model = xr.open_dataset(\n",
-    "    os.path.join(cachedir, \"layer_test.nc\"), mask_and_scale=False\n",
+    "layer_model_from_cache = xr.open_dataset(\n",
+    "    os.path.join(cachedir, \"layer_test.nc\"), mask_and_scale=False, decode_coords='all'\n",
     ")"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# compare cache with original\n",
+    "assert layer_model_from_cache.equals(layer_model)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -131,7 +141,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "layer_model = nlmod.read.regis.get_combined_layer_models(\n",
+    "layer_model2 = nlmod.read.regis.get_combined_layer_models(\n",
     "    extent=[95000.0, 105000.0, 494000.0, 500000.0],\n",
     "    use_geotop=False,\n",
     "    cachedir=cachedir,\n",
@@ -158,10 +168,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "This is the flowchart of an ordinary function call:\n",
+    "This is the flowchart of an ordinary function call:<br>\n",
     "![image function call](img/ordinary_function_call.png)\n",
     "\n",
-    "This is the flowchart of a function call using the caching from nlmod:\n",
+    "This is the flowchart of a function call using the caching from nlmod:<br>\n",
     "![image cache function call](img/cache_function_call.png)"
    ]
   },
@@ -190,8 +200,8 @@
     "## Checking the cache\n",
     "One of the steps in the caching process is to check if the cache was created using the same function arguments as the current function call. This check has some limitations:\n",
     "\n",
-    "- Only function arguments with certain types are checked. These types include: int, float, bool, str, bytes, list, tuple, dict, numpy.ndarray, xarray.DataArray and xarray.Dataset. If a function argument has a different type the cache is never used. In future development more types may be added to the checks.\n",
-    "- If one of the function arguments is an xarray Dataset the check is somewhat different. For a dataset we only check if it has identical dimensions and coordinates as the cached netcdf file. There is no check if the variables in the dataset are identical.\n",
+    "- Only function arguments with certain types are checked. These types include: int, float, bool, str, bytes, list, tuple, dict, numpy.ndarray, xarray.DataArray, pandas.DataFrame, pandas.Series and xarray.Dataset. If a function argument has a different type the cache is never used. In future development more types may be added to the checks.\n",
+    "- If one of the function arguments is an xarray Dataset the check is somewhat different. Not all the data in a dataset it checked as it would take to much time. Therefore we use the `nlmod.cache.ds_contains` function to specify which coordinates, data variables and attributes should be checked. The arguments for this are specified in the cache decorator. Users of nlmod do not have to worry about these.\n",
     "- It is not possible to cache the results of a function with more than one xarray Dataset as an argument. This is due to the difference in checking datasets. If more than one xarray dataset is given the cache decoraters raises a TypeError.\n",
     "- If one of the function arguments is a filepath of type str we only check if the cached filepath is the same as the current filepath. We do not check if any changes were made to the file after the cache was created.\n",
     "\n",
@@ -220,7 +230,7 @@
    "source": [
     "## Clearing the cache\n",
     "\n",
-    "Sometimes you want to get rid of all the cached files to free disk space or to support your minimalistic lifestyle. You can use the `clear_cache` function to clear all cached files in a specific cache directory."
+    "Sometimes you want to get rid of all the cached files to free disk space or to support your minimalistic lifestyle. You can use the `clear_cache` function to clear all cached files in a specific cache directory. Note: when calling this function an extra confirmation [Y/N] is required to actually delete all the cache files."
    ]
   },
   {
@@ -349,14 +359,38 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Properties\n",
+    "### Notes\n",
     "\n",
     "1. All function arguments are pickled and saved together with the netcdf file. If the function arguments use a lot of memory this process can be become slow. This should be taken into account when you decide to use caching.\n",
     "2. Function arguments that cannot be pickled using the `pickle` module raise an error in the caching process.\n",
-    "3. A function with mutable function arguments that are modified during function execution should not be used in caching. It can be used but the cache will never be used. The check on function arguments will always be False since the original function arguments are compared with the modified function argument.\n",
-    "4. If one of the function arguments is an xarray Dataset we only check if the dataset has the same dimensions and coordinates as the cached netcdf file. There is no check on the variables (DataArrays) in the dataset because it would simply take too much time to check all the variables in the dataset. Also, most of the time it is not necessary to check all the variables as they are not used to create the cached file. There is one example where a variable from the dataset is used to create the cached file. The `nlmod.read.jarkus.get_bathymetry` uses the 'Northsea' DataArray to create a bathymetry dataset. When we access the 'Northsea' DataArray using `ds['Northsea']` in the `get_bathymetry` function there would be no check if the 'Northsea' DataArray that was used to create the cache is the same as the 'Northsea' DataArray in the current function call. The current solution for this is to make the 'Northsea' DataArray a separate function argument in the `get_bathymetry` function. This makes it also more clear which data is used in the function.\n",
-    "5. There is a check to see if the module where the function is defined has been changed since the cache was created. This helps not to use the cache when changes are made to the function. Unfortunately when the function uses other functions from different modules these other modules are not checked for recent changes.\n",
-    "6. The `cache_netcdf` decorator uses `functools.wraps` and some homemade magic to add properties, such as the name and the docstring, of the original function to the decorated function. This assumes that the original function has a docstring with a \"Returns\" heading. If this is not the case the docstring is not modified."
+    "3. If one of the function arguments is an xarray Dataset the argument check is somewhat different. Not all the data in a dataset it checked as it would take to much time. Therefore we use the `nlmod.cache.ds_contains` function to specify which coordinates, data variables and attributes should be checked. The arguments for this are specified in the cache decorator.\n",
+    "4. There is a check to see if the module where the function is defined has been changed since the cache was created. This helps not to use the cache when changes are made to the function. Unfortunately when the function uses other functions from different modules these other modules are not checked for recent changes.\n",
+    "5. The `cache_netcdf` decorator uses `functools.wraps` and some homemade magic to add properties, such as the name and the docstring, of the original function to the decorated function. This assumes that the original function has a docstring with a \"Returns\" heading. If this is not the case the docstring is not modified.\n",
+    "6. There are some additional options for caching in the NLMOD_CACHE_OPTIONS. The options are:\n",
+    "    - `nc_hash`: Save a hash of the stored netcdf file with the function arguments to ensure that the stored netcdf file and the pickled function arguments belong together. Default is True\n",
+    "    - `dataset_coords_hash`: Save a hash of the dataset coordinates with the function arguments and check against the dataset coordinates that are used to call the function. This is to ensure that the cached file was created using the same dataset coordinates as the current function call, default is True.\n",
+    "    - `dataset_data_vars_hash`: Save a hash of the data variables with the function arguments and check against the dataset variables that are used to call the function. This is to ensure that the cached file was created using the same dataset variables as the current function call, default is True.\n",
+    "    - `explicit_dataset_coordinate_comparison`: If the `dataset_coords_hash` and `dataset_data_vars_hash` do not work (for example see this issue: https://github.com/gwmod/nlmod/issues/389) it is adviced to use an explicit check to see if the cached file was created using the same dataset coordinates as the current function call. This can be done with this settings, the default is False."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# show cache options\n",
+    "nlmod.config.NLMOD_CACHE_OPTIONS"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# set cache option\n",
+    "nlmod.config.set_options(nc_hash=False)"
    ]
   },
   {
@@ -370,8 +404,22 @@
   }
  ],
  "metadata": {
+  "kernelspec": {
+   "display_name": "nlmod",
+   "language": "python",
+   "name": "python3"
+  },
   "language_info": {
-   "name": "python"
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.4"
   }
  },
  "nbformat": 4,

nlmod/__init__.py

@@ -3,7 +3,7 @@
 
 NLMOD_DATADIR = os.path.join(os.path.dirname(__file__), "data")
 
-from . import dims, gis, gwf, gwt, modpath, plot, read, sim, util
+from . import config, dims, gis, gwf, gwt, modpath, plot, read, sim, util
 from .dims import base, get_ds, grid, layers, resample, time, to_model_ds
 from .util import download_mfbinaries
 from .version import __version__, show_versions