Refactor modules to avoid unnecessary imports

[djhoese]

Feature Request

Is your feature request related to a problem? Please describe.
I have a project that uses Satpy but does not need most of the "usual" functionality that Satpy provides. Specifically I need to read ABI and AHI input data, no resampling, no compositing, and write to a NetCDF file using the "awips_tiled" writer. This project involves creating a docker image so reducing the number of unnecessary dependencies installed can make a huge difference on the size of the docker image.

Additionally, the more imports needed to do basic functionality in Satpy means longer import times overall when the functionality using those imports isn't needed.

Describe the solution you'd like
I'd like to refactor a few key modules in Satpy into a series of submodules. As part of that I'd also like to remove "alias" imports as much as possible. I started this with the "satpy.writers" sub-package in #3122, but naming still needs to be finalized. Other modules that should be looked at:

• satpy/tests/utils.py - This one maybe doesn't make sense if we think that public users should not be importing stuff from tests. In my case I wanted to use the make_dataid function but the utils module was importing a lot of stuff that wasn't necessary.
• satpy/readers/__init__.py - Hopefully not as difficult as the others described here since most of the functionality is using builtin python features. Refactor reader imports #3127 and Refactor resampler module to submodules #3124
• satpy/composites/__init__.py - We have a ton of composite classes here. They should really be divided up. Refactor composites to separate modules #3157
• satpy/enhancements/__init__.py - All the generic builtin enhancements are in here and require too many imports (trollimage) that most don't use. Refactor enhancements #3155
• satpy/multiscene/__init__.py: These are all alias imports when I refactor multiscene.py into separate modules. They should be removed. Additionally _blend_funcs.py should be made public. Switch MultiScene blend functions to a public satpy.multiscene.blend_funcs module #3236
• satpy/writers/ - Refactor satpy.writers to separate modules #3122
• I'm sure others, but these are the ones I've been bumping into.

After these are refactored the __init__.py modules could get a __getattr__ function that produces a warning when someone tries to access the old import.

Describe any changes to existing user workflow
Users will need to change their imports.

[djhoese]

Care should be taken on imports that are used only for type annotations. In those cases it is likely that someone (likely me) was lazy and did a normal import when it was only used in typing. In those cases it can be changed from a normal import to:

import typing

if typing.TYPE_CHECKING:
    from trollimage.xrimage import XRImage

One case where I noticed this was satpy/enhancements/__init__.py.

[djhoese]

Oh another thing that could be done is added another CI environment (ubuntu probably) with a "minimal" set of dependencies. Another YAML environment file could be made for this. To me this would mean not having trollimage, rasterio, zarr, pooch, or pykdtree...and maybe others.

Edit: And of course this can't happen until the hard requirements of satpy are updated to not include these libraries (the conda-forge package has more hard requirements that the PyPI package), but that's a whole different discussion.

[pnuu]

First glance at readers.__init__.py at the airport:

• some top level imports from yaml_reader.py
• yaml_reader.py has top level imports from
• yaml
• pyresample.boundary
• various area def stuff from pyresample.geometry
• trollsift
• bunch of stuff from Satpy itself

I think

• things in readers/__yaml__.py could be moved to .core and/or .utils module(s)
  • FSFile could be module of its own, or in utils
• yaml_reader.py
  • trollsift.parser.globify is used in 2 methods, move the imports there
  • trollsift.parser.parse is used in 1 method, move the import there
  • AreaDefBoundary and Boundary are used in the same one method, move the imports
  • SwathDefinition is used in one method
  • StackedAreaDefinition is used in 5 methods
  • AreaDefinition is used in 1 method, move the import
  • satpy.aux_download.DataDownloadMixin is used as baseclass of FileYamlReader,
  • from satpy.resample import add_crs_xy_coords, get_area_def -> refactor satpy/resample.py
• satpy/resample.py
  • might need a a big: the pyresample.resampler.BaseResampler is used as a baseclass for the resampler classes. And the resampler classes (two of which are imported directly from Pyresample) are used in a global dict
  • from pyresample.geometry import SwathDefinition is not used, just remove the import

[djhoese]

In the long run I think I'm ok with pyresample and trollsift being hard requirements. Trollsift is small and used in reading (filename matching/parsing) and in writing (optionally to format filenames). Pyresample geometries are used in .attrs["area"] so that seems justified as a hard requirement.

I guess reducing imports until they are necessary would be nice though.

[pnuu]

I think the composite refactoring should wait until the RGB Workshop naming and recipe checks/changes so that the class references in the YAML files can be changed at the same time.

I'll think about anyway how the compositors could be split.

[pnuu]

Oh, and the same goes for enhancements.

[pnuu]

Public functions in enhancements/__init__.py:

• things that make grayscale to have colors
  • colorize
  • palettize
  • lookup
• stretching
  • stretch
  • gamma
  • invert
  • piecewise_linear_stretch
  • cira_stretch
  • reinhad_to_srgb
  • jma_true_color_reproduction
  • btemp_threshold
• convolution
  • three_d_effect
• colormap creation
  • create_colormap
• context managers
  • exclude_alpha
  • on_separate_bands
  • on_dask_array
  • using_map_blocks

Public functions in composites/__init__.py:

• check_times
• sub_arrays
• add_alpha_bands
• enhance2dataset
• add_bands
• zero_missing_data

Public classes in composites/__init__.py:

• exceptions
  • IncompatibleAreas
  • IncompatibleTimes
• compositors
  • basic
    • CompositeBase
    • SingleBandCompositor
    • GenericCompositor
    • RGBCompositor deprecated
    • StaticImageCompositor
  • arithmetic
    • DifferenceCompositor
    • RatioCompositor
    • SumCompositor
  • filling
    • FillingCompositor
    • Filler
    • MultiFiller
  • masking
    • MaskingCompositor
    • LongitudeMaskingCompositor
  • cloud "detection" -> move these to cloud_products.py
    • CloudCompositor
    • HighCloudCompositor
    • LowCloudCompositor
  • sharpening
    • RatioSharpenedRGB
    • SelfSharpenedRGB
    • LuminanceSharpeningCompositor
    • SandwichCompositor
  • B&W -> colors
    • ColormapCompositor deprecated
    • ColorizeCompositor deprecated
    • PaletteCompositor deprecated
  • stack something on top of something other
    • DayNightCompositor
    • BackgroundCompositor
    • HighlightCompositor in glm.py
  • SEVIRI-only
    • RealisticColors
  • other
    • NaturalEnh
    • CategoricalDataCompositor

[djhoese]

Nice job @pnuu! I don't remember the exact details of what the create_colormap function does, but I think a majority of the functionality was moved to trollimage in one or more methods on the Colormap:

https://trollimage.readthedocs.io/en/latest/api/trollimage.html#trollimage.colormap.Colormap.from_file

Maybe it is time the function in Satpy gets removed and we bump the minimal version of trollimage required?