Skip to content

Conversation

@pierreglaser
Copy link

during unpickling, in order to recreate the previous environnement an object lived in,
the module under the base_globals name is re-imported. However, it
is not always possible, especially if base_globals pointed to a
temporary or locally defined module. This PR introduces a fix that
checks if the module still exists during unpickling.

@pierreglaser
Copy link
Author

pierreglaser commented Sep 6, 2018

More precisely:

base_globals contains the name of the module the pickled object lived in.

Initially, in the case of interactively defined functions, #187 allowed the
creation the global variables of the pickled function that do not exist when unpickling,
but not overriding the ones that already exist in __main__. It was done by doing

base_globals = vars(sys.modules[base_globals])

and then updating func.__globals_.

(in this case base_globals is __main__)

Extending it to other modules can cause errors for cases when the module does
not exist in sys.modules.
Namely:

  1. the instanciation of a new namedtuple subclass (the __new__ method is created
    by it is done calling exec with hardcoded globals argument,
    including the module name __name__ that is in this case
f'namedtuple_{typename}'
  1. using dynamic modules or faulty modules that are not referenced in
    sys.modules
  2. using local modules, that can be imported because '' is in sys.path

For 1 and 2, I am not sure there is a need to handle gloabals updating,
as it looks to me that these modules are aimed to be created on the fly, saved, and then thrown
away or untouched until they get reloaded

For 3, the fix in this PR checks if base_globals exists in sys.modules during
unpickling.

ogrisel and others added 14 commits September 11, 2018 15:04
during unpickling, in order to recreate the previous environnement an object lived in,
the module under the base_globals name is re-imported. However, it
is not always possible, especially if base_globals pointed to a
temporary or locally defined module. This PR introduces a fix that
checks if the module still exists during unpickling
two main fixes here:
* added a _BASE_GLOBALS_CACHE, that will track a functions globals in the
environnement where it is loaded

* fixed a python 2 issue due to dynamic_subimport
this modification was done to prevent side effects, and affectation
of cloudpickle's global variable, (here, _dynamic_modules).
Now, those global variables are only changed in ephemere child
processes
dynamic modules globals are not stored in a dict.
Each time a function from a dynamic module gets unpickled,
a new set of globals is created
@pierreglaser pierreglaser force-pushed the more-func-base-globals-fixes branch from f76c3c2 to da966e4 Compare September 11, 2018 13:07
@ogrisel
Copy link
Owner

ogrisel commented Sep 13, 2018

Fixed by cloudpipe#198.

@ogrisel ogrisel closed this Sep 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants