PestSolver pypestworker raises a `TypeError: cannot pickle 'TextIOWrapper' instances`

[cnicol-gwlogic]

Example notebook pest_pastas_ies.ipynb and pest_pastas_glm_hp.ipynb both fail when using the default solver argument use_pypestworker=True. Windows 11 machine(s). The (rather long) traceback is below. Any ideas what I have wrong? Ta.

[C:\Python313\Lib\site-packages\pastas_plugins\pest\solver.py:387](file:///C:/Python313/Lib/site-packages/pastas_plugins/pest/solver.py#line=386): SyntaxWarning: invalid escape sequence '\s'
[C:\Python313\Lib\site-packages\pastas_plugins\pest\solver.py:506](file:///C:/Python313/Lib/site-packages/pastas_plugins/pest/solver.py#line=505): SyntaxWarning: invalid escape sequence '\s'
[C:\Python313\Lib\site-packages\pastas_plugins\pest\solver.py:512](file:///C:/Python313/Lib/site-packages/pastas_plugins/pest/solver.py#line=511): SyntaxWarning: invalid escape sequence '\s'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[5], line 16
     14 # ml_ies.set_parameter("constant_d", initial=27.5, vary=False)
     15 ml_ies.add_solver(solver)
---> 16 ml_ies.solver.run_ensembles()
     17 ml_ies.solver.nfev = solver.noptmax

File [C:\Python313\Lib\site-packages\pastas_plugins\pest\solver.py:681](file:///C:/Python313/Lib/site-packages/pastas_plugins/pest/solver.py#line=680), in PestIesSolver.run_ensembles(self, ies_add_base, par_sigma_range, observation_noise_standard_deviation, observation_noise_correlation_coefficient, ies_parameter_ensemble_method, pestpp_options, silent)
    677 pst.pestpp_options.update(pestpp_options)
    679 self.write_pst(pst=pst, version=2)
--> 681 pyemu.os_utils.start_workers(
    682     worker_dir=self.temp_ws,  # the folder which contains the "template" PEST dataset
    683     exe_rel_path=self.exe_name.name,  # the PEST software version we want to run
    684     pst_rel_path="pest.pst",  # the control file to use with PEST
    685     num_workers=self.num_workers,  # how many agents to deploy
    686     worker_root=self.master_ws.parent,  # where to deploy the agent directories; relative to where python is running
    687     master_dir=self.master_ws,  # the manager directory
    688     port=self.port_number,  # the port to use for communication
    689     verbose=silent,
    690     silent_master=silent,
    691     reuse_master=self.use_pypestworker,
    692     ppw_function=self.ppw_function
    693     if self.use_pypestworker
    694     else None,  # the function to run in the agent
    695     ppw_kwargs={"ml": self.ml}
    696     if self.use_pypestworker
    697     else {},  # the arguments to pass to the ppw_function
    698 )
    700 phidf = pd.read_csv(self.master_ws / "pest.phi.meas.csv", index_col=0)
    701 self.nfev = phidf.index[-1]

File [C:\Python313\Lib\site-packages\pyemu\utils\os_utils.py:412](file:///C:/Python313/Lib/site-packages/pyemu/utils/os_utils.py#line=411), in start_workers(worker_dir, exe_rel_path, pst_rel_path, num_workers, worker_root, port, rel_path, local, cleanup, master_dir, verbose, silent_master, reuse_master, restart, ppw_function, ppw_kwargs)
    410     for i in range(num_workers):
    411         p = mp.Process(target=ppw_function,args=args,kwargs=ppw_kwargs)
--> 412         p.start()
    413         procs.append(p)
    415 else:

File [C:\Python313\Lib\multiprocessing\process.py:121](file:///C:/Python313/Lib/multiprocessing/process.py#line=120), in BaseProcess.start(self)
    118 assert not _current_process._config.get('daemon'), \
    119        'daemonic processes are not allowed to have children'
    120 _cleanup()
--> 121 self._popen = self._Popen(self)
    122 self._sentinel = self._popen.sentinel
    123 # Avoid a refcycle if the target function holds an indirect
    124 # reference to the process object (see bpo-30775)

File [C:\Python313\Lib\multiprocessing\context.py:224](file:///C:/Python313/Lib/multiprocessing/context.py#line=223), in Process._Popen(process_obj)
    222 @staticmethod
    223 def _Popen(process_obj):
--> 224     return _default_context.get_context().Process._Popen(process_obj)

File [C:\Python313\Lib\multiprocessing\context.py:337](file:///C:/Python313/Lib/multiprocessing/context.py#line=336), in SpawnProcess._Popen(process_obj)
    334 @staticmethod
    335 def _Popen(process_obj):
    336     from .popen_spawn_win32 import Popen
--> 337     return Popen(process_obj)

File [C:\Python313\Lib\multiprocessing\popen_spawn_win32.py:97](file:///C:/Python313/Lib/multiprocessing/popen_spawn_win32.py#line=96), in Popen.__init__(self, process_obj)
     95 try:
     96     reduction.dump(prep_data, to_child)
---> 97     reduction.dump(process_obj, to_child)
     98 finally:
     99     set_spawning_popen(None)

File [C:\Python313\Lib\multiprocessing\reduction.py:60](file:///C:/Python313/Lib/multiprocessing/reduction.py#line=59), in dump(obj, file, protocol)
     58 def dump(obj, file, protocol=None):
     59     '''Replacement for pickle.dump() using ForkingPickler.'''
---> 60     ForkingPickler(file, protocol).dump(obj)

TypeError: cannot pickle 'TextIOWrapper' instances

[martinvonk]

Hi @cnicol-gwlogic,

Thank you for testing the PestSolver. As you noticed, it is still very much under development (on my personal Linux computer 🐧).

The way to solve this issue, I think, is to use: multiprocessing.set_start_method("fork") statement before calling the solve method. The "fork" backend is the default on Linux (changed in Python 3.14), but not on MacOS and Windows. That's why I never noticed this issue on Linux. You might also need a if __name__ == "__main__": statement to ensure memory safety when using Python's parallel processing methods on Windows.

if __name__ == "__main__":
    multiprocessing.set_start_method("fork")

I'm not entirely sure if this is an issue for the PyEmu pypestworker or the Pastas PestSolver. And this also changes again with Python 3.14 for Linux. So let's leave this issue open for now and check:

• Is this an Pyemu or Pastas issue?
  Answer: Pastas
• What happens with Python 3.14
• Add a warning in pastas-plugins or the notebooks for other OS's

[cnicol-gwlogic]

Thanks so much for the quick response @martinvonk, really appreciate it (along with this PestSolver business - great stuff). It seems "fork" method is not available on Windows, only "spawn". However, I think I've resolved this really simply now - it seems that passing the pastas.Model object to run_pypestworker() is the culprit. To resolve:
In PestSolver, change pypestworker function to this:

def run_pypestworker(
    pst: str | pyemu.Pst,
    host: int,
    port: int,
    ml_dict: dict,
    parameter_index: dict,
) -> None:
    from pastas.io.base import (load as load_model, _load_model)

    ppw = pyemu.os_utils.PyPestWorker(
        pst=pst,
        host=host,
        port=port,
        verbose=False,
    )
    # load pastas model
    ml = _load_model(ml_dict) #load_model(ml_file)

    pvals = ppw.get_parameters()
    if pvals is None:
        return None

    while True:
        for pname, val in pvals.items():
            pname = parameter_index[pname]
            #pname = pname.split(":")[-1] if ":" in pname else pname
            #pname = pname.replace("_g", "_A") if pname.endswith("_g") else pname
            #pname = pname.replace("wellmodel","WellModel")
            ml.set_parameter(pname, optimal=val)
        sim = ml.simulate()
        obsvals = sim.loc[ml.observations().index]
        obsvals.index = ppw._pst.observation_data.index
        ppw.send_observations(obsvals=obsvals)
        pvals = ppw.get_parameters()
        if pvals is None:
            break

And for example in the IesSolver:

        pyemu.os_utils.start_workers(
            worker_dir=self.temp_ws,  # the folder which contains the "template" PEST dataset
            exe_rel_path=self.exe_name.name,  # the PEST software version we want to run
            pst_rel_path="pest.pst",  # the control file to use with PEST
            num_workers=self.num_workers,  # how many agents to deploy
            worker_root=self.master_ws.parent,  # where to deploy the agent directories; relative to where python is running
            master_dir=self.master_ws,  # the manager directory
            port=self.port_number,  # the port to use for communication
            verbose=silent,
            silent_master=silent,
            reuse_master=self.use_pypestworker,
            ppw_function=self.ppw_function
            if self.use_pypestworker
            else None,  # the function to run in the agent
            ppw_kwargs={
                "ml_dict": self.ml.to_dict(),
                "parameter_index": self.parameter_index
                }
            if self.use_pypestworker
            else {},  # the arguments to pass to the ppw_function
        )

Then it runs ok, even on the inferior person's OS :)

I'm working on some significant additions to your PestSolver actually; I can add the above edits to that PR if its not already been addressed before I submit. Thanks again.

[martinvonk]

Awesome. I'll test your method. I'm afraid it might be a little slower since you have to read the pastas.Model from a file. But I can test that. If it's not much slower, your method is the way to go I think.

If it is much slower, I might have to dig a little bit deeper into the error message. Because I think it is a bit strange still, since you can just pickle and unpickle a pastas Model fine.

[cnicol-gwlogic]

@martinvonk - better fix might be to ml.to_dict() and pass that to run_pypestworker() instead of the model.pas file. Edited above fyi.

I did also find an issue with the pname replacement stuff in run_pypestworker(). I added pval = parameter_index[pname] dict argument above to sort that - works now.

[raoulcollenteur]

Thanks for reporting @cnicol-gwlogic ! Just to add for future reference, I have the same issue on MacOS, and the solution multiprocessing.set_start_method("fork") works for me. I also checked the solution above using ml.to_dict, which also works here.

[martinvonk]

Only one thing that happens as well with the to_dict method, or reading the model from file, is that not everything in Pastas has a to_dict method. So you might not get exactly the same model. But I think only some solvers at the moment don't have this method so we should be fine I think.

[raoulcollenteur]

Only one thing that happens as well with the to_dict method, or reading the model from file, is that not everything in Pastas has a to_dict method. So you might not get exactly the same model. But I think only some solvers at the moment don't have this method so we should be fine I think.

Which parts do not have a to_dict? That should actually not be case. So if you know of any please open a big issue on Pastas :)

[martinvonk]

The Pest Solvers don't have a to_dict for instance, and the EmceeSolver, for instance, in Pastas doesn't have one either. So that should be fine, since we only simulate the Model in the Pest backend, for which a solver is not needed. But the model used in the Pest backend is not 100% the same, but the components used for simulating the model are the same.

[cnicol-gwlogic]

@martinvonk @raoulcollenteur - fyi, I think the issue here with pastas.Model is that pickle (apparently used by multiprocessing in "spawn" mode, but not in "fork" mode) cannot pickle active logger instances. The solution is just a matter of including something like this in pastas.Model:

    def __getstate__(self):
        # Exclude the logger and its handlers from the state to be pickled
        state = self.__dict__.copy()
        if 'logger' in state:
            del state['logger']
        return state

    def __setstate__(self, state):
        # Reconstruct the logger after unpickling
        self.__dict__.update(state)
        logger = getLogger(__name__)

Note however that for PestSolver I'm now using the "dill" :) package rather than pickle anyway, as this issue affected other used classes (pyemu for example), not just Pastas.Model. Pyemu.pst_utils (used by pypestworker via Pyemu.pst.pst_handler) still causes pickling to fail (because multiprocessing uses pickle as its Pickler) - there's a lambda function somewhere in there that it really does not like. Still have to look into that...urr. I tried overriding the multiprocessing pickler with dill in pyemu but it then causes some other issues...gawd.

[martinvonk]

Hi @cnicol-gwlogic,

That looks like a nice solution, but the Pastas Model does not have a logger as an attribute any more. So it would be weird if this solves the issue. But I don't know enough about logging in Python so I'll discuss this with the rest of the team.

[cnicol-gwlogic]

@martinvonk yep - I noticed it working the other day (after also messing with a bunch of other stuff in my pest plugin setup) - so maybe I have it wrong. Will report back once I get back into this work. Cheers.