Errors when loading in python3 a file dumped in python2

[tayciryahmed]

Hi,

I am trying to load in python3.7 a file - sklearn pipeline that was serialised using dill.dump in python2.7.
I am following this tutorial: https://rebeccabilbro.github.io/convert-py2-pickles-to-py3/

I get the below error :

import dill, pickle 
dill._dill._reverse_typemap["ObjectType"] = object
with open(sklearn_pipeline, "rb") as f:
    loaded = pickle.load(f, encoding="latin1")

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'dill.dill'

dill.__version__
'0.3.3'

Note that loading a simple sklearn classifier, works well with this snippet.

=> I tried to downgrade dill to v0.2.5, I get the following error:

pip install dill==0.2.5
import dill, pickle 
dill.dill._reverse_typemap["ObjectType"] = object
with open(sklearn_pipeline, "rb") as f:
       loaded = pickle.load(f, encoding="latin1")

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/local/lib/python3.7/site-packages/dill/dill.py", line 503, in _load_type
    return _reverse_typemap[name]
KeyError: 'TypeType'

Then, I tried to set dill.dill._reverse_typemap["TypeType"] = type I get the below error:

pip install dill==0.2.5
import dill, pickle 
dill.dill._reverse_typemap["ObjectType"] = object
dill.dill._reverse_typemap["TypeType"] = type
with open(sklearn_pipeline, "rb") as f:
       loaded = pickle.load(f, encoding="latin1")

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: an integer is required (got type str)

I also tried the bytes encoding and it returns the same error: TypeError: an integer is required (got type str).

Using dill==0.3.3:

import dill
dill._dill._reverse_typemap["ClassType"] = type
dill._dill._reverse_typemap["TypeType"] = type
dill._dill._reverse_typemap["ObjectType"] = object
with open(sklearn_pipeline, "rb") as model_file:
    u = dill.Unpickler(model_file)
    u.encoding = "latin1"
    loaded = u.load()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 481, in load
    obj = StockUnpickler.load(self)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x83 in position 32: ordinal not in range(128)

import dill
dill._dill._reverse_typemap["TypeType"] = type
dill._dill._reverse_typemap["ObjectType"] = object
with open(sklearn_pipeline, "rb") as f:
    loaded = dill.load(f, encoding="latin1")

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 278, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 481, in load
    obj = StockUnpickler.load(self)
TypeError: an integer is required (got type str)

Any ideas on how to solve this ?

[mmckerns]

This is unsupported. It can work... but it's unsupported. First off, it appears that the object that was serialized was with an old dill, which moved from dill.dill to dill._dill, as you surmised. You could try to rename the _dill.py file to dill.py and go with a more recent version of dill... but I have no idea if that would work. Essentially, you are going to be dealing with every incompatible difference between python2 and python3 as the objects are unpickled, and you'll need to manually convert the python2 object to a python3 object by registering a custom deserialization method in the typemap. Not impossible, but difficult and tedious. Good luck.

[tayciryahmed]

Thanks for the quick reply.
I decomposed my picked pipeline to track the object that is causing this error TypeError: an integer is required (got type str). I found out that pickling and un-pickling the custom classes causes the same error. So, I have tried the following steps:

In python2.7 and dill==0.3.3:

import dill

class test(object):
    def __init__(self): pass
    def fit(self, x): return x

t = test()

f = open('test', 'wb')
dill.dump(t, f)
f.close()

Then in python3.7 and dill==0.3.3:

import dill 

dill._dill._reverse_typemap["ClassType"] = type
dill._dill._reverse_typemap["TypeType"] = type
dill._dill._reverse_typemap["ObjectType"] = object

t = dill.load(open('test', 'rb'))
t.fit(1)

XXX lineno: 3, opcode: 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in fir
SystemError: unknown opcode

I get the above error SystemError: unknown opcode. I also tried setting the encoding to latin1 and bytes, it returns the same error.

Any ideas on how to debug this?