run_classifier.py: error: unrecognized arguments: --load_model

[Joerg99]

I'd like to classify my data with a pretrained model.
I followed the instructions on the readme page and tried to run one of these commands:
python3 run_classifier.py --load_model ama_sst.pt # classify Binary SST
python3 run_classifier.py --load_model ama_sst_16.pt --fp16 # run classification in fp16
python3 run_classifier.py --load_model ama_sst.pt --text-key --data <path.csv> # classify your own dataset
But it caused this error: run_classifier.py: error: unrecognized arguments: --load_model

[raulpuric]

Oh sorry, that's a README typo that I haven't changed.
it should be just --load.

[Joerg99]

Thanks! It's working now.
I'm using this to classify:
python3 run_classifier.py --load transformer_semeval.clf --data test2.csv --model "transformer"
I wonder why the results are not what I expected:
i love my life, everything is so great --> ['anticipation']
bad mood, bad weather. everything is shit! --> ['joy']
i love you --> []
this is so bad --> []
Worst movie of all times -- > ['anticipation', 'joy', 'trust']

Do you have any ideas how to improve the quality in terms of preprocessing? Is there a minimum or maximum length input length?
Also, what do you think about domain adaption. Imagine you use the model on old poems17th, 18th century) with many uncommon words. What performance would you expect?

[dadelani]

I tried running the following script
python3 run_classifier.py --load model/transformer_sst.clf --data data/binary_sst/test.csv --model "transformer"

It runs smoothly but the result is not accurate for the transformer model, it only predicts the same class for all the reviews in binary_sst/test.csv. However, mLSTM model gives good result

did any other person experienced this? many thanks @raulpuric @Joerg99

[MichaMucha]

@dadelani same experience with the pre-trained transformer_sst.clf
All results coming back with probability around 0.50 and all the same class.

[raulpuric]

that's rather peculiar, I'll be sure to take a look as soon as I can.

[MichaMucha]

Thank you @raulpuric
Also a note on the mLSTM pretrained language model, when I load it, it tells me about a tensor dimension mismatch on one axis. The pickled model has 257 while the code expects 256. I wouldn't expect to fix it, but interestingly, changing this line to 257 let me load the model.

sentiment-discovery/generate.py

Line 70 in fa52435

args.data_size = 256

Here's what invoked the error:

michalmucha$ python -i generate.py --load_model pretrained/mlstm.pt
Creating mlstm
Traceback (most recent call last):
  File "generate.py", line 92, in <module>
    model.load_state_dict(sd)
  File "/Users/michalmucha/Python/reddit/sentiment-discovery/model/model.py", line 56, in load_state_dict
    self.decoder.load_state_dict(state_dict['decoder'], strict=strict)
  File "/Developer/anaconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 769, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Linear:
        size mismatch for weight: copying a param with shape torch.Size([257, 4096]) from checkpoint, the shape in current model is torch.Size([256, 4096]).
        size mismatch for bias: copying a param with shape torch.Size([257]) from checkpoint, the shape in current model is torch.Size([256]).

Unfortunately after the change, the model generates a sequence of random chars.

Thanks for publishing this work

[raulpuric]

ahhh that makes sense, the mLSTM was trained with an older version of the tokenizer I suspect that all these innacuracy issues are due to the model vocab mismatching the tokenizer by 1 position.

[raulpuric]

ok to fix your random string problem I think it's because the vocab is 257 but python expects characters to be 0-255 for chr, and ord. To fix this you could manually place increment/decrement tokens by 1 where appropriate. Alternatively you can use our CharacterLevelTokenizer class which should handle this for you automatically.

[raulpuric]

@MichaMucha @dadelani I think that's a lack of documentation on my part. I think it's because you're not running with sentencepiece tokenization as in https://github.com/NVIDIA/sentiment-discovery#training-language-models--distributedfp16-training. Try adding these arguments: --tokenizer-type SentencePieceTokenizer --vocab-size 32000 \ #train a transformer model with our sentencepiece tokenization --tokenizer-type bpe --tokenizer-path ama_32k_tokenizer.model --model transformer \ --decoder-layers 12 --decoder-embed-dim 768 --decoder-ffn-embed-dim 3072 \ --decoder-learned-pos --decoder-attention-heads 8

Let me know if this works and I'll add it to the readme

[MichaMucha]

@raulpuric thanks for looking into it.

Unfortunately the binary classifier transformer is still not serving expected results. Here is an example:

invocation (basically delicious copy pasta of your suggestion)

python run_classifier.py 
--load pretrained/transformer_sst.clf 
--data test_comments.csv 
--model "transformer" 
--write-results wtf4.csv 
--text-key "text" 
--tokenizer-type SentencePieceTokenizer 
--vocab-size 32000 
--tokenizer-path pretrained/ama_32k_tokenizer/ama_32k_tokenizer.model 
--decoder-layers 12 
--decoder-embed-dim 768 
--decoder-ffn-embed-dim 3072 
--decoder-learned-pos 
--decoder-attention-heads 8

output with two obvious sentences I just noted down to verify easily:

label,label pred,label prob,text
-1.0,1.0,0.5084109,this is so bad I hate it
-1.0,1.0,0.51755476,the best thing that ever happened. Love!

For the record, the result is different than previously, and also it runs wayy quicker. I think I got lost in the number of argparse arguments and didn't notice that it would spend time training the tokenizer if I hadn't provided one..

Will try the character ord bump

[MichaMucha]

Character bump worked: ord +1, chr -1 and it goes on reviewing things :)

b"\n If you are in the market for a great laptop computer, this is the one. I have been playing Cive 4 Theater and Netflix since 1990. It had the best games ever, and also feature take -not next-gen popular stuff from the 70's-'99/''05. I really liked setting up a backup in order to actually learn the latest features.But the Special offer is handy and allows me to record important movies in the event it's playing. I've been used to Virtual DVD so, several applications come with it, both newer and less supported. This has the option to stop the audio then' zooms inother sound wave and incorporates his live audio chase.If you never watched this negative review, please re-evaluate the film's title, these older films with interesting features. I'm a huge fan of Kickstarter. I bought them as a gift to the producers and director of the pilot. They are just as picky of as me ....just for the record. " 
 b'\n Overall, to the title of this review, the Gradius version of this game may not be the

[dadelani]

@raulpuric, thanks for your response. @MichaMucha, which part of the code did you modify to prevent the model from generating random characters? I also had this issue but I could not fix it, I'm currently using the older version of the code published over one year ago https://github.com/Athenagoras/sentiment-discovery. It was trained on mlstm and the generate function (visualize.py) works well with a few compatibility issues of pytorch which can be easily fixed. Also, did you generate the text using mlstm or transformer pre-trained model?

[MichaMucha]

Hi @dadelani , I went to generate.py and added the -1 in line 168 like so:
chrs.append( chr(input.data[0] - 1) )

then there are three uses of ord() so modifying each with a plus one: int(ord(c)+1)

Hope this helps, I saw you're using chr in your script but didn't have time to read how do you generate characters from neural net outputs. Raul said the new pretrained model has tokens shifted by one, so if you map your reverse-tokenization correctly you should have your result. Let me know how you get on.
All the best

[raulpuric]

Sorry the code has gotten so out of sync, we tried to incorporate our latest work with the old codebase.
@MichaMucha do you have a colab notebook I can run against to see what's happening for you.
Additionally, do you see bad eval problems with the semeval classifier?

[dadelani]

Thanks a lot @MichaMucha , the generate.py code produces good sentences after the character modification for the mlstm pretrained model but the transformer is still having isses. Did you try generating texts using the transformer pretrained model? Does the generate.py code support transformer.pt? @raulpuric