Accuracy print-outs of supervised_mnist.py -example seem incorrect

[Danielle-Schuman]

In the supervised_mnist.py -example, the "All activity accuracy" and the "Proportion weighting accuracy" are calculated every update_interval iterations. Also, the "assignments" of labels to the neurons in the excitatory layer are calculated, which are used for plotting.
This would be very handy to keep track of how the training is progressing, were it not for the fact that this seems to not be working accurately.
E.g. if you run the example with default arguments, at the end of the training you still get print-outs like
"All activity accuracy: 2.40 (last), 5.26 (average), 93.20 (best)
Proportion weighting accuracy: 82.00 (last), 18.16 (average), 93.20 (best)",
while I personally would have assumed it to be a lot better after 5000 training-iterations.

I think the cause of the issue is that the networks predictions are just compared to the label of the one datum for the next iteration instead of to all the labels of all the data of the last update_interval, for which they are the predictions. This also becomes clear if you look at the description of the method "assign_labels", which calculates the "assignments", "proportions" and "rates": Here, there is explicitly talk of labels as a vector of size n_samples with data labels corresponding to the spiking activity of the network.

P.S. Thank you so much for the great package, by the way :)

[Danielle-Schuman]

I managed to fix the problem by introducing an additional tensor "labels" that is used to save the labels of the last update_interval iterations, and replacing "label" with this in the appropriate places.

I now get outputs like
"All activity accuracy: 100.00 (last), 88.71 (average), 100.00 (best)
Proportion weighting accuracy: 100.00 (last), 88.80 (average), 100.00 (best)"
as the learning process progresses further (this example is at iteration 2250/5000 with the default parameters) and also, the plots which use the assignments etc. show sensible data now.

I attached my solution here, once as .txt (unfortunately, .py files are not attachable here), and once as .pdf where I marked the changes I made in pink, as you might want to add these or similar changes to supervised_mnist.py

supervised_mnist_labels_fixed.txt
supervised_mnist_labels_fixed.pdf

[Hananel-Hazan]

You are right, good catch.
I added your suggestion to the repo.
For next time, you can also commit a pull request

[wusaifei]

@Danielle-Schuman Hello, I'd like to ask supervised_ mnist.py Is there any problem in the test part of? I tested the results at 10%, and the predicted tags are all 9. I'm looking forward to your reply

[Danielle-Schuman]

Hello @wusaifei, Regarding your question about the testing phase: I don‘t know, to be honest, because I‘ve been working with an older version of the code that doesn’t have an explicit testing phase in it. (Also, I‘m not sure what you mean by „at 10%”.) But since Mr. Hazan just pushed some new code regarding testing, maybe that fixes your problem?

[wusaifei]

@Danielle-Schuman Thank you very much for your reply. The accuracy rate of MNIST test set is about 10%. With the improvement of the new version, the accuracy of MNIST's test set has been improved, but it has been maintained at 18%. The accuracy of training set is as high as 98%.

[wusaifei]

@Danielle-Schuman In the new version just released, the accuracy of MNIST test set has not improved significantly, and the accuracy of test set remains at 18%.