Bad results - Investigate reason

[PavlosMelissinos]

Metric	IoU	area	maxDets	Result
Average Precision	0.50:0.95	all	100	0.001
Average Precision	0.50	all	100	0.004
Average Precision	0.75	all	100	0.000
Average Precision	0.50:0.95	small	100	0.000
Average Precision	0.50:0.95	medium	100	0.000
Average Precision	0.50:0.95	large	100	0.004
Average Recall	0.50:0.95	all	1	0.005
Average Recall	0.50:0.95	all	10	0.005
Average Recall	0.50:0.95	all	100	0.005
Average Recall	0.50:0.95	small	100	0.000
Average Recall	0.50:0.95	medium	100	0.001
Average Recall	0.50:0.95	large	100	0.019

This is using the official mscoco script.

Setup as: full image as input, each pixel gets classified using a one hot vector with a size of 81, 0 to 80 inclusive, that correspond to the actual category ids in MS-COCO. More specifically, index 0 is background, ..., index 12 corresponds to class id 13 (stop sign), ..., and index 80 is in fact class 90 (toothbrush). Output is the full image, not a crop. Then a script is used to separate the pixels of each detected object. No classes were used in the evalCOCO.py script (useCats = False).

These are really bad scores, and at the moment I have no idea why it's like that. I'll push the changes soon.

Which script do you use for evaluation @athundt ? If you have a working version maybe I should just replace mine with it. Does this work for mscoco?

[ahundt]

Sadly so far my mscoco results in Keras-FCN were no good. The loss function went negative with the full image segmentation and one_hot encoding, then I ran out of time to investigate the details.

[PavlosMelissinos]

That's too bad... FCNSS doesn't report performance on MS-COCO iirc but in the PSPNet paper, some IoU results are mentioned on page 8 (Table 6) and they seem to be pretty decent.

Hi guys,
So, have you been able to reproduce the ENet results from the paper?
Best,
D.

[PavlosMelissinos]

Not really, sorry. It does converge but not well enough.

In the paper, the encoder is pretrained on ImageNet and the full pipeline is then fine-tuned on Cityscapes, CamVid and Sun RGB-D. However, I haven't set them up yet so I've only trained the network on MS-COCO (which often gives awful results). I'd like to finish the project at some point but I've had to move on to other stuff so at the moment I don't have the resources to do it properly, unfortunately. :(

No, worries I'll pick it up from here and see what's the problem.
Not sure will be allowed to share code though.

[ahundt]

There has been a bugfix in densenet that solved some problems so it might work better now!

https://github.com/farizrahman4u/keras-contrib/blob/master/keras_contrib/applications/densenet.py

[jmtatsch]

@ahundt can you elaborate further how the densenet fix may be applicable to enet-keras? it seems as if the main gradient flow and the pooling indices are connected properly or am I missing something?

[ahundt]

@jmtatsch Sorry my post is totally irrelevant I must have mixed up tabs on my browser or something.

Hi guys,
So it seems that I have probably managed to successfully retrain ENet on our own dataset by loading pretrained weights from torch and using adadelta (adagrad didn't work as well for me).
You can load the weights from the torch model with torchfile.
One more minor thing which shouldn't make much difference is that I added batch norm after the initial layer, following the paper, which I think was not there in the code.
Anyway, I need to investigate a bit more the per-class accuracy and will get back to you.
Best,
D.

[jmtatsch]

@dkorkino the PReLu also seems to be missing as compared to https://github.com/e-lab/ENet-training/blob/master/train/models/encoder.lua#L86 Could you maybe publish the converted weights?

[PavlosMelissinos]

You're both right, @ghost and @jmtatsch. I also noticed a division bug in MaxPoolingWithArgmax2D that resulted in unwanted behavior on python 3 and another in the data generator.
All three should be fixed now but let me know about any problems you might encounter.

Thanks a lot for the feedback 👍.

Sorry for taking this long to tackle the issue but I'd been on vacation until yesterday.

[ColdCodeCool]

@dkorkino @jmtatsch I am also looking forward to the release of the converted weights.

[PavlosMelissinos]

Does anyone have any idea why it takes so long to train?

I'm getting something like 25K seconds per epoch on MS-COCO (~80K samples) on a K40 for input dimensions of 256x256.

That amounts to ~0.3s per sample, so let's say about 10 fps for just the forward pass. That's much slower than the reported performance (135.4 fps for 640x360 on a Titan X)

I used to think it might be due to preprocessing but it actually only takes a fraction of that time. Any thoughts?