tc.image_classifier.create should have a way to use extracted features

[NeerajKomuravalli]

The model building with turi create takes most of its time building features and if someone wants to retrain the model with the same data or add just some amount of data it goes through the process of extracting features again and this is a waste of time. Turi Create should have a method of extracting the features and saving the same in sFrame data format and using the same to create the model with either the extracted features or the image data itself.
I could not really find a method which addresses this anywhere in the code or documentation and I would really think this a great feature to have. Please let me know if this is already there and I am not able to find it and if it's not there I would not mind working on it and send a merge request.

[TobyRoseman]

This is a great idea. It's something we have talked about doing before, see #1916.

The main reason we haven't done it already is that we're not sure where to put this new function. The same deep features are used by both the image classifier and the image similarity toolkit.

So do we add the get_deep_features method under tc.image_classifier.? Or add it under tc.image_similarity.? We could add it in both places but I think that could be confusing.

My current thinking is that the best places to add it might be under tc.image_analysis..

If you have any opinions about where to add it, please let me know.

[NeerajKomuravalli]

I think using the same piece of code at two places will be both confusing and redundant so yes that might be a bad idea but why not create a new module where we can just extract features and use the same in both places. In addition we can make the tc.image_classifier.create accept data in both formats, sframe with image_path or sframe with extracted features and will have an option of saving features after model creation.

What are your thoughts?

[TobyRoseman]

I agree we'll need to modify the tc.image_classifier.create (and tc.image_similarity.create) to accept deep features rather than just images.

I think adding a new module (say we call it deep_features) would be a little strange since there would only be one thing inside that of that module. If we add:
tc.deep_features.get_deep_features(image_sarray, model_name).
There isn't going to be anything else under tc.deep_features.

Another option would be to add a new module and make each model have its own function, i.e. remove the model_name parameter. Then we would have three method:
tc.deep_features.Resnet(image_sarray)
tc.deep_features.Squeezenet(image_sarray)
tc.deep_features.VisionFeaturePrint_Scene(image_sarray)
This might be a cleaner solution.

Let me know which of these two options you had in mind. Or if I misunderstood and you had something else in mind.

[NeerajKomuravalli]

I like the later one better and yes It will be relatively cleaner and we can definitely do it.
I don't mind working on it and sending a PR but is there anything in specific I need to keep in mind before working on it?
Or let me know if you have someone else in your mind who will be working on it.

[NeerajKomuravalli]

any comments?

[TobyRoseman]

@NeerajKomuravalli - Sorry for the delay. I wanted to discuss this matter with my manager before responding. We think having one function under tc.image_analysis would be best.

I think it should look like tc.image_analysis.get_deep_features(images, model_name)
Where images is an SArray of images and model_name is a string.

Below is a list of things that will should to be done. There is likely more needed but this should be most of it.

Get Deep Features

• Create tc.image_analysis.get_deep_features and write the docstring
• Parameter checking for tc.image_analysis.get_deep_features. Make sure images is a non-empty SArray of images. Make sure model_name is a string with one of the three allowed values (note “VisionFeaturePrint_Scene” is only allowed on certain macOS version -- see existing code). If any of the parameter aren't right the user should get an informative error message.
• Implement tc.image_analysis.get_deep_features. You should be able to just use the existing code from either the image classifier or image similarity.

Image Classifier

• Modify code to check if the input are deep features. We already do something similar for the sound classifier. Creating a helper function here would probably be good so you can also use it for image similarity too.
• If deep features are not passed in, the code should just use tc.image_analysis.get_deep_features to get the deep features. This is so we don't have duplicated code.
• Update the docstring.
• Write image classifier unit tests for this new functionality.

Image Similarity

• Modify code to check if the input are deep features.
• If deep features are not passed in, the code should use tc.image_analysis.get_deep_features`.
• Update the docstring.
• Write unit tests for this new functionality.

Let me know if you have any questions. Feel free to put up a work in progress pull request at any point, if you would like feedback.

[NeerajKomuravalli]

@TobyRoseman Thanks for the response, that was a very detailed doc. I agree with all your point and will try to send a PR at the earliest and will let you know if I have any doubts.
Thanks!

[NeerajKomuravalli]

@TobyRoseman I was working on all the things we discussed and I realized, to extract features we need to call the _image_feature_extractor._create_feature_extractor.extract_features and it accepts parameters dataset and feature which is basically SFrame and feature column name respectively. So if we want to call this from tc.image_analysis.get_deep_features we need to send SFrame dataset in get_deep_features or we need to change the extract_feature function in _image_feature_extractor.
I personally feel like we should change the input parameters of tc.image_analysis.get_deep_features from SArray to SFrame. This will make sure we don't have to change a lot of the code and yet the same can be achieved by just passing dataset, feature, model_name and batch_size. And even when people want to save features they can directly save the the output as it has a direct association of images and the features (This is a side benefit and it's advantages are subjective and might not be as important).
Let me know what are you thoughts?

[TobyRoseman]

I think dataset should be a SArray not an SFrame. There's no reason for it to be an SFrame since we only need the image column in order to generate the deep features. It's just cleaner/simpler to have it be an SArray.

I don't think it should be much work to for it be an SArray rather than an SFrame. The current code should not need to change too much.
If dataset is now a SArray, I think you can just change:
image_sf = tc.SFrame({"image": dataset[feature]})
to:
image_sf = tc.SFrame({"image": dataset})

You're right we will need a batch_size parameter in tc.image_analysis.get_deep_features. We should also have an optional verbose flag. So let's add verbose=True and batch_size=64 to the definition of tc.image_analysis.get_deep_features.

[NeerajKomuravalli]

yes, you are right. I can just create a SFrame and get this done and there is no need for accepting SFrame as input.