hope support mobilenetv3 small

[jiangjianping]

hello Floe,

Could you tell me how to add the support of mobilenetv3 small?

[floe]

Not quite sure what you mean, deeplab v3 is already based on mobilenet v3? Or do you have a link for the specific network you want to support?

[jiangjianping]

FYI
https://ai.googleblog.com/2020/10/background-features-in-google-meet.html

[floe]

Ah, interesting. First thing you would need is the actual model file in .tflite format, apparently it's not available in https://github.com/google/mediapipe/tree/master/mediapipe/models ?

[jiangjianping]

Mobilnetv3 Large?
https://tfhub.dev/sayakpaul/lite-model/deeplabv3-mobilenetv3-cityscapes/1/default/2

[floe]

Deeplab v3 is exactly what my project here already uses: https://github.com/floe/deepbacksub/blob/master/deeplabv3_257_mv_gpu.tflite

The only difference I can see is that the one you just posted is trained on Cityscapes, which won't help a lot with person segmentation.

The one from Google Meet would be much more interesting, because it is specifically optimized for person segmentation, but you would need to find the specific .tflite file for that Google Meet network.

[jiangjianping]

Floe,
Thanks, Hope lucky to get that tfilte. Do you have idea about joint bilateral filter?

[floe]

Right now, it seems you can download the models from https://meet.google.com/_/rtcvidproc/release_1wttl/345264209/segm_full_v679.tflite and https://meet.google.com/_/rtcvidproc/release_1wttl/345264209/segm_lite_v681.tflite - and according to the model card they should be licensed with Apache 2.0, so you should in theory be allowed to re-use them. I'll give it a try and see if these can be integrated.

[jiangjianping]

Floe,
Great! Thanks a lot.

[jiangjianping]

Hello Floe,
I tried the new model, but the AllocateTensors() failed. I guess that the xnnpack need to be included.

[jiangjianping]

Got something may be useful

tensorflow/tfjs#4177

[floe]

I've just copied the relevant custom op from MediaPipe for now. See latest commit (306da1b) for a implementation that's somewhat working (still some sort of synchronization error between capture thread and network evaluation happening here, though).

[jiangjianping]

Great! I will try.

[floe]

I'm closing this for now, thanks for pointing me to the Google Meet segmentation model. Please open a new issue if something is still wonky.