Update BodyPix to version 2.0

[gohai]

Note: all this work was done by @VivianChenyc5519 and @ch3926. 👏 @gohai is shamelessly opening this PR after merely doing just a little bit of cleanup.

TODOs (hope those can be tackled after merging):

• does the new result enable users to do everything they were able to in the old library? (maybe @MOQN knows?)
• rework to match the the latest API pattern? (detect(), detectStart(), detectStop(), etc) à la @ziyuan-linn
• rework examples to use preload?
• add examples for imported image and video later
• add Mediapipe support

[VivianChenyc5519]

Assigning variable in the callback and only painting in draw would not work for body-segmentation in this case. I received an error trying to do so -- segmentation is not defined in draw(). And the main reason, I suppose, is that by the time I accessed segmentation, the assignment is not complete yet. And according to this stack overflow post, the only solution is to put everything under the callback function.

So I will stick with your work-around for now!

[gohai]

Assigning variable in the callback and only painting in draw would not work for body-segmentation in this case. I received an error trying to do so -- segmentation is not defined in draw(). And the main reason, I suppose, is that by the time I accessed segmentation, the assignment is not complete yet.

@VivianChenyc5519 You're right, because draw() will be called before the first gotResults(), we'd need to take some extra care not to access properties of the object before its ready. The most common way to do that is to test with an if (undefined will be "falsy", which makes this work):

if (segmentation) {
  image(segmentation.backgroundMask, 0, 0, width, height);
}

[shiffman]

This is wonderful! @gohai I think you can merge when you feel it is ready. I would love @MOQN's thoughts also as I don't have much experience teaching with this model.

I agree we should emulate the detectStart() API model, but we could also consider segmentStart()? I'm not sure.

There is also the question as to whether this should be named ml5.bodypix() or if we should use a generic term like ml5.imageSegmentation("bodypix") with the name of the model passed in. The latter matches the ml5.bodyPose() and ml5.handPose() convention. However, BodyPix is perhaps developed in a unique/custom way that trying to make a class work with other image segmentation models might be too much trouble to maintain.

[shiffman]

Just noting we might want to include a "hello world" version of this that just uses defaults. The options can sometimes overwhelm / confuse a beginner.

[shiffman]

Is this something that the BodyPix model does natively or something you all added? I wonder if we should include an option like this for other models as well?

[VivianChenyc5519]

Hi Dan! If your question meant "why these four paramters", that's the latter three (multiSegmentation, segmentBodyParts, flipHorizontal) actually depends case by case. For example, if the user wants to detect multiple people and output different segmentation, then multiSegmentation needs to be turned on. As for outputStride, that's because I noticed this parameter could affect model performance, in terms of accuracy, so I thought this might be a good way to familiarize our users to machine learning.
I have set a default values for all these in index.js. We can create a default input for different cases like webcam or image for parameters like flipHorizontal and segmentBodyParts, but I think it might be a good idea to leave multiSegmentation (one segmentation for multiple people or multiple segmentation) to the user?
Lemme know if there's still something I didn't explain clearly or understand wrongly! Be happy to elaborate/correct :)

[shiffman]

For students to work from, in my experience it's best to have the drawing in draw() and use a if as @gohai suggests. Things go haywire quickly when students try to build on the code when there is drawing in a separate callback.

[gohai]

@VivianChenyc5519 @ch3926 Dear Vivian & Connie - pinging to make sure you see Dan's feedback (above). If you don't find the time to make any changes to this before our DURF wrap-up, then we can also do this at some later point.

[VivianChenyc5519]

Yes! I’m working on this right now. I will make  commit once everything is running properly.Best,VivianOn Sep 2, 2023, at 07:26, Gottfried Haider ***@***.***> wrote: @VivianChenyc5519 @ch3926 Dear Vivian & Connie - pinging to make sure you see Dan's feedback (above). If you don't find the time to make any changes to this before our DURF wrap-up, then we can also do this at some later point. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[ch3926]

Another reason to keep it as ml5.imageSegmentation("bodypix") is to keep parallel with how tensorflow moved away from 'body-pix' towards "body-segmentation". [image: Screenshot 2023-09-02 at 2.15.48 PM.png] On Sat, Sep 2, 2023 at 9:17 AM VivianChenyc5519 ***@***.***> wrote:

…

Yes! I’m working on this right now. I will make commit once everything is running properly.Best,VivianOn Sep 2, 2023, at 07:26, Gottfried Haider ***@***.***> wrote: @VivianChenyc5519 @ch3926 Dear Vivian & Connie - pinging to make sure you see Dan's feedback (above). If you don't find the time to make any changes to this before our DURF wrap-up, then we can also do this at some later point. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub <#40 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASFWNSWEYYOV57UAULAU473XYMWWNANCNFSM6AAAAAA34VFXTQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[shiffman]

Another reason to keep it as ml5.imageSegmentation("bodypix") is to keep parallel with how tensorflow moved away from 'body-pix' towards "body-segmentation".

Oh! Thanks for pointing this out @ch3926! Body segmentation is a special case so maybe it's worth having a specific bodySegmentation() function instead of imageSegmentation("body")? I'm not sure! Here are some ideas:

// If we had different body segmentation models then an argument could be passed in
let bodySegmenter = ml5.bodySegmentation();
// If the function were just "image" you would have to specify body (and maybe a specific model?)
let bodySegmenter = ml5.imageSegmentation("body");

bodySegmentation() probably makes more sense b/c general image segmentation wouldn't know about body parts and other things specific to this kind of model that we may want to include (multiple people, "selfie" background removal, etc.)

[VivianChenyc5519]

This is wonderful! @gohai I think you can merge when you feel it is ready. I would love @MOQN's thoughts also as I don't have much experience teaching with this model.

I agree we should emulate the detectStart() API model, but we could also consider segmentStart()? I'm not sure.

There is also the question as to whether this should be named ml5.bodypix() or if we should use a generic term like ml5.imageSegmentation("bodypix") with the name of the model passed in. The latter matches the ml5.bodyPose() and ml5.handPose() convention. However, BodyPix is perhaps developed in a unique/custom way that trying to make a class work with other image segmentation models might be too much trouble to maintain.

I think segmentStart would fit better in the context of body-segmentation. However, detectStart would make it more consistent. I am also a bit not sure about this. @ch3926 Just wondering what's Connie's view on this!

I would vote for
// If we had different body segmentation models then an argument could be passed in let bodySegmenter = ml5.bodySegmentation();

I will first integrate the detectStart() (or segmentStart() maybe) API model to body-segmentation and then work on the rest of the issues mentioned here.