Sam2StreamEngine attempts to apply Segment Anything Model 2 (SAM 2) on live videos.
Given that SAM 2 allows users to use a click, box, or mask as the input to select an object on any frame of video, we use single_click for a positive prompt, telling the model to generate a segmentation mask for the object containing that point. double_click marks areas to exclude from segmentation. draw_box via a drag mouse event specifies a rectangular region containing the object to segment, resulting in a precise segmentation mask for the object inside.
-
Make sure you have Rust, uv, and sam2 installed. It currently uses
sam2.1_hiera_small.ptmodel checkpoint andsam2_hiera_s.yamlfor model configurations. -
Run the following commands
cd video_streamer
cargo run
cd src
python main.py- You will see a GUI pop up; simply click on any object on the video frame, it will create a green mask predicted by SAM 2.
Currently, video freezes after the first 2 clicks despite successfully generating the green mask.