@lillyguisnet: In 2023 I used a combination of SAM1 + a custom vit classifier to ID and automate the segmentation of my microscopy ima…

In 2023 I used a combination of SAM1 + a custom vit classifier to ID and automate the segmentation of my microscopy images. Today, SAM3.1 can do it zero-shot. (-ish) The difficulty with extracting a mask from images such as those is that they are full of other crap you don't care about. How can you direct the model to extract only your item of interest and not the other object-like things? You could just click each image one by one and get beautiful masks, but that's time consuming, trying to do better. After trying a good bunch of approaches, something cool emerged. You can exploit the small similarity that exists between microscopy images to trick the model into propagating the object mask, as if it was a video. The final best approach segmented 100% of the images. > Group the images by SSIM similarity > give the model a single point prompt > propagate on the group "video" > use 2-3 overlap images from the first group as prompts to propagate in your second group > continue jumping through groups > establish failure flags that are known for your context (empty mask, size thresholds, position thresholds, etc) > failed images are regrouped until pass That's it. A simple algorithm automates the task! Text prompting per-image (with prompt optimization) also achieved 100% segmentation. BUT many images included false positives (multi-masks) which were not automatically filtered out. The single-ID video approach forces the model to return a single mask and our prompt targets the object of interest. I like how today's models with a little program optimization magic can solve your problems :) The segmentation quality is not quite there tho, looking into that