OpenMV H7 for fish identification

Hello all,

I’ve recently ordered an H7 and am quite excited to get going with it. I’m a PhD student and all about developing low-cost instrumentation for scientists (citizen or otherwise). I’ve just finished a project developing a small PCB that lets cheap action cameras be deployed on the seafloor for ~1 month at a time to monitor growth of the corals I study. I’m really curious about using the H7 for two purposes, but am wondering how reliable the software side will be.

Task 1: A static camera which is counting and classifying fish as they swim in and out of the field of view. I’m not concerned about double-counting, but I would like to classify fish to at least the family level and count their abundance, for example the abundances of angelfish, pufferfish, and a parrotfish (just naming three visually distinct groups). I can provide the training data, but these are going to be deployed in dark environments (by comparison to daylight on the surface) of about 1000-5000 lux/ft irradiance. I understand this is a specialized task, but which provided software examples would be a good start? Or is this more of a deep learning task?

Task 2: Track increased object growth over time. I’m actually a coral disease ecologist, and rather than my cameras collecting imagery and having to process it back home, I’d love it if an H7 could process it in the field. I’m pretty sure I can get it to deploy for at least a month since it’s power consumption is about 2x that of the cameras I normally use. What I’d like would be for the H7 to take a photo of a blob (let’s say a brown coral on a mixed background), and determine what percent more it takes up of the frame vs a prior reference image. Color recognition probably isn’t an option here unless the H7 is capable of discerning different shades of brown or green from one another. Is blob recognition an option and can it estimate the size (percentage of frame, not true size) of a blob?

According to Sparkfun it’ll be a month before my H7 arrives, which is disappointing, but I figured I’d ask if these tasks are even possible with the H7 in a challenging underwater environment. Any thoughts, brainstorming, or help is appreciated!

Best,
Austin

Hi, we are about to release TensorFlow for the system in firmware v3.5.0. The firmware is complete and on GitHub. I just have to finish the documentation and then get it into OpenMV IDE to be sent to all customers.

With TensorFlow on board you just train a TensorFlow lite model, quanitize it, and then it will run on the camera. The model can’t be too big… But, otherwise you can classify images easily.

We also will just image segmentation so that you can determine what parts of the image are what and then you can use our blob finder to track the classified blobs.

I don’t think we will support object tracking for a while until Google releases an example on how to do that using less RAM. Large nets like SSD don’t work at all on our system. So, just classification and segmentation. Even segmentation is a stretch a little bit too.

The out of the box example on firmware 3.5.0 is a person detector. But, you can basically run any TensorFlow classifier model. So, literally, whatever training data you have can be used to map to a list of float outputs representing class scores.

So, just collect training data and train robust models. Note that the camera color may be different from the training data so you need to make your model robust to that…

Thanks for the quick reply Kwabena, I’ll look forward to applying TensorFlow via 3.5.0 then. Really excited at the potential applications of H7 in the biological sciences. Currently, almost all classification is done by hand and it typically limits the scope of research (classifying images manually is expensive and time consuming). Hoping the H7 can demonstrate a way forward!