Gesture Recognition through OpenMV Cam M7

OpenMV Products
#1

Hello,

Aim:
We are planning to implement gesture recognition on MCU (STM32F7) but as a proof of concept evaluation we want to do it using openMV Cam M7 board.

Gestures:

  1. User approaches to camera with Closed fist ----> and opens the fist to form a palm.
  2. User approaches to camera with open palm ----> and closes palm to form a fist.

Our initial path was by using openCV and PC (no resource constraint approach), which provided as with favorable results. In this approach we used contours and convexhull to determine the gesture performed by the user.
We would like to replicate similar performance (not necessarily using same algorithms) using openMV. (open to using different approach)

So far,
After reading documentation provided by openMV,

We standard that blob is similar to contours. So, we thought of using blob as it would help us to determine centroid of palm and then using this we can either draw a circle to identify palm surface and determine that it is open palm. Or we can draw lines using image.line(), which draws lines in all directions. Using the pixel intensity difference in lines we can identify finger tips (Not sure how to do it)

Also, please clarify if convex hull functionality is present in openMV. We believe it is not available.

It would be helpful if you can suggest an approach to move forward, or if there is a better approach we are open to trying it. Please, let us know if you need more information.

Regards .

#2

Hi, we don’t store blob edge pixels so it’s not possible to make a convex hull calculation on the OpenMV Cam. That said, we plan to live up to our name soon and focus everything on doing work with CNNs. You’ll be able to use find blobs to find the hand area and then run a CNN on the area to determine if the hand is closed or not.

That said, CNN support will be released by the end of the month. It technically is working right now but not yet that much documentated. If you are willing to do a little bit of work without much of a guide I can give you some sample code to run a CNN on a hand.

In the meantime, it may be more useful for you to just just CNNs on hands.

Let me know if this sounds interesting and I’ll let you know what type of hyper parameters you should use.

#3

Thank you @kwagyeman.

Sure. Your suggestion sounds interesting.

But we are interested in solving this problem with a geometric approach rather than using convolutional neural network.

#4

Hi, we offer the features available using find_blobs(). To support a lot of the OpenCV methods requires you to store a data structure with all points on the boundary of a blob which we chose not to given microcontroller memory limits. In particular, on the M4 things were quite limited. They are not anymore now but I don’t wish to add new features to find_blobs().

Find blobs will give you the width, height, and number of pixels within a blog along with the rotation of that blob. You may be able to determine if a person is holding a fist or hand up to the camera given these things.

If this is not sufficient you can always edit the c code and make the camera do whatever you want. It wouldn’t be that hard to add a list of saved points to a blob.

…

That said, we plan to focus our future feature dev on ML stuff. Previous computer vision approaches made possible with OpenCV… while computationally cheap… are not going to be considered valuable moving into the future.