My team and I are working on a software/hardware solution for cursor control. We want to take the place the user is looking at the screen and move the mouse to that position. We are using the Nicla Vision to do so. As of now we tried to use the face_eye_detection_uart.py and iris_detection_uart.py we found these scripts in a previous message here on this forum and we realized that the iris detection only works accurately under strong light directed in the face of the user else the detection almost doesn’t work. To make the iris detection work accurately we had to buy LED lights and connect the LEDs on a breadboard along with the Nicla Vision to get good lighting however again when we were testing we had to sit really close up to the Nicla Vision for the detection to work or in other words for the haarcascade to even notice there was an iris in the frame. Another approach we took was to collect datasets from the team and we trained our model using edge impulse and we got 100% accuracy on edge impulse but once we flashed the firmware on the Nicla Vision the accuracy was almost zero. We moved to another approach were we are flashing the Nicla Vision with a model we found online and the model size is 261kb “https://huggingface.co/qualcomm/MediaPipe-Face-Detection” and here is the model. After flashing the Nicla Vison we keep consistently getting an error “Value error: Failed to allocate tensors”. So we are wondering if there are any other approaches that we can try or if there are ways to fix the error mentioned above.
Hi, we have a new camera which can run AI models under development that we Kickstarted earlier this year. The OpenMV AE3. It can do all these things. We are about to approve mass production, we are just waiting on verifying our final prototypes antenna tuning, and then we will start building 1500 units. The new system can run that exact network at hundreds of FPS.
Another approach we took was to collect datasets from the team and we trained our model using edge impulse and we got 100% accuracy on edge impulse but once we flashed the firmware on the Nicla Vision the accuracy was almost zero.
This probably means the dataset was overfit by the model, you typically won’t get 100%. If you add more data sources to your model you should be able to fix this.
As for the HaarCascades, yes, they don’t work except in very strong lighting situations.
The new camera sounds amazing, and we’d definitely love to explore using it in our future projects once it becomes available. However, due to current time constraints, our immediate focus will remain on the Nicla Vision. We’ll begin by addressing the model’s overfitting issue to see if that helps close the performance gap.
In the meantime, if you have any suggestions for alternative approaches we could explore to help us reach our goal, we’d greatly appreciate your input. We’re open to trying other methods that could support the successful completion of this project.
Generally shining an IR light on a persons eyes and then looking for the brightest pixels in the field of view works for this type of thing.
doesn’t the Nicla Visions camera have an IR filter that needs to be removed first?
Oh, yes… and the lens is not removable. Mmm… I’d refine the Edge Impulse FOMO Algorithm then. That’s the most likely way.