Artifacts from camera after running CNN and frame differencing

[ngattuso]

Hello,

I noticed an issue where there's artifacts in my frame which is then causing my frame differencing to think that there's a change.

A short description of my program:
I use frame differencing to detect when an object comes in. I count 30 frames with an object before sending the latest frame to my CNN. After checking the frame to the network, I update the background by replacing it with a new sensor.snapshot().

What I noticed is that the artifacts come in after running the CNN at least once. Please see attached image for an idea of what I'm seeing. I have a video but it is too big to attached.

If I run my code without the CNN then the background updates and I don't see the artifacts at the bottom of my frame differencing.

This problem is stopping me from detecting new objects after detecting my first object because the frame differencing is constantly recognizing a difference now.

I was wondering if you have any suggestions on what I can do to either reset the camera to the state it was before the CNN or what I can do to not get the artifacts.

Thanks,
Nicholas

[kwagyeman]

Hmm, was resolution are you running at? I know for a fact the frame buffer doesn't respect the frame buffer stack at all. So, it will overwrite it and causes a crash (this is something I will fix soon). The frame buffer stack when pushed does respect what is in the frame buffer and errors out if there was about to be an collision.

Can you post some simple code to see the error and we'll get this fixed quickly.

[ngattuso]

Thank you for your reply!

I'm using QQVGA right now. But, I also tried it with QVGA and a window of 128x128. The issue arose with both.

I attached a simplify version of my code (basically removed unnecessary sections like servo control and LEDs). In addition, the network file is also included.

Do you know of any alternate solutions / temporary fix like a reset the camera or frame buffer? I have a prototype demo for my project next Tuesday.

Thank you!
Nicholas

[kwagyeman]

Okay, will do. I've asked Ibrahim to look at it but I will be able to take a look over the weekend too.

[iabdalkader]

Hi, I tested your code and saw the artifacts, it's an issue with extra_fb that we will fix very soon. For now, you should save the background image to disk, and pass a path to img.difference. Please let me know if this fixes the issue.

[ngattuso]

Hello,

Thank you both for your time to help me with the issue!

In order to save to disk, I'll need to use a SD card right? I remember reading that we shouldn't write to the openMV flash.

Once I get a microSD, I'll try your suggestion and let you know if that works. Additionally, may you please inform me once the issue with the extra_fb will be fix as well?

Thank you,
Nicholas

[kwagyeman]

The issue is that some.of the methods in our code pop the extra allocated fb accidentally causing memory corruption. We have to fix these in a lot of places.

Anyway, just save the background images to disk as bmp files and then pass a path to them from difference and the OpenMV Cam will use the disk as the background image buffer instead until we get this fixed.

...

Yes, you will need a micro SD card.

[iabdalkader]

Hi, this issue has been fixed, please try the attached firmware if you don't want to wait for the next release.

[ngattuso]

Hello,

Thank you for the fix! It is working now!!!

I also like the alternative of saving the background image of the microSD card while this wasn't working.

Thanks again for the quick work and great product!
Nicholas

[kwagyeman]

Hi, you're the first customer to actually use this feature that I know of. For the H7 it's going to be much more useful. Anyway, if you have some time and a blog, a write up on like Hacksters would be appreciated. Likez, for example, sharing your idea, but not he implementation.

[ngattuso]

Sorry, which feature exactly? The frame differencing with extra frame buffer, the CNN or, both?

I definitely would love to write a blog. I can probably do one at the end of February when my project reaches an end. Currently, I've been sharing short videos on my LinkedIn showing results with the openMV.

Thanks!

[kwagyeman]

Both at the same time. Generally, folks don't actually want to train a CNN and collect the data for it.

[ngattuso]

Yeah, I can see why haha! The data collection is very time consuming and challenging. Even then too, you have to be sure to properly capture the data or else you won't have good performance.

I'm actually modifying my program now to use the SD card to save images. This way I'll be able to gather my data with the proper environment.

In any case, I have it in my agenda to make a blog post about my project with the OpenMV. I'll let you know when it's written!

[kwagyeman]

Awesome!