Artifacts from camera after running CNN and frame differencing

Discussion related to "under the hood" OpenMV topics.
ngattuso
Posts: 23
Joined: Thu Dec 27, 2018 8:29 pm

Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Mon Jan 28, 2019 11:17 pm

Hello,

I noticed an issue where there's artifacts in my frame which is then causing my frame differencing to think that there's a change.

A short description of my program:
I use frame differencing to detect when an object comes in. I count 30 frames with an object before sending the latest frame to my CNN. After checking the frame to the network, I update the background by replacing it with a new sensor.snapshot().

What I noticed is that the artifacts come in after running the CNN at least once. Please see attached image for an idea of what I'm seeing. I have a video but it is too big to attached.

If I run my code without the CNN then the background updates and I don't see the artifacts at the bottom of my frame differencing.

This problem is stopping me from detecting new objects after detecting my first object because the frame differencing is constantly recognizing a difference now.

I was wondering if you have any suggestions on what I can do to either reset the camera to the state it was before the CNN or what I can do to not get the artifacts.

Thanks,
Nicholas
Attachments
noise in frame differencing.jpeg
User avatar
kwagyeman
Posts: 2713
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Tue Jan 29, 2019 2:17 am

Hmm, was resolution are you running at? I know for a fact the frame buffer doesn't respect the frame buffer stack at all. So, it will overwrite it and causes a crash (this is something I will fix soon). The frame buffer stack when pushed does respect what is in the frame buffer and errors out if there was about to be an collision.

Can you post some simple code to see the error and we'll get this fixed quickly.
Nyamekye,
ngattuso
Posts: 23
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Tue Jan 29, 2019 3:15 pm

Thank you for your reply!

I'm using QQVGA right now. But, I also tried it with QVGA and a window of 128x128. The issue arose with both.

I attached a simplify version of my code (basically removed unnecessary sections like servo control and LEDs). In addition, the network file is also included.

Do you know of any alternate solutions / temporary fix like a reset the camera or frame buffer? I have a prototype demo for my project next Tuesday.

Thank you!
Nicholas
Attachments
test issue.zip
(20.55 KiB) Downloaded 9 times
User avatar
kwagyeman
Posts: 2713
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Wed Jan 30, 2019 12:38 pm

Okay, will do. I've asked Ibrahim to look at it but I will be able to take a look over the weekend too.
Nyamekye,
User avatar
iabdalkader
Posts: 671
Joined: Sun May 24, 2015 3:53 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby iabdalkader » Wed Jan 30, 2019 2:05 pm

Hi, I tested your code and saw the artifacts, it's an issue with extra_fb that we will fix very soon. For now, you should save the background image to disk, and pass a path to img.difference. Please let me know if this fixes the issue.
ngattuso
Posts: 23
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Wed Jan 30, 2019 8:17 pm

Hello,

Thank you both for your time to help me with the issue!

In order to save to disk, I'll need to use a SD card right? I remember reading that we shouldn't write to the openMV flash.

Once I get a microSD, I'll try your suggestion and let you know if that works. Additionally, may you please inform me once the issue with the extra_fb will be fix as well?

Thank you,
Nicholas
User avatar
kwagyeman
Posts: 2713
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Thu Jan 31, 2019 2:32 am

The issue is that some.of the methods in our code pop the extra allocated fb accidentally causing memory corruption. We have to fix these in a lot of places.

Anyway, just save the background images to disk as bmp files and then pass a path to them from difference and the OpenMV Cam will use the disk as the background image buffer instead until we get this fixed.

...

Yes, you will need a micro SD card.
Nyamekye,
User avatar
iabdalkader
Posts: 671
Joined: Sun May 24, 2015 3:53 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby iabdalkader » Thu Jan 31, 2019 6:08 pm

Hi, this issue has been fixed, please try the attached firmware if you don't want to wait for the next release.
Attachments
firmware.zip
(906.9 KiB) Downloaded 5 times
ngattuso
Posts: 23
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Thu Jan 31, 2019 10:20 pm

Hello,

Thank you for the fix! It is working now!!!

I also like the alternative of saving the background image of the microSD card while this wasn't working.

Thanks again for the quick work and great product!
Nicholas
User avatar
kwagyeman
Posts: 2713
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Fri Feb 01, 2019 1:37 am

Hi, you're the first customer to actually use this feature that I know of. For the H7 it's going to be much more useful. Anyway, if you have some time and a blog, a write up on like Hacksters would be appreciated. Likez, for example, sharing your idea, but not he implementation.
Nyamekye,
ngattuso
Posts: 23
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Thu Feb 07, 2019 8:12 pm

Sorry, which feature exactly? The frame differencing with extra frame buffer, the CNN or, both?

I definitely would love to write a blog. I can probably do one at the end of February when my project reaches an end. Currently, I've been sharing short videos on my LinkedIn showing results with the openMV.

Thanks!
User avatar
kwagyeman
Posts: 2713
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Fri Feb 08, 2019 12:30 am

Both at the same time. Generally, folks don't actually want to train a CNN and collect the data for it.
Nyamekye,
ngattuso
Posts: 23
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Tue Feb 12, 2019 8:29 pm

Yeah, I can see why haha! The data collection is very time consuming and challenging. Even then too, you have to be sure to properly capture the data or else you won't have good performance.

I'm actually modifying my program now to use the SD card to save images. This way I'll be able to gather my data with the proper environment.

In any case, I have it in my agenda to make a blog post about my project with the OpenMV. I'll let you know when it's written!
User avatar
kwagyeman
Posts: 2713
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Wed Feb 13, 2019 12:49 am

Awesome!
Nyamekye,

Return to “Technical Discussion”

Who is online

Users browsing this forum: No registered users and 3 guests