Artifacts from camera after running CNN and frame differencing

Discussion related to "under the hood" OpenMV topics.
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Mon Jan 28, 2019 11:17 pm

Hello,

I noticed an issue where there's artifacts in my frame which is then causing my frame differencing to think that there's a change.

A short description of my program:
I use frame differencing to detect when an object comes in. I count 30 frames with an object before sending the latest frame to my CNN. After checking the frame to the network, I update the background by replacing it with a new sensor.snapshot().

What I noticed is that the artifacts come in after running the CNN at least once. Please see attached image for an idea of what I'm seeing. I have a video but it is too big to attached.

If I run my code without the CNN then the background updates and I don't see the artifacts at the bottom of my frame differencing.

This problem is stopping me from detecting new objects after detecting my first object because the frame differencing is constantly recognizing a difference now.

I was wondering if you have any suggestions on what I can do to either reset the camera to the state it was before the CNN or what I can do to not get the artifacts.

Thanks,
Nicholas
Attachments
noise in frame differencing.jpeg
User avatar
kwagyeman
Posts: 2898
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Tue Jan 29, 2019 2:17 am

Hmm, was resolution are you running at? I know for a fact the frame buffer doesn't respect the frame buffer stack at all. So, it will overwrite it and causes a crash (this is something I will fix soon). The frame buffer stack when pushed does respect what is in the frame buffer and errors out if there was about to be an collision.

Can you post some simple code to see the error and we'll get this fixed quickly.
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Tue Jan 29, 2019 3:15 pm

Thank you for your reply!

I'm using QQVGA right now. But, I also tried it with QVGA and a window of 128x128. The issue arose with both.

I attached a simplify version of my code (basically removed unnecessary sections like servo control and LEDs). In addition, the network file is also included.

Do you know of any alternate solutions / temporary fix like a reset the camera or frame buffer? I have a prototype demo for my project next Tuesday.

Thank you!
Nicholas
Attachments
test issue.zip
(20.55 KiB) Downloaded 234 times
User avatar
kwagyeman
Posts: 2898
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Wed Jan 30, 2019 12:38 pm

Okay, will do. I've asked Ibrahim to look at it but I will be able to take a look over the weekend too.
Nyamekye,
User avatar
iabdalkader
Posts: 732
Joined: Sun May 24, 2015 3:53 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby iabdalkader » Wed Jan 30, 2019 2:05 pm

Hi, I tested your code and saw the artifacts, it's an issue with extra_fb that we will fix very soon. For now, you should save the background image to disk, and pass a path to img.difference. Please let me know if this fixes the issue.
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Wed Jan 30, 2019 8:17 pm

Hello,

Thank you both for your time to help me with the issue!

In order to save to disk, I'll need to use a SD card right? I remember reading that we shouldn't write to the openMV flash.

Once I get a microSD, I'll try your suggestion and let you know if that works. Additionally, may you please inform me once the issue with the extra_fb will be fix as well?

Thank you,
Nicholas
User avatar
kwagyeman
Posts: 2898
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Thu Jan 31, 2019 2:32 am

The issue is that some.of the methods in our code pop the extra allocated fb accidentally causing memory corruption. We have to fix these in a lot of places.

Anyway, just save the background images to disk as bmp files and then pass a path to them from difference and the OpenMV Cam will use the disk as the background image buffer instead until we get this fixed.

...

Yes, you will need a micro SD card.
Nyamekye,
User avatar
iabdalkader
Posts: 732
Joined: Sun May 24, 2015 3:53 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby iabdalkader » Thu Jan 31, 2019 6:08 pm

Hi, this issue has been fixed, please try the attached firmware if you don't want to wait for the next release.
Attachments
firmware.zip
(906.9 KiB) Downloaded 243 times
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Thu Jan 31, 2019 10:20 pm

Hello,

Thank you for the fix! It is working now!!!

I also like the alternative of saving the background image of the microSD card while this wasn't working.

Thanks again for the quick work and great product!
Nicholas
User avatar
kwagyeman
Posts: 2898
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Fri Feb 01, 2019 1:37 am

Hi, you're the first customer to actually use this feature that I know of. For the H7 it's going to be much more useful. Anyway, if you have some time and a blog, a write up on like Hacksters would be appreciated. Likez, for example, sharing your idea, but not he implementation.
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Thu Feb 07, 2019 8:12 pm

Sorry, which feature exactly? The frame differencing with extra frame buffer, the CNN or, both?

I definitely would love to write a blog. I can probably do one at the end of February when my project reaches an end. Currently, I've been sharing short videos on my LinkedIn showing results with the openMV.

Thanks!
User avatar
kwagyeman
Posts: 2898
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Fri Feb 08, 2019 12:30 am

Both at the same time. Generally, folks don't actually want to train a CNN and collect the data for it.
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Tue Feb 12, 2019 8:29 pm

Yeah, I can see why haha! The data collection is very time consuming and challenging. Even then too, you have to be sure to properly capture the data or else you won't have good performance.

I'm actually modifying my program now to use the SD card to save images. This way I'll be able to gather my data with the proper environment.

In any case, I have it in my agenda to make a blog post about my project with the OpenMV. I'll let you know when it's written!
User avatar
kwagyeman
Posts: 2898
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Wed Feb 13, 2019 12:49 am

Awesome!
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Sat Mar 16, 2019 6:59 pm

Hi,

This issue suddenly came back again even on the previous fixed firmware. I see the artifacts after doing the xor for the frame buffer in order to get back the original image after doing a frame differencing.

Doing the following after a frame difference causes the artifacts:
img.b_xor(tempImage)
tempImage.b_xor(img)
img.b_xor(tempImage)


Commenting out the code above shows that there's no artifacts produce. However, I need it in my program because I want to pass the frame to my CNN after.

The issue also occurs when saving the temp image on the SD. It's really the process of getting back the original frame in the buffer.

I attached the code of the full program I'm running.


EDIT: Temporary solution, I left the frame buffer as is after the frame difference and passed the tempImage to the CNN instead. Additionally, after reverting back, I still see the artifacts but now the artifacts aren't causing a trigger for detecting a new object which is a bit odd.

Thanks,
Nicholas
Attachments
trashscan.py
(7.53 KiB) Downloaded 228 times
User avatar
kwagyeman
Posts: 2898
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Sat Mar 16, 2019 8:57 pm

The artifacts are produced by the nn.forward() method correct?
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Sat Mar 16, 2019 9:21 pm

Actually, I've been seeing it from the image.difference method and resetting the frame buffer to normal. Using a allocated frame buffer for the background image is causing the artifacts to be produced after doing the difference. Where as, putting the frame buffer back is causing artifacts for both with an extra fb and with the background image.
User avatar
kwagyeman
Posts: 2898
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Sat Mar 16, 2019 10:01 pm

Do you have a simple test script I can try out? One that doesn't need any files?
Nyamekye,
User avatar
iabdalkader
Posts: 732
Joined: Sun May 24, 2015 3:53 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby iabdalkader » Sun Mar 17, 2019 12:25 pm

This is not related to NN or any other image function. There's an issue with extra_fb in particular and with fb memory in general, I'll try to fix this soon before the next release (the fix will make extra_fb have less memory, so be prepared for that). For now the only workaround is to avoid using extra_fb.
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Sun Mar 17, 2019 5:09 pm

Yes this is what I was deducing it to as well.

Do you know when the next release will be? I'm using the alternative of saving the background but would much prefer to use the fb

Thanks,
Nicholas
User avatar
kwagyeman
Posts: 2898
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Sun Mar 17, 2019 9:31 pm

A PR was submitted to fix this issue. https://github.com/openmv/openmv/pull/489. Attached is the fixed firmware.
Attachments
firmware.zip
(1.82 MiB) Downloaded 225 times
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Mon Mar 18, 2019 9:41 pm

I tried the new firmware but was unable to see the fix because my program got an error right at the beginning.

I was getting "couldn't find file at this line:

Code: Select all

net = nn.load('trashscan-5.network')
However, my file is loaded in the SD. Reverting back to the previous firmware and it's able to find the file.
User avatar
kwagyeman
Posts: 2898
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Tue Mar 19, 2019 12:19 am

Can you put a slash in front of the file name?
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby ngattuso » Tue Mar 26, 2019 7:23 pm

Sorry I haven't answered this in a while. I've been too busy to try out the slash in front.

I just tried to run my program on the latest release (3.3.1). I downloaded it from the Github. The program runs no problem and I didn't have to add the slash. I reimplemented my code to run with the background frame buffer and there's no issue with artifacts.

Thanks,
Nicholas
User avatar
kwagyeman
Posts: 2898
Joined: Sun May 24, 2015 2:10 pm

Re: Artifacts from camera after running CNN and frame differencing

Postby kwagyeman » Wed Mar 27, 2019 12:28 pm

Great, we were able to fix something. Ibrahim is working on the accuracy issues now.
Nyamekye,

Return to “Technical Discussion”

Who is online

Users browsing this forum: No registered users and 5 guests