Artifacts from camera after running CNN and frame differencing

[ngattuso]

Hello,

I noticed an issue where there's artifacts in my frame which is then causing my frame differencing to think that there's a change.

A short description of my program:
I use frame differencing to detect when an object comes in. I count 30 frames with an object before sending the latest frame to my CNN. After checking the frame to the network, I update the background by replacing it with a new sensor.snapshot().

What I noticed is that the artifacts come in after running the CNN at least once. Please see attached image for an idea of what I'm seeing. I have a video but it is too big to attached.

If I run my code without the CNN then the background updates and I don't see the artifacts at the bottom of my frame differencing.

This problem is stopping me from detecting new objects after detecting my first object because the frame differencing is constantly recognizing a difference now.

I was wondering if you have any suggestions on what I can do to either reset the camera to the state it was before the CNN or what I can do to not get the artifacts.

Thanks,
Nicholas

[kwagyeman]

Hmm, was resolution are you running at? I know for a fact the frame buffer doesn't respect the frame buffer stack at all. So, it will overwrite it and causes a crash (this is something I will fix soon). The frame buffer stack when pushed does respect what is in the frame buffer and errors out if there was about to be an collision.

Can you post some simple code to see the error and we'll get this fixed quickly.

[ngattuso]

Thank you for your reply!

I'm using QQVGA right now. But, I also tried it with QVGA and a window of 128x128. The issue arose with both.

I attached a simplify version of my code (basically removed unnecessary sections like servo control and LEDs). In addition, the network file is also included.

Do you know of any alternate solutions / temporary fix like a reset the camera or frame buffer? I have a prototype demo for my project next Tuesday.

Thank you!
Nicholas

[kwagyeman]

Okay, will do. I've asked Ibrahim to look at it but I will be able to take a look over the weekend too.

[iabdalkader]

Hi, I tested your code and saw the artifacts, it's an issue with extra_fb that we will fix very soon. For now, you should save the background image to disk, and pass a path to img.difference. Please let me know if this fixes the issue.

[ngattuso]

Hello,

Thank you both for your time to help me with the issue!

In order to save to disk, I'll need to use a SD card right? I remember reading that we shouldn't write to the openMV flash.

Once I get a microSD, I'll try your suggestion and let you know if that works. Additionally, may you please inform me once the issue with the extra_fb will be fix as well?

Thank you,
Nicholas

[kwagyeman]

The issue is that some.of the methods in our code pop the extra allocated fb accidentally causing memory corruption. We have to fix these in a lot of places.

Anyway, just save the background images to disk as bmp files and then pass a path to them from difference and the OpenMV Cam will use the disk as the background image buffer instead until we get this fixed.

...

Yes, you will need a micro SD card.

[iabdalkader]

Hi, this issue has been fixed, please try the attached firmware if you don't want to wait for the next release.

[ngattuso]

Hello,

Thank you for the fix! It is working now!!!

I also like the alternative of saving the background image of the microSD card while this wasn't working.

Thanks again for the quick work and great product!
Nicholas

[kwagyeman]

Hi, you're the first customer to actually use this feature that I know of. For the H7 it's going to be much more useful. Anyway, if you have some time and a blog, a write up on like Hacksters would be appreciated. Likez, for example, sharing your idea, but not he implementation.

[ngattuso]

Sorry, which feature exactly? The frame differencing with extra frame buffer, the CNN or, both?

I definitely would love to write a blog. I can probably do one at the end of February when my project reaches an end. Currently, I've been sharing short videos on my LinkedIn showing results with the openMV.

Thanks!

[kwagyeman]

Both at the same time. Generally, folks don't actually want to train a CNN and collect the data for it.

[ngattuso]

Yeah, I can see why haha! The data collection is very time consuming and challenging. Even then too, you have to be sure to properly capture the data or else you won't have good performance.

I'm actually modifying my program now to use the SD card to save images. This way I'll be able to gather my data with the proper environment.

In any case, I have it in my agenda to make a blog post about my project with the OpenMV. I'll let you know when it's written!

[kwagyeman]

Awesome!

[ngattuso]

Hi,

This issue suddenly came back again even on the previous fixed firmware. I see the artifacts after doing the xor for the frame buffer in order to get back the original image after doing a frame differencing.

Doing the following after a frame difference causes the artifacts:

img.b_xor(tempImage)
tempImage.b_xor(img)
img.b_xor(tempImage)

Commenting out the code above shows that there's no artifacts produce. However, I need it in my program because I want to pass the frame to my CNN after.

The issue also occurs when saving the temp image on the SD. It's really the process of getting back the original frame in the buffer.

I attached the code of the full program I'm running.


EDIT: Temporary solution, I left the frame buffer as is after the frame difference and passed the tempImage to the CNN instead. Additionally, after reverting back, I still see the artifacts but now the artifacts aren't causing a trigger for detecting a new object which is a bit odd.

Thanks,
Nicholas

[kwagyeman]

The artifacts are produced by the nn.forward() method correct?

[ngattuso]

Actually, I've been seeing it from the image.difference method and resetting the frame buffer to normal. Using a allocated frame buffer for the background image is causing the artifacts to be produced after doing the difference. Where as, putting the frame buffer back is causing artifacts for both with an extra fb and with the background image.

[kwagyeman]

Do you have a simple test script I can try out? One that doesn't need any files?

[iabdalkader]

This is not related to NN or any other image function. There's an issue with extra_fb in particular and with fb memory in general, I'll try to fix this soon before the next release (the fix will make extra_fb have less memory, so be prepared for that). For now the only workaround is to avoid using extra_fb.

[ngattuso]

Yes this is what I was deducing it to as well.

Do you know when the next release will be? I'm using the alternative of saving the background but would much prefer to use the fb

Thanks,
Nicholas

[kwagyeman]

A PR was submitted to fix this issue. https://github.com/openmv/openmv/pull/489. Attached is the fixed firmware.

[ngattuso]

I tried the new firmware but was unable to see the fix because my program got an error right at the beginning.

I was getting "couldn't find file at this line:

Code: Select all

net = nn.load('trashscan-5.network')

However, my file is loaded in the SD. Reverting back to the previous firmware and it's able to find the file.

[kwagyeman]

Can you put a slash in front of the file name?

[ngattuso]

Sorry I haven't answered this in a while. I've been too busy to try out the slash in front.

I just tried to run my program on the latest release (3.3.1). I downloaded it from the Github. The program runs no problem and I didn't have to add the slash. I reimplemented my code to run with the background frame buffer and there's no issue with artifacts.

Thanks,
Nicholas

[kwagyeman]

Great, we were able to fix something. Ibrahim is working on the accuracy issues now.