Frame differencing provides false changes when using on disk reference image

Project Discussion
1

Hey.

I’m facing a difficult problem.
I use basic frame differencing to detect foreign objects.
Lights are controlled and powerful enough.
Camera auto controls are turned off.

When I run in memory frame differencing, everything works perfectly.
But as soon i use on disk frame differencing, problems will arise: after taking reference image and starting frame differencing, system will work as excpected but when time goes, false detections will occur ant differencing number goes up.

I’m using OpenMV H7 Plus camera, resolution is VGA (640x480px) monochrome.

My first guess is that image object is compressed in memory, and uncompressing it will produce different image. Also, when taking a reference image, storing it on disk, and testing it later, is always results differences between reference and actual images even when there is no differences.

It only happens using on disk frame differencing, but i can’t use in memory mode because i need to turn off system daily and always taking a new reference image is no go.

2

Please store images as BMP files which are lossless and not JPEG files.

3

Sorry, i forgot to tell that im using bitmap file.
I made some tests and found that error is smaller when using QVGA (320x240px) frame size.
But like I already mentioned, there will be increasing frame difference number.
I modified the code to perform it using extra framebuffer and then result is better.

import sys, sensor, image, time, os, pyb, time, micropython
from pyb import Pin, SPI


sensor.reset() # Initialize the camera sensor.
sensor.set_pixformat(sensor.GRAYSCALE) # or sensor.GRAYSCALE
sensor.set_framesize(sensor.QVGA) # or sensor.QQVGA (or others)
sensor.set_auto_whitebal(False) # Turn off white balance.
sensor.set_auto_exposure(False)
sensor.set_auto_gain(False)
sensor.skip_frames(time = 3000) # Let new settings take affect.
clock = time.clock() # Tracks FPS.

if not "frame" in os.listdir(): os.mkdir("frame") # Make a frame directory

#sensor.skip_frames(time = 3000) # Give the user time to get ready.

if not "reference.bmp" in os.listdir("/frame"):
    img = sensor.snapshot()
    img.laplacian(1, sharpen=True)
    img.gaussian(2)
    img.save("frame/reference.bmp")
    sensor.snapshot().save
    print("reference image saved")
    
# load reference image
refimage = image.Image("frame/reference.bmp")

extra_fb = sensor.alloc_extra_fb(sensor.width(), sensor.height(), sensor.GRAYSCALE)
extra_fb.replace(refimage)

while(True):


    clock.tick() # Track elapsed milliseconds between snapshots().
    i = 0
    diff = 0
    diffarray = bytearray(10)
    while i < 10:
        sensor.skip_frames(time = 100)
        img = sensor.snapshot() # Take a picture and return the image.
        img.laplacian(1, sharpen=True)
        img.gaussian(2)

        # Replace the image with the "abs(NEW-OLD)" frame difference.
        img.difference(extra_fb)

        hist = img.get_histogram()
    # This code below works by comparing the 99th percentile value (e.g. the
    # non-outlier max value against the 90th percentile value (e.g. a non-max
    # value. The difference between the two values will grow as the difference
    # image seems more pixels change.

        diffarray[i] = hist.get_percentile(0.99).l_value() - hist.get_percentile(0.90).l_value()
        i +=1

    print(diffarray)

Result:

bytearray(b’\x01\x02\x02\x01\x02\x02\x01\x01\x02\x01’)
bytearray(b’\x01\x02\x02\x01\x02\x02\x01\x01\x02\x01’)
bytearray(b’\x01\x02\x02\x01\x02\x02\x01\x01\x02\x02’)
bytearray(b’\x01\x02\x02\x01\x01\x02\x01\x01\x02\x02’)
bytearray(b’\x01\x02\x02\x01\x01\x02\x01\x01\x02\x02’)
bytearray(b’\x01\x02\x02\x01\x01\x02\x01\x01\x02\x02’)
bytearray(b’\x01\x02\x02\x01\x01\x02\x01\x01\x02\x02’)
bytearray(b’\x01\x02\x02\x01\x02\x02\x01\x01\x02\x01’)
bytearray(b’\x01\x02\x02\x01\x02\x02\x01\x01\x02\x01’)
bytearray(b’\x01\x02\x02\x01\x02\x02\x01\x01\x02\x01’)
bytearray(b’\x01\x02\x02\x01\x02\x02\x01\x01\x02\x01’)
bytearray(b’\x01\x02\x02\x01\x02\x02\x01\x01\x02\x01’)

As you see, there is some differences, probably caused by sensor noise.

But when using this code, error will be increasing constantly: like +1 in 30 seconds

import sys, sensor, image, time, os, pyb, time, micropython
from pyb import Pin, SPI


sensor.reset() # Initialize the camera sensor.
sensor.set_pixformat(sensor.GRAYSCALE) # or sensor.GRAYSCALE
sensor.set_framesize(sensor.VGA) # or sensor.QQVGA (or others)
sensor.set_auto_whitebal(False) # Turn off white balance.
sensor.set_auto_exposure(False)
sensor.set_auto_gain(False)
sensor.skip_frames(time = 3000) # Let new settings take affect.
clock = time.clock() # Tracks FPS.

if not "frame" in os.listdir(): os.mkdir("frame") # Make a frame directory

#sensor.skip_frames(time = 3000) # Give the user time to get ready.

if not "reference.bmp" in os.listdir("/frame"):
    img = sensor.snapshot()
    img.laplacian(1, sharpen=True)
    img.gaussian(2)
    img.save("frame/reference.bmp")
    sensor.snapshot().save
    print("reference image saved")
    
# load reference image
#refimage = image.Image("frame/reference.bmp")

#extra_fb = sensor.alloc_extra_fb(sensor.width(), sensor.height(), sensor.GRAYSCALE)
#extra_fb.replace(refimage)

while(True):


    clock.tick() # Track elapsed milliseconds between snapshots().
    i = 0
    diff = 0
    diffarray = bytearray(10)
    while i < 10:
        sensor.skip_frames(time = 100)
        img = sensor.snapshot() # Take a picture and return the image.
        img.laplacian(1, sharpen=True)
        img.gaussian(2)

        # Replace the image with the "abs(NEW-OLD)" frame difference.
        img.difference("frame/reference.bmp") #extra_fb

        hist = img.get_histogram()
    # This code below works by comparing the 99th percentile value (e.g. the
    # non-outlier max value against the 90th percentile value (e.g. a non-max
    # value. The difference between the two values will grow as the difference
    # image seems more pixels change.

        diffarray[i] = hist.get_percentile(0.99).l_value() - hist.get_percentile(0.90).l_value()
        i +=1

    print(diffarray)

I have lot of OpenMV modules, at the moment i’ve tested two of them, and both will produce similar output so its not a single problem caused by faulty hardware.

4

Hmm… it’s probably related to disk i/o traffic causing increased power usage and heat.

Do this. Use the Image() constructor to alloc a new fb on the frame buffer stack with the image from disk. This way you read in the image once versus every call.

This should fix your issue.

5

I had few days busy with other things, started today and encountered a strang memory error:
MemoryError: memory allocation failed, allocating 307200 bytes

It happen when code loads reference image, i stripped code down to few lines and even now it throw exception

import sys, sensor, image, time, os, pyb, time, micropython


micropython.alloc_emergency_exception_buf(100)



refimg = image.Image("frame/reference.bmp")


del refimg
gc.collect()
6

It’s because you aren’t using the frame buffer but the heap.

add “load_to_fb=True” to your Image call.

7

Finally back to openmv.

I stil lhave problems with loading image to perform frame differencing, and it is related with newestfirmware. It does not have any meaning when adding “load_to_fb=True” to Image call.
It still crashes. Now crach occurs even lower size images. I have H7 Plus, it has enough memory i think.
So, how do I debug memory allocation error?

8

It’s copy_to_fb not load_to_fb

https://docs.openmv.io/library/omv.image.html#image.image.Image

9

Yep, just figured it out. It fixed the error, but sometimes it still craches.
Is it possible to increase heap size? I don’t need computing speed, my application has lot of time to make calculations.

10

What do you mean by crashes ? Like a hardfault and cam resets or a memory error ? And no the heap can’t be increased.

11

Test function is:

    def PerformFrameDifferencing(self):
        #print("alustan frame diff func")
        # load reference image
        refimage = image.Image("frame/reference.bmp", copy_to_fb=True)

        #extra_fb = sensor.alloc_extra_fb(sensor.width(), sensor.height(), sensor.GRAYSCALE)
        #extra_fb.replace(refimage)

        #clock.tick() # Track elapsed milliseconds between snapshots().
        i = 0
        diffarray = bytearray(10)
        while i < 10:
            img = sensor.snapshot() # Take a picture and return the image.
            img.gaussian(2)


            # Replace the image with the "abs(NEW-OLD)" frame difference.
            img.difference(refimage)

            hist = img.get_histogram()

            diffarray[i] = hist.get_percentile(0.99).l_value() - hist.get_percentile(0.90).l_value()
            i += 1


        print("frames collected")

        # sort array
        i = 0
        j = 0
        tmp = 0

        while i < 10:
            j = i + 1
            while j < 10:

                if diffarray[i] > diffarray[j]:
                    tmp = diffarray[i]
                    diffarray[i] = diffarray[j]
                    diffarray[j] = tmp
                j += 1

            i += 1
        print("array sorted")

        # remove min and max values to remove some strange sensor noise 
		# that can sometimes occur
        diffarray[0] = 0
        diffarray[1] = 0
        diffarray[8] = 0
        diffarray[9] = 0

        # lets sum and then divide by 6
        print("equalized")
        diff = 0

        i = 0
        while i < 10:
            diff += diffarray[i]
            i += 1

        diff = diff // 6
        print("calculated")

        return diff

Camera Config:

    def PrepareSensor(self):
        sensor.reset() # Initialize the camera sensor.
        sensor.set_pixformat(sensor.GRAYSCALE) # or sensor.GRAYSCALE
        sensor.set_framesize(sensor.VGA) # or sensor.QQVGA (or others)
        sensor.set_auto_whitebal(False) # Turn off white balance.
        sensor.skip_frames(time = 3000) # Let new settings take affect.

        #print("framediff sensor configured")

So far I have seen crach here: refimage = image.Image(“frame/reference.bmp”, copy_to_fb=True)

It craches randomly, most times function will finish without error, and by crach i mean exception, when running from ide, it tells me MemoryError: memory allocation failed, allocating xxx bytes

Other question is memory management, as far as I understand, camera has three types of memory.

  1. framebuffer
  2. stack
  3. heap

How is H7 Plus memory organized? H7 Plus has 1Mbytes of internal ram and 32MBytes of external SDRAM.

How big can framebuffer be?
Stack size?
Heap size?

12

An exception is not a crash, you’re just running out of memory, and it seems random only because it depends on how much memory is used/available at that point. Although when using copy_to_fb it shouldn’t run out of memory, so it could be something else. If you want me to test it attach a test script + reference image (in a zip file) and anything else that allows me to reproduce the error and debug it.

As for the memory configuration it’s in board config files in src/omv/boards/OPENMV*/omv_boardconfig.h on github.

13

Hey, thank you for the answer.

Crach is feedback from field tests :slight_smile:
I need to modify code if i want to send it to you, because my system is not just a openmv camera itself. System includes interface board that manages communication, data isolation, power and light switching and temperature sensors. Communication is made over SPI, openmv camera is master, data frame is 8 bytes. I need to think how to get system to run without interface board.