4.5.8 Memory Leaks and abnormal behavior (on custom hardware with identical H7 plus components )

OpenMV Boards
85

Yes you can compare any release to a previous release/tag:

Keep in mind that one change could be updating a submodule with 100s of other changes

Is it possile to change the working freq of SDRAM via python script?

No you have to rebuild the firmware to change that, and the timings must match the frequency and exact SDRAM part.

86

I thought form other posts in the last years you might have remembered me. Sorry about that. I did indeed think that bootstrapping hardware development aspect of machine vision projects was also part of the concept behind your project.

If you can point me in the right direction Iā€™d be greatful and I can continue to evolve our product with your sensors.

87

I do remember you and your projects, but I donā€™t remember that youā€™re using a custom hardware. This is a very relevant piece of info to mention especially when weā€™re trying to reproduce on different hardware. But at least testing it on our boards seems to show that the issue is in yours.

This doesnā€™t include providing support for hardware that we didnā€™t build, our time and resource can never allow that (even if itā€™s paid support, weā€™re usually very busy). But even if we try we couldnā€™t possibly know what might be wrong with your hardware, someone may need to go through the schematics and layout. BTW this could still be a firmware issue just only showing on your hardware.

If you can point me in the right direction Iā€™d be greatful and I can continue to evolve our product with your sensors.

The SDRAM config is here: micropython/ports/stm32/boards/OPENMV4P/mpconfigboard.h at openmv-1.23 Ā· openmv/micropython Ā· GitHub

And instructions to build the firmware are here: openmv/src/README.md at master Ā· openmv/openmv Ā· GitHub

88

Hi, not to break the thread, but, hereā€™s a fix for your performance issue.

Please do not use the ImageIO object as a FIFO. This is already built into the sensor driver. You just need to set the number of frame buffers to 950 and disable full flush. After doing so it will accumulate frames at the sensor frame rate and never drop anything once it fill up. As long as you only then read 950 frames you will see continuous frames in a row.

After reading 950 frames toggling the resolution is enough to reset the frame buffer to use again.

The API for this is weird because the feature is designed for being used as an elastic buffer for SD card recording. E.g. itā€™s not meant to need to be reset. However, you can also use it as a high speed fifo. I may improve the API in the future to make this more obvious.

The script below hits the expected FPS fo 371.

# This work is licensed under the MIT license.
# Copyright (c) 2013-2023 OpenMV LLC. All rights reserved.
# https://github.com/openmv/openmv/blob/master/LICENSE
#
# Image Memory Stream I/O Example
#
# This example shows how to use the ImageIO stream to record frames in memory and play them back.
# Note: While this should work on any board, the board should have an SDRAM to be of any use.
import sensor
import image
import time

# Number of frames to pre-allocate and record
N_FRAMES = 950

sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE)
sensor.set_framesize(sensor.HQQQVGA) #resolution
sensor.set_windowing([3,13,57,24])
sensor.set_auto_exposure(False, exposure_us = 2700)
sensor.skip_frames(100)
sensor.set_framebuffers(N_FRAMES)
sensor.disable_full_flush(True)

clock = time.clock()

while True:
    for i in range(N_FRAMES):
        #print("Playing Frame" , i)
        clock.tick()
        img = sensor.snapshot()
        median1 = img.get_statistics(roi=[36,0,21,4]).median()
        median2 = img.get_statistics(roi=[2,0,4,5]).median()
        stdev1 = img.get_statistics(roi=[35,0,9,18]).stdev()
        stdev2 = img.get_statistics(roi=[30,0,27,14]).stdev()
        print(clock.fps())
        # Do machine vision algorithms on the image here.
    # Toggle res to reset frame buffer fifo.
    sensor.set_framesize(sensor.HQQVGA) #resolution
    sensor.set_framesize(sensor.HQQQVGA) #resolution
    sensor.set_framebuffers(N_FRAMES)
    sensor.disable_full_flush(True)
89

Thanks. I get 371 FPS with this. It does crash at a certain point like the other example which makes me wonder even more what the problem may be since there is no SDRAM involved.

I cannot use it in my application because a first function extracts features which tags image frames based on statistical data and then another function after that takes the tagged frames and passes through ML to get more data and finally come out with an output. The classification part where ML comes in, needs to know information beyond that of the current frame. Maybe re architecturing the code I can find a way to make this work but I doubt it and I put signifcant effort to get what I have stable.

I will wait for the next official release. If I can take snapshots and write to SDRAM as fast as before , Iā€™ll try to tweak SDRAM timing together with chat gpt to see if I can get rid of the crashing.

Otherwise I ā€˜ll stick to good ole 4.5.5 for the remainder of my productsā€™ lifecycle.

Appreciate all your help.

90

Thanks. I get 371 FPS with this. It does crash at a certain point like the other example which makes me wonder even more what the problem may be since there is no SDRAM involved.

Hi, everything involves SDRAM with the frame buffer. However, running the above script on my OpenMV Cam H7 Plus with the latest dev release does not crash.

91

I get 371 FPS on my custom boards but it DOES crash.

So the perofrmance of the connection between STM32 and SDRAM must be inferior to yours on my boards. the section of my board is identical to yours except for the traces, We have one huge board with all electronics and connetors required. I didnā€™t have the space and didnā€™t want to manage a bunch of wires so I opted at the time to do it that way.

Since tests were always fine we never suspected a problem. Although not 100% confirmed I think we have enough data to say there is most likely a SDRAM timing issues.

Iā€™ll try to fix with the next stable release in the firmware if I am able to write snapshots at 370 FPS as per your examples so that I can retrieve the images at later instances.

92

You should then compile your own custom images and fix the SD timings for your board per Ibrahimā€™s last post.

93

I will indeed if in a future release I get 370FPS with the same method as beforeā€¦ Thanks for the tips. I hope something useful came out of this for you.

In any case before shipping out our product I have to compile the firmware with the py files so they arent accessible from flash so I will have to learn how to do that and can try lowering the SDRAM timing to see if I have any luck there.

Have a nice day.

94

https://docs.openmv.io/openmvcam/tutorial/production.html

95

Yeah thanks I had that link in my todo list as a last step.

96

If this is the case please keep me in mind and let me know if you do come accorsss a bug of that kind. Thanks once again for your support.

97

Hi Joe,

One thought, you can use the frame buffer fifo with your ImageIO fifo. They can be used at the same time. However, you will not have any performance issues anymore, crashing aside.

98

How would that look in the example code? Im afraid I canā€™t follow. Ultimately I need to be able to record a sequence of images and recall them any time.

99

You just write to the ImageIO object via snapshot like you were doing before. However, because the frame buffer fifo is set to 950 there is no possibility of any frames during the 950 being dropped. Note the the FPS may not be 317, but, this is fine as no frames are dropped which is what you care about.

100

Good news and maybe something useful for you to know. I don^t know why it would be connected to my boards but if I throw in a gc.collect() every 100 loops I get a stable run. Using latest dev release where also FPS performance is back to 370. Tested for 1 hour no crashes.

Also I tested the script that was crashing on 3 different custom boards and they all have the same result which leads me to believe even more that it is a SDRAM timing issue if it is a hardware issue and not corrupt memory sectors.

Works same also with FIFO buffer:

sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE)
sensor.set_framesize(sensor.HQQQVGA) #resolution
sensor.set_windowing([3,13,57,24])
sensor.set_auto_exposure(False, exposure_us = 2700)
sensor.skip_frames(100)
sensor.set_framebuffers(N_FRAMES)
sensor.disable_full_flush(True)

import sensor
import image
import time
import pyb
import gc

# Number of frames to pre-allocate and record
N_FRAMES = 950

sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE)
sensor.set_framesize(sensor.HQQQVGA) #resolution
sensor.set_windowing([3,13,57,24])
sensor.set_auto_exposure(False, exposure_us = 2700)
sensor.set_framebuffers(3)
sensor.skip_frames(100)

clock = time.clock()

# Write to memory stream
stream = image.ImageIO((57, 24, sensor.GRAYSCALE), N_FRAMES)

for i in range(0, N_FRAMES):
    clock.tick()
    stream.write(sensor.snapshot())
    print(clock.fps())

loop_counter = 0

while True:
    gc.collect()
    stream.seek(0)
    print("Start new loop", loop_counter)
    loop_counter += 1
    for i in range(N_FRAMES):
        clock.tick()
        img = stream.read(copy_to_fb=False, pause=False, loop = False)
        median1 = img.get_statistics(roi=[36,0,21,4]).median()
        median2 = img.get_statistics(roi=[2,0,4,5]).median()
        stdev1 = img.get_statistics(roi=[35,0,9,18]).stdev()
        stdev2 = img.get_statistics(roi=[30,0,27,14]).stdev()
        if (i % 100 == 0):  # Every 100 frames
            gc.collect()
101

Can it be that if gc.collect() is not called frequently enough, auto gc kicks in during a memory read and crashes the memory reading process?

102

No but calling gc.collect() could prevent GC from ever needing to use the secondary GC block which is located in SDRAM. Why would using the secondary GC block located in SDRAM cause a crash ? Iā€™m not sure. You can try to disable it by commenting out these two lines and rebuilding the firmware:

#define OMV_GC_BLOCK1_MEMORY                  DRAM   // Extra GC block 0.
#define OMV_GC_BLOCK1_SIZE                    (4M)
103

This problem vanished with 4.5.8 release. Will test with latest release and not report back if it works The workaround I had found was not robust but now all looks good. Thanks I appreciate your continuous improvement process.

2 Likes