4.5.8 Memory Leaks and abnormal behavior (on custom hardware with identical H7 plus components )

OpenMV Boards
64

So IDE 4.2.1 else all as reported last:

Recording goes smooth, crashing always in while loop.


import sensor
import image
import time
import pyb

# Number of frames to pre-allocate and record
N_FRAMES = 950

sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE)
sensor.set_framesize(sensor.HQQQVGA) #resolution
sensor.set_windowing([3,13,57,24])
sensor.set_auto_exposure(False, exposure_us = 2700)
sensor.set_framebuffers(3)
sensor.skip_frames(100)

clock = time.clock()

# Write to memory stream
#stream = image.ImageIO((57, 24, sensor.GRAYSCALE), N_FRAMES)
stream_out = image.ImageIO("Test_file.bin", "w")

for i in range(0, N_FRAMES):
    clock.tick()
#    stream.write(sensor.snapshot())
    stream_out.write(sensor.snapshot())
    print(clock.fps())
print("DONE Recording")
pyb.delay(2000)

stream_out.close()

while True:
    # Rewind stream and play back
    stream_in=image.ImageIO("Test_file.bin", "w")
    stream_in.seek(0)
    for i in range(0, N_FRAMES):
        print("Playing Frame" , i)

#        img = stream.read(copy_to_fb=False, pause=False)
        img = stream_in.read(copy_to_fb=False, pause=False, loop=False)
        median1 = img.get_statistics(roi=[36,0,21,4]).median()
        median2 = img.get_statistics(roi=[2,0,4,5]).median()
        stdev1 = img.get_statistics(roi=[35,0,9,18]).stdev()
        stdev2 = img.get_statistics(roi=[30,0,27,14]).stdev()
        print(median1)
        # Do machine vision algorithms on the image here.
stream_in.close()
print("DONE Looping")
65

Can you record a stream with one script, load it with the other, and if it crashes attach the stream here ? I just want to replicate the exact same setup you have. In the meantime Iā€™m trying the script again and checking for other issues.

66

Test_file_1.bin (1.3 MB)
Produced with:


import sensor
import image
import time
import pyb

# Number of frames to pre-allocate and record
N_FRAMES = 950

sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE)
sensor.set_framesize(sensor.HQQQVGA) #resolution
sensor.set_windowing([3,13,57,24])
sensor.set_auto_exposure(False, exposure_us = 2700)
sensor.set_framebuffers(3)
sensor.skip_frames(100)

clock = time.clock()

# Write to memory stream
#stream = image.ImageIO((57, 24, sensor.GRAYSCALE), N_FRAMES)
stream_out = image.ImageIO("Test_file_1.bin", "w")

for i in range(0, N_FRAMES):
    clock.tick()
#    stream.write(sensor.snapshot())
    stream_out.write(sensor.snapshot())
    print(clock.fps())
print("DONE Recording")
pyb.delay(2000)

stream_out.close()

Crashed reproduced with:
import image
import time
import pyb

# Number of frames to pre-allocate and record
N_FRAMES = 950


while True:
    # Rewind stream and play back
    stream_in=image.ImageIO("Test_file_1.bin", "w")
    stream_in.seek(0)
    for i in range(0, N_FRAMES):
        print("Playing Frame" , i)

#        img = stream.read(copy_to_fb=False, pause=False)
        img = stream_in.read(copy_to_fb=False, pause=False, loop=False)
        median1 = img.get_statistics(roi=[36,0,21,4]).median()
        median2 = img.get_statistics(roi=[2,0,4,5]).median()
        stdev1 = img.get_statistics(roi=[35,0,9,18]).stdev()
        stdev2 = img.get_statistics(roi=[30,0,27,14]).stdev()
        print(median1)
        # Do machine vision algorithms on the image here.
stream_in.close()
print("DONE Looping")

With looping from .bin I sometimes manage to get through 1 loop of 950 whereas when looping from SDRAM it always crashes during first loop.

67

Seems like latested released IDE is 4.2.1. Only if you think can add value should I download and install the IDE dev release?

68

Yes please give a test, itā€™s unlikely, but the stable does have some issues, like sending repeated commands etcā€¦

One last question, do you store the stream on SD card or internal flash ? I saved it to internal flash.

69

Typically I always use SD card. This time given it was a 1 off recording , although signifcantly slower I used flash because SD card is in the other room.

70

Thatā€™s good, letā€™s keep using the flash we donā€™t want to introduce another variable. Iā€™ve made a small change to the script to print the iteration, and itā€™s going through the 10th playback, with no crash. I will keep testing this and get back to you. Thereā€™s another issue with USB/CDC/text buffer that might be related, so will try to fix that first.

71

I tested with different boards and I donā€™t get the problem.
Only diffrerence is that the SPI isnā€™t wired on those.

I will proceed to try to keep the master processor (the one that sends out SPI commands) reset whilst running the openmv script. In theory I have been testing with no messages going out from the master but it may be that anyway somehow it is interacting.

72

Note that Iā€™m testing with OpenMV-4 Plus with nothing attached to it.

Does the development release change anything ?

73

With new dev release (firmware and ide) I got through 1 loop and then after crash.
I think maybe print buffer on ide may be different so I wouldnā€™t make much of that.
Will report shortly on test isolating the master processor. I cannot physically disconnect SPI connections but I can put the processor in permanent reset.

74

With Master Processor (SPI Master) in reset script still crashes. Not sure what state the OMV pins are when in reset state.
The SPI pins are routed on my pcb together I cannot disconenct them.

Can I get the Openmv to ignore those Pins somehow in the script I provided to try to isolate?

Maybe has nothing to do with that but trying to replicate identical situation .

75

image
On the right in red the Openmv pins as labled in your spec.
On the Left Inputs/outputs to master proc.

The pin readouts from OMV are such when main proc is in reset:
Initial P0: 1
Initial P1: 1
Initial P2: 1
Initial P3: 1
Initial P4: 1
Initial P5: 0
Initial P6: 1
Initial P7: 1
Initial P8: 1
Initial P9: 0

76

Played around with new dev release. I managed more than once to get through 2 to 3 loops. Something improved with last dev release. With prior dev release crashed always on first loop. On 4.5.8 never crashed.

I often get the error:
AttributeError: ā€˜strā€™ object has no attribute ā€˜medianā€™

77

I am about to write a script to check each memory sector to see if I can write / read. Could be a HW failure - > i.e.a damaged sector .

Do you have such a script already?

78

No, but thereā€™s an SDRAM self-test thatā€™s enabled by default, but by default it runs in quick mode. You can change it to exhaustive mode by setting this to true in main.c

sdram_test(false);
79

Iā€™m still not setup to compile firmware (on my todo list). Could you provide me a build? I donā€™t know your opinion but I think this might very well be the reason that somehow is masked with 4.5.8 and prior releases. Where are the faults logged after the test?

80

Yes but I need to fix something first before I can send you a firmware image. In the meantime, if you think itā€™s a hardware issue, you should try a different board. This is not a custom board right ?

81

I thried other openmv boards and cannot replicate crash.

The boards in my product have exact same components and layout of the openmv board and have been using for 3 years. I needed custom to integrate all electronics on one board for my product.

This particular board has been in a prototype since 1 year and never had issues.

I will test another custom board and see if issue persists.

Questions:

  1. Is it possible to have a build where I can set a flag to turn off whatever is being done differenty from 4.5.8 in terms of memory management? Would that make sense?

  2. Can we also have an option to turn DMA off? The no DMA build effectively made such that there were no crahes.

  3. Is the loew camera sensor performance understood or should I expect lower perfromance in the next releases?

  4. If there are damaged SDRAM memory sectors , which can happen , does the firmware cope dynamicaly or would one have to specify in the firmware to skip some memory regions (assuming they are identifiable)

82

You should have mentioned that this is custom board. While we support our products, we donā€™t have the resources to support yours as well. We canā€™t possibly know whatā€™s wrong with your hardware or how recent firmware changes may have affected it.

I doubt that as we normally donā€™t share the layout for high-end boards, and if your layout is different, it might affect SDRAM timings.

No that doesnā€™t make any sense. There were about 300+ changes since release 4.5.5 that span a 3 repositories, may as well just use the older firmware.

Youā€™re getting crashes on a script that doesnā€™t use SPI at all so Iā€™m not sure how disabling SPI DMA will help, but we canā€™t add this anyway.

The performance will be restored for sure. As for SDRAM, I donā€™t think a PC can do that let alone a microcontroller with a single-chip DRAM.

I do see a different bug now in the dev firmware, but itā€™s been there after the stable release, so itā€™s unlikely that itā€™s affecting you, but will be fixing it anyway.

83

I wrote a script to check all SDRAM memory sectors. Script was crashing on my boards but not on openmv boards. I introduced delays to try to capture failures. The script ran to completion on the entire memory and didnā€™t find any faults.

I went back to our script to reproduce the failure and introduced a delay after getting statistics
1ms ā†’ crashing
100ms ā†’ not crashing
50 ms ā†’ crashing

Seems like something with the timing. Indeed related to my boards. I cannot work with delays greater than 1 ms in that loop .

Do you have reccomednations ?
Can you pinpoint what change between 4.5.5 and dev release is causing this behavior now?

I wouldnā€™t suspect RAM because that is part of the chip but SDRAM is an external periphial and the impedance of the connections on my boards might be slightly different then yours thus maybe more sensitive to timing.

Is it possile to change the working freq of SDRAM via python script?