Memory allocation failed issue on Portenta

[prithulc]

Starting this new thread based on question asked on another one:

@prithulc

Hi @kwagyeman, I am facing roadblocks when I try to use a model after the device goes to deep/lightsleep for a while.

At first, I was loading my model once upon boot and using it multiple times, but after a few iterations and usage of sleep, I started getting invoke failed in my log file (since IDE disconnects when using sleep, I wrote a code to output errors with datetime in a .csv file). I assumed this is because the garbage collector is considering the object unused and deleting it. To work around this, I started calling my model in every iteration like this:

# Model (this is outside the main loop)
def invoke_model():
    return ml.Model("/model/trained.tflite", load_to_fb=uos.stat('/model/trained.tflite')[6] > (gc.mem_free() - (64*1024)))

# Inference (part of the main loop)
def inference(images):
    send_to_log("Starting inference")                           # Send string to .csv log file
    net = invoke_model()                                        # Call model
    prob = [None]*len(images)                                   # Empty array to store predictions
    for i in range(len(images)):                                # Predict using loaded model
        predict = list(zip(labels, net.predict([images[i]])[0].flatten().tolist()))
        prob[i] = predict[1][1]
    del net                                                     # Delete model
    return([prob])

This works fine for a few iterations until I start getting memory allocation failed, allocating 2561328 bytes. At the end of every iteration, I call gc.collect() and print available memory using gc.mem_free which is pretty much around 2740656 for all iterations. I assumed that gc would work for framebuffer too but I could be wrong.

I also read the docs to get some clarity; is there a special method to deallocate fb?

PC

@kwagyeman

Hi, the way you have it coded is fine. Deleting the net object frees the frame buffer stack along with the heap it uses. If the model was in flash then no frame buffer stack would have actually been used.

The memory allocation failed is the TensorFlow flow arena allocation failing and/or allocation failing when invoking.

Large buffers are allocated when the model you are using runs… and when though you can use gc.collect to cleanup the heap it still fragments which results in allocations potentially failing.

You generally want to not constantly load the model over and over again. If the heap is large enough then this would not have problems… are you on the H7 Plus or RT1062?

[prithulc]

@kwagyeman

Deleting the net object frees the frame buffer stack along with the heap it uses

Okay that’s good, thanks for clarifying!

@kwagyeman

are you on the H7 Plus or RT1062?

I’m on Portenta H7 Vision Shield (LoRa)

@kwagyeman

You generally want to not constantly load the model over and over again.

Right, that’s what I’d like to avoid too. When I wasn’t using sleep to save power, this wasn’t happening so my initial hypothesis was that the Portenta is probably losing the pointer to the loaded model or maybe the gc.collect() or (as used during testing) gc.enable() deems it inactive in some random allocation and deletes it.

UPDATE

Tried invoking model only once upon boot again. Works fine for a few iterations but then when it goes to sleep and wakes up for iteration, throws an Invoke failed.

PC

[kwagyeman]

Hi,

This sounds like memory corruption caused by light sleep.

If you only use deep sleep does it work correctly? It should be fine without any sleep too. It sounds like light sleep corrupts memory. We don’t edit the code from MicroPython upstream here so this may be a general bug with the STM32 HAL from MicroPython.

[prithulc]

@kwagyeman

If you only use deep sleep does it work correctly?

Let me test this and get back to you. None of my recent tests are on deepsleep since I realised for my application lightsleep is sufficient.

@kwagyeman

We don’t edit the code from MicroPython upstream here so this may be a general bug with the STM32 HAL from MicroPython.

I see. Maybe I’ll reach out to Micropython through their forum for this. Meanwhile, please let me know if you have any other suggestion or contacts to reach the right person from STM or Micropython.

PC

[kwagyeman]

Verify first that everything is good without the sleeping and then add deep sleep and then switch that too light sleep. This will give you a pretty straight up smoking gun here.

After that, a minimal example of the least amount of lines of code to generate the error will help. Particularly, if you are able to make it happen using standard non OpenMV libraries only.

[prithulc]

Just simulated without machine.lightsleep() for 1 hour and there’s no memory issue. I have been working on this approach for a few months now and all that testing confirms the same thing.

I’m not sure if I can test this using machine.deepsleep() 'cause everytime the Portenta wakes up, the program starts executing again from the beginning as expected since RAM is not retained during deepsleep.

I opened a query on Micropython’s STM discussion page a few days ago after our discussion here. I just posted a minimal program there Memory allocation failed when using machine.lightsleep on Arduino Portenta (STM32H747) · micropython · Discussion #16832

PC

[kwagyeman]

@prithulc - MicroPython folks aren’t going to debug anything related to our firmware. You need to reduce the example down to something that breaks purely on off the shelf Python code without any of our library components and make an extremely minimal example.

… anyway, as for what to do next. Please push Arduino to assign debugging lightsleep() to use and Ibrahim can take a look at this as part of our service contract. Otherwise, please remove lightsleep() from your code.

As for the way to report this to MicroPython: Waking machine.lightsleep(): timer and external interruption · Issue #2621 · openmv/openmv · GitHub

You need to overwrite our firmware with Pure MicroPython code and then test to see if the machine.lightsleep() method is failing.

See also: Installing MicroPython | Arduino Documentation

[prithulc]

Hi @kwagyeman, thanks for your response.

My bad! I thought I was using pure MicroPython libraries there. Well, I did start off like that by compartmentalising each component of the program to see what’s breaking and until I use sensor and ml, lightsleep works fine, and I lost focus on using only MP libs.

@kwagyeman

Please push Arduino to assign debugging lightsleep()

Sorry but there’s no clarity on how to push issues to Arduino. I already tried raising through the contact form (arduino.cc/en/contact-us) when you suggested this last time but have had no response. I’ll try again this way.

@kwagyeman

As for the way to report this to MicroPython: Waking machine.lightsleep(): timer and external interruption · Issue #2621 · openmv/openmv · GitHub

I posted this query. Are you saying this is the right way to raise issue and that I should be raising this memory corruption issue that we are discussing in such manner?

Thank you for your patience, truly.

PC

[kwagyeman]

Hi Prithulc,

They should be monitoring the contact form.

…

Thanks for this: machine.lightsleep() won’t wake using ExtInt() when timer wakeup is used in the same program · Issue #16862 · micropython/micropython

It’s not our firmware issue. In this case Arduino assigning us to fix it would be the best. Otherwise… you need to wait for MP folks to fix.

…

As for what you should do now. Please comment out lightsleep from your code in the meantime. I’ll get back to you if you need to do anything other than contacting Arduino.

[iabdalkader]

Perhaps it’s the SDRAM that gets corrupted on STOP mode entry/exit. Maybe it’s the self-refresh mode, PLLs or something else.
There might be a way to try reproduce this issue with pure MicroPython, without messing with any GC or ML models, if you use uctypes.bytearray_at to create a buffer at say the last MB of SDRAM and fill it with data and check it after wakeup.

uctypes – access binary data in a structured way — MicroPython latest...