What is the best way to check memory usage?

Discussion related to "under the hood" OpenMV topics.
GregR
Posts: 9
Joined: Thu Sep 14, 2017 12:23 am

What is the best way to check memory usage?

Postby GregR » Sat Sep 30, 2017 11:13 pm

I have a project that has about 500 lines of code and one class in an external .py file. I am getting memory errors when I try to start the project.

Code: Select all

print('Used: ' + str(gc.mem_alloc()) + ' Free: ' + str(gc.mem_free()))
Claims that I have about 23K free yet the errors I get complain about memory allocation failed for hundreds of bytes.
The code that pushes the project over the edge doesn't involve any large memory allocations. I can light switch the problem by adding and removing a dozen lines of basic code.
What is the best way to troubleshoot problems like this?

Regards,

Greg
User avatar
kwagyeman
Posts: 1175
Joined: Sun May 24, 2015 2:10 pm

Re: What is the best way to check memory usage?

Postby kwagyeman » Sat Sep 30, 2017 11:19 pm

Can you post the code that's the issue? Python hides a lot of memory alloc sometimes. Usually, heap fragmentation is a problem if you allocate large things and remove them repeatedly.
Nyamekye,
GregR
Posts: 9
Joined: Thu Sep 14, 2017 12:23 am

Re: What is the best way to check memory usage?

Postby GregR » Sat Sep 30, 2017 11:29 pm

Hi Kwagyeman,

I can't post all of it as it's for a client. I'm using the OpenMV as rapid proto tool...

I have the project working in pieces and didn't hit the problem till I tried to combine the parts.
The interesting thing is that I'm using some dotstar LEDs from adafruit for the illumination.
I was able to lightswitch the problem by just adding or removing a block like:

Code: Select all

dotStarWhite = [0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A]

illuminatorWhite = dotStarPrefix
illuminatorWhite += bytes(dotStarWhite)
illuminatorWhite += dotStarSuffix
Regards,

Greg
User avatar
kwagyeman
Posts: 1175
Joined: Sun May 24, 2015 2:10 pm

Re: What is the best way to check memory usage?

Postby kwagyeman » Sun Oct 01, 2017 12:28 am

From my knowledge of MicroPython that array is actually 4X larger than you think since each number is stored in 4 bytes. Then, MP will allocate a new array on the += and another new array on the += again. Your script takes space too in RAM when it's parsed into a byte array.

...

This seems to work on my M7:

Code: Select all

import gc
print('Used: ' + str(gc.mem_alloc()) + ' Free: ' + str(gc.mem_free()))
dotStarPrefix = [0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A]
print('Used: ' + str(gc.mem_alloc()) + ' Free: ' + str(gc.mem_free()))
dotStarWhite = [0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A,\
                0xFF,0x0F,0x0F,0x0A]
print('Used: ' + str(gc.mem_alloc()) + ' Free: ' + str(gc.mem_free()))
illuminatorWhite = dotStarPrefix
print('Used: ' + str(gc.mem_alloc()) + ' Free: ' + str(gc.mem_free()))
illuminatorWhite += bytes(dotStarWhite)
print('Used: ' + str(gc.mem_alloc()) + ' Free: ' + str(gc.mem_free()))
illuminatorWhite += dotStarPrefix
print('Used: ' + str(gc.mem_alloc()) + ' Free: ' + str(gc.mem_free()))
print(illuminatorWhite[0])
print('Used: ' + str(gc.mem_alloc()) + ' Free: ' + str(gc.mem_free()))
It prints:

Used: 3072 Free: 52816
Used: 3440 Free: 52448
Used: 3792 Free: 52096
Used: 4000 Free: 51888
Used: 4432 Free: 51456
Used: 4928 Free: 50960
255
Used: 5120 Free: 50768
Nyamekye,
GregR
Posts: 9
Joined: Thu Sep 14, 2017 12:23 am

Re: What is the best way to check memory usage?

Postby GregR » Mon Oct 02, 2017 3:27 pm

That is interesting and certainly good to know about the 4x.
The part I didn't explain very well is that adding or removing that block of code can light switch the problem.
Based on your experiment we are talking about roughly 2k.

Code: Select all

print('Used: ' + str(gc.mem_alloc()) + ' Free: ' + str(gc.mem_free()))
Claims that I still have around 20k available.
Any suggestions for further troubleshooting this?

Regards,

Greg
User avatar
kwagyeman
Posts: 1175
Joined: Sun May 24, 2015 2:10 pm

Re: What is the best way to check memory usage?

Postby kwagyeman » Mon Oct 02, 2017 7:03 pm

Mmm, since I can't see the code it's kinda hard. Can you make some code that shows the error that I can try? I can debug then.

One thing to also try is to preallocate. I.e. allocate a bunch of large things first, then, deallocate one, put a new thing in it's place, and keep going. The heap may be fragmented where it can't find a large chunk free. I guess we should add a call to MP that returns the max free block size available.

By allocating your large items first you free prevent space they use from being fragmented.
Nyamekye,
User avatar
iabdalkader
Posts: 363
Joined: Sun May 24, 2015 3:53 pm

Re: What is the best way to check memory usage?

Postby iabdalkader » Wed Oct 04, 2017 7:53 pm

GregR wrote:
Mon Oct 02, 2017 3:27 pm
Any suggestions for further troubleshooting this?
Hi, yes use this to dump GC alloc table:

Code: Select all

import machine
machine.info(1)
Dots are free blocks, anything else is an allocated GC block. Note 1 GC block == 16 bytes.

Return to “Technical Discussion”

Who is online

Users browsing this forum: No registered users and 1 guest