Arduino Nicla out of Fast Frame Buffer Stack Memory (it's different this time, I promise)

Arduino Boards
1

Hello,

I really respect and appreciate all the work in the OpenMV project, so I’ve tried to browse as much as possible. I know memory is very limited on this board. I am looking for apriltags in a loop. Each loop runs perfectly fine, but at a random number of loops (sometimes 10, sometimes 100), I run out of memory. The funny thing is, this only happens when there are no apriltags in frame. When there are some, it works fine for whatever reason.

Maybe it is related to this discussion: Openmv AprilTag--Out of fast Frame Buffer Stack Memory! - #11 by kwagyeman

Idk enough about the apriltag algo. I would’ve imagine it has a bounded memory use. I tried using gc.collect() and deleting the image every loop, but that did not seem to work.

import math, sensor, time, machine, image
import gc

sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE)
sensor.set_framesize(sensor.QVGA)
sensor.ioctl(sensor.IOCTL_SET_FOV_WIDE, True) # doesn't seem to do anything
sensor.set_auto_exposure(False, exposure_us=500)
sensor.set_auto_whitebal(False)  # must be turned off for color tracking
sensor.skip_frames(time=200)

# QVGA Camera properties
sensor_pixel_h = .00175 # mm
sensor_pixel_w = .00175 # mm
qvga_sensor_focal_len_avg_px = 450 # px
qvga_sensor_focal_length_mm = qvga_sensor_focal_len_avg_px * sensor_pixel_h
qvga_x_res = 320
qvga_y_res = 240
qvga_sensor_w_mm = sensor_pixel_w * qvga_x_res
qvga_sensor_h_mm = sensor_pixel_h * qvga_y_res
qvga_f_x = (qvga_sensor_focal_length_mm / qvga_sensor_w_mm) * qvga_x_res
qvga_f_y = (qvga_sensor_focal_length_mm / qvga_sensor_h_mm) * qvga_y_res
qvga_c_x = qvga_x_res / 2
qvga_c_y = qvga_y_res / 2
QVGA_SEARCH_IMG_AREA = qvga_x_res * qvga_y_res

def perform_coarse_search(img, coarse_scale_x, coarse_scale_y):
    coarse_img = img.crop(x_scale = coarse_scale_x, y_scale = coarse_scale_y, hint = image.BILINEAR, copy=True)
    tag_list = coarse_img.find_apriltags(
        fx=qvga_f_x * coarse_scale_x,
        fy=qvga_f_y * coarse_scale_y,
        cx=qvga_c_x * coarse_scale_x,
        cy=qvga_c_y * coarse_scale_y)
    return tag_list

def find_apriltag_optimized(img):
    scale_x = scale_y = .25
    tag_list = perform_coarse_search(img, scale_x, scale_y)
    if not tag_list:
        scale_x = scale_y = .5
        tag_list = perform_coarse_search(img, scale_x, scale_y)
    return tag_list, scale_x, scale_y

def draw_apriltags(img, tag_list, scale_x, scale_y):
    scaling_factor_x = 1/scale_x
    scaling_factor_y = 1/scale_y
    for tag in tag_list:
        img.draw_rectangle((
            int(tag.rect[0]*scaling_factor_x),
            int(tag.rect[1]*scaling_factor_y),
            int(tag.rect[2]*scaling_factor_x),
            int(tag.rect[3]*scaling_factor_y),
        ))
        img.draw_cross(int(tag.cx*scaling_factor_x), int(tag.cy*scaling_factor_y))


def translation_to_mm(translation, tag_size):
    # translation is in decimeters...
    return ((translation * 100) * tag_size) / 210

def find_distance(tag):
    return math.sqrt(
        translation_to_mm(tag.x_translation, 100)**2 +
        translation_to_mm(tag.y_translation, 100)**2 +
        translation_to_mm(tag.z_translation, 100)**2
    )

# Main Loop
clock = time.clock()
last_hb_time = time.ticks_ms()
i = 0
while True:
    clock.tick()

    img = sensor.snapshot()
    tag_list, scale_x, scale_y = find_apriltag_optimized(img)
    if tag_list:
        draw_apriltags(img, tag_list, scale_x, scale_y)
    machine.idle()

    i+=1
    print("Loop", i, "- Free mem:", gc.mem_free())
2

I added a gc.collect() to the end of the loop using the latest firmware and it appears to be fine? It made it past 1K iterations.

Loop 1093 - Free mem: 342512
Loop 1094 - Free mem: 342512
Loop 1095 - Free mem: 342512
Loop 1096 - Free mem: 342512
Loop 1097 - Free mem: 342512
Loop 1098 - Free mem: 342512
Loop 1099 - Free mem: 342512
Loop 1100 - Free mem: 342512
Loop 1101 - Free mem: 342512
Loop 1102 - Free mem: 342512
Loop 1103 - Free mem: 342512
Loop 1104 - Free mem: 342512

Using the latest firmware. 4.6.20

3

@kwagyeman

I am now seeing this issue with the OpenMV AE3. See the example below. When I try setting the scale any higher “.02”, it gives the out of memory error:

Traceback (most recent call last):
File “”, line 18, in
MemoryError: Out of fast frame buffer stack memory
OpenMV v4.8.1; MicroPython v1.26.0-77; OpenMV-AE3 with AE302F80F55D5AE
Type “help()” for more information.. :frowning:

import csi
import time

csi0 = csi.CSI()
csi0.reset()
csi0.pixformat(csi.GRAYSCALE)
csi0.framesize(csi.QVGA)
csi0.snapshot(time=2000)

clock = time.clock()

while True:
clock.tick()
img = csi0.snapshot()
img.crop(x_scale=.01, y_scale=.01)
tag_list = img.find_apriltags()
print(clock.fps())

image
image180×123 207 Bytes

4

Hi, why would you want to scale by 1/100th in each direction?

5

Haha, I do not. I was trying to find the max size image I could pass without getting the memory error. I fixed this by flashing my own firmware with this change to boards/OPENMV_AE3/imlib_config.h

// Enable high res find_apriltags() - uses more RAM
// #define IMLIB_ENABLE_HIGH_RES_APRILTAGS

I just commented out this line, recompiled and flashed, and it works now! I think something in this high res business was immediately sucking up all the memory.

In my code example, if you set x_scale and y_scale any higher, say 0.02, you should run in to that fast frame buffer error.

1 Like