Memory Corruption with Image.image(ndarray)?

OpenMV Boards
1

Hi, there,

I have encountered a similar issue to this post regarding creating an image on the fly from an ulab ndarray and then convert it back to an ndarray. The code I have is pretty lengthy so I will try describing it first. If we need a minimum example to replicate the problem, I can provide that too.

In our case, I did not use sensor.alloc_extra_fb to host the image on the stack. Instead, the image is created as a local variable from a local ndarray in a function called at each iteration of a loop. After applying .morph() method on the image for the purpose of convolution, the image is converted back to an ndarray and we extract the information from it, leaving the local scope without explicitly calling del methods.

The problem is, this method seemingly “corrupts” random memory locations, specifically the step of creating the image from the ndarray and converting it back to an ndarray. In more details, we implemented the aforementioned function in a class. The class has some member variables and functions. The behaviors that lead to this suspicion include the followings:

  1. A member float variable (that serves as a constant in the code) has its sign flipped.
  2. Accessing a member function throws a NotImplementedError with null opcode
  3. Accessing an existing member variable throws AttributeError, stating the class does not have “”
  4. draw_rectangle() does not draw anything on the snapshot() image. # ← This one happens less often so it might be the values I get from the ndarray are not correct.
  5. A board crash which causes reconnection to the computer/everything onboard (LED states, UART communication) to halt forever. After reconnection, the error trace would make no sense as they refer to functions that we had but no longer exist, for example.
    These behaviors would happen at random time from starting the program. 2., 3., and 5. would stop the program. 1. and 4. would keep the program running but the results basically become invalid. Only one of these behaviors would happen at one runtime. If I remove the image creation and the conversion to ndarray, none of the behaviors happen any more.

We tested the same program on both NiclaVision (Firmware 4.5.9) and OpenMV RT1062 (Firmware 4.6.0). It happens more often and sooner on NiclaVision which made me lean toward that suspicion more since Nicla has way less RAM than RT1062.

2

An update:
Creating a buffer using sensor.alloc_extra_fb(num_cols, num_rows, sensor.GRAYSCALE).bytearray() and loading the image using img = image.Image(ndarray_img, buffer=self.grid_img_bf) definitely fixed the problem on both Nicla and OpenMV. It might indicate that creating an image.Image on the heap is buggy now.

3

Can you make a bug tracker on github with a code sample that produces the error? Please also list the OpenMV Cam models and versions. Image.Image() works well for FOMO image processing. We just xalloc the expected bytes an image needs versus grabbing a pointer to the buffer.

My hunch would be that the error is from memory corruption on the heap and you just see the error more easily when using the heap for large arrays versus the frame buffer stack.

4

Thank you for the reply! I have created a bug tracker on Github

Also an update, it turns out storing the image on an extra frame buffer did not fix the issue, neither did creating a bytearray on the heap as the buffer

5

Yeah, this might be an issue in Numpy. That library is NOT free of bugs. I ran into a bunch when trying to write Non-max-Supression code. There are methods in the lib which just crash the system.

I’d do this by commenting out the numpy code and adding it back until things break. This should help you determine where there’s an error. Alternatively, replace things in numpy with trying to do it in Python.

1 Like
6

Another update: after more tests and breaking down numpy operations, I figured that the source of problem was mostly possibly not the creation of the image using image.Image(ndarray) but converting it back to the ndarray with
ndarray_morphed_metric = metric_image.to_ndarray(dtype=np.uint8).flatten(). It seems that converting the grayscale image back to a uint8/'B' ndarray corrupts the memory. I have changed the code to ndarray_morphed_metric = metric_image.to_ndarray(dtype='f').flatten(). On both NiclaVision and OpenMV Cam RT1062, the previously mentioned error did not occur any more on even hour-long program execution.