Find_displacement with usb_vcp

I tried the following code as on Nicla Vision Board

import sensor, ustruct, pyb, time, image

usb_vcp = pyb.USB_VCP()

green_led = pyb.LED(2)
blue_led  = pyb.LED(3)


originList = [(207, 282), (204, 194), (202, 106), (200, 19), (161, 238), (159, 151), (158, 64), (118, 283), (117, 195), (115, 108), (113, 21), (75, 240), (72, 152), (71, 66), (30, 197), (30, 285), (27, 109), (25, 21)]
extra_fb = sensor.alloc_extra_fb(sensor.width(), sensor.height(), sensor.RGB565)

while True:
    command = usb_vcp.recv(4, timeout=5000)

    if command == b'snap':
        image = sensor.snapshot()

        marker_dx = []
        marker_dy = []
        marker_response = []

        for o in originList:
            o_x = o[1]
            o_y = o[0]
            x = o_x - 32
            y = o_y - 32
            displacement =  extra_fb.find_displacement(img, roi = (x, y, BLOCK_W, BLOCK_H), template_roi = (x, y, BLOCK_W, BLOCK_H))

            d_x = int(displacement.x_translation() * 5) / 5.0
            d_y = int(displacement.y_translation() * 5) / 5.0

            # marker_origin.append(o)
        list_final = marker_dx + marker_dy + marker_response
        # list_final = [0,0]
        image1 = image.compress()
        usb_vcp.send(ustruct.pack('<L', image1.size()))
        command_d = usb_vcp.recv(4, timeout=5000)

        if command_d == b'list':
            s = ustruct.pack('54d', *list_final)

It doesn’t communicate as it encounters the find_displacemnt function inside the code.
Could you help to find what the bug is here? If I comment out the part that calculates optical flow, everything works fine.

The python script that use on my computer is as follows:

import io
import struct
import sys

import numpy as np
import serial
from PIL import Image as PILImage
import matplotlib.pyplot as plt

import cv2

class OpenMVCamRead:

    def __init__(self, device='COM3'):
        self.port = serial.Serial(device, baudrate=115200,
                                  xonxoff=False, rtscts=False,
                                  timeout=None, dsrdtr=True)


        self.initArea = np.zeros((3,4), dtype=np.float64)
        self.init0d = np.zeros((3,4), dtype=np.float64)
        self.init45d = np.zeros((3,4), dtype=np.float64)
        self.init90d = np.zeros((3,4), dtype=np.float64)
        self.init135d = np.zeros((3,4), dtype=np.float64)
        self.initPlusCross = np.zeros((3,4), dtype = np.float64)

        self.folder_name = "dataset"

        self.originList = [(207, 282), (204, 194), (202, 106), (200, 19), (161, 238), (159, 151), (158, 64), (118, 283), (117, 195), (115, 108), (113, 21), (75, 240), (72, 152), (71, 66), (30, 197), (30, 285), (27, 109), (25, 21)]

    def readFromNicla(self, count):
        snap = 'snap'
        if sys.version_info[0] >= 3:
            snap = snap.encode('utf-8')


        size = struct.unpack('<L',[0]
        image_data =
        image = np.array(

        print("IMAGE DATA")

        list_ = 'list'
        if sys.version_info[0] >= 3:
            list_ = list_.encode('utf-8')


        list_ = struct.unpack('54d',

        # list_ = [1,1]

        return image, list_

    def plot_mesh(self, image):
        block_h = 64
        block_w = 64
        val = 32

        for o in self.originList:
          o_x = o[1]
          o_y = o[0]
          image =, (o_x,o_y), 1, (0,0,0), 1)
          x = o_x - val
          y = o_y - val
          image = cv2.rectangle(image, (x, y), (x+block_w, y+block_h), (255,0,0), 1)

        return image

    def interpretAndStore(self, image, list_, imgCount):

        if imgCount < 100:
            count = "0" + str(imgCount)
            count = str(imgCount)

        # store data from nicla
        marker_dx = (np.array(list_[0:18]))
        marker_dy = (np.array(list_[18:36]))
        marker_confidence = (np.array(list_[36:54]))

        # store original img
        name_img = "..\\" + self.folder_name + "\\" + count + "img.png"
        cv2.imwrite(name_img, cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

        # review trhe field
        figure1, axis1 = plt.subplots(1, 1)
        image_field = self.plot_mesh(image)
        plt.savefig('..\\'+ self.folder_name +'\\' + count + 'field.png')

        # find correspondence according to our algorithm
        gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        blur = cv2.GaussianBlur(gray, (0,0), sigmaX=33, sigmaY=33)
        divide = cv2.divide(gray, blur, scale=255)
        # cv2.imshow(divide)
        ret, frame = cv2.threshold(divide, 120, 255, cv2.THRESH_BINARY)
        # cv2.imshow(frame)

        kernel1 = np.ones((3,3), np.uint8)
        kernel2 = np.ones((2,2), np.uint8)

        iter1 = 1
        iter2 = 1
        img_erosion = cv2.erode(frame, kernel1, iterations=iter1)
        img_dilation = cv2.dilate(img_erosion, kernel2, iterations=iter2)
        # cv2.imshow(img_dilation)
        me_marker_dx = []
        me_marker_dy = []

        for o in self.originList:
            ind = np.array(np.where(img_dilation[o[0]-32:o[0]+32, o[1]-32:o[1]+32]==0))
            dy = (np.mean(o[0]-32 + ind[0])) - o[0]
            dx = (np.mean(o[1]-32 + ind[1])) - o[1]

        # dump displacement according to nicla and mine
        print("\nnicla displacement x:\n", marker_dx)
        print("\nnicla displacement y:\n", marker_dy)
        print("\nme displacement x:\n", me_marker_dx)
        print("\nme displacement x:\n", me_marker_dx)

        # plot nicla 

        # plot me

if __name__ == '__main__':
    instance = OpenMVCamRead()

    folder_name = input("Write the folder name")
    instance.folder_name = folder_name

    imgCount = 1
    while (True):
        image, list_ = instance.readFromNicla(imgCount)
        instance.interpretAndStore(image, list_, imgCount)

Hi, please post the specific line that has the error.

We will not debug your code when you just post two scripts and then say they don’t work.

We have been getting bombarded lately with people posting scripts in whole asking for us to debug their entire script.

It is expected that you ask us a specific question about something that is not working as it should in our firmware or MicroPython. If you haven’t taken the time to narrow down the issue we will not do that for you.

sorry, I will rephrase it more specifically
as I had mentioned, if I comment out the lines specific to optical flow in, nicla vision can send images and empty lists to host computer.
I am referring to the lines below:

displacement =  extra_fb.find_displacement(img, roi = (x, y, BLOCK_W, BLOCK_H), template_roi = (x, y, BLOCK_W, BLOCK_H))
d_x = int(displacement.x_translation() * 5) / 5.0
d_y = int(displacement.y_translation() * 5) / 5.0

Its just three lines when added to the that doesn’t stops the communication. Nicla Vision doesn’t execute anything after it encounters the line displacement = extra_fb.find_displacement(img, roi = (x, y, BLOCK_W, BLOCK_H), template_roi = (x, y, BLOCK_W, BLOCK_H))

The function call requires a rather large amount of RAM to execute successfully and may not be appropriate on the nicla. The code might have some out of memory issues. We have not edited that function in a while.

Find displacement needs to allocate two FFTs of each image which is 4 bytes per pixel when using floats so it requires 8x the size of an image in RAM. This is likely causing an out of memory exception unless your image size is very small.

The example script for included with openmv IDE works well with Nicla when I execute it through IDE. When I try to run the same script with additions of usb_vcp, it doesn’t. Is this still the issue of memory? I am unsure I understand why the script works on IDE and not on Nicla alone. Could you please explain what special functionality is IDE using such that the example script is running

Does your script run in the IDE ? Remove the VCP code and run it in the IDE and you will see if it’s running out of memory. Note the example code uses a very small image (64x64 pixels).

sensor.set_framesize(sensor.B64X64) # Set frame size to 64x64... (or 64x32)...

Yes, my script for QVGA does run on IDE.
i tried running the example script made for 64x64 pixel as with usb_vcp code lines, but it still doesn’t work

I see you are using the ROI feature so this likely saves the RAM.

Can you use the UART version of the rcp interface to enable debugging and use a USB to serial converter? I can’t debug without an error message. However, the function is definitely throwing and error.

Hey, sorry to ask this dumb question, but just to confirm, why can’t we use the usb_vcp in rpc to debug?
I believe to access UART pins I would have to solder the nicla vision on header pins, and I have already attached the sensor to the project model.

You can but since it’s the data channel it’s hard to break debug info out.

I don’t know why the method works when plugged into the IDE and then fails when not. This tells me something else is going on. There should be no difference in behavior.