Interface Library Discussion

kwagyeman · January 20, 2020, 9:26pm

Hi all,

We want to make the OpenMV Cam more friendly to interface to other processors and we will now support Interface libraries for the camera to the Arduino and RaspberryPi. The plan is to offer interface libraries for:

The OpenMV Cam as slave processor over Aysnc Serial (UART, RX, TX) to the Arduino
The OpenMV Cam as an I2C Slave processor (SCL, SDA) to the Arduino
The OpenMV Cam as an SPI Slave processor (SS, SCK, MISO, MOSI) to the Arduino
The OpenMV Cam as slave processor over VCP Serial (USB) to the RapsberryPi (or any linux processor)

I have looked at porting the firmata library but it’s a rather large amount of code and functionality to emulate. For this library I’m trying a light touch.

The interface library modules will be written in python and then baked into the OpenMV Cam firmware as frozen modules. We want to do it this way so that you have access to the python code easily if you want to modify our library… but, otherwise, if you are using it unmodified it’s already onboard.

We will also supply the other half of the code running on the Arduino/Pi that will talk to the camera.

The plan is to make these 4 libraries just about control of the camera and not about running any particular functionality. This is so that others can build off of them.

Anyway, the point of this forum thread is to gather feedback on what features I should put in these interface libraries. Thoughts are welcome.

Here’s my plan for the library setup:

First there’s an init method which will create the library and the underlying hardware channel for you.

Second, you will register callbacks on the OpenMV Cam that can be run on receiving a command from the other device. Callbacks will execute on request from the host processor and return a response.

And that’s it. There will be no while(true): loop anymore in your code. Instead you just create the interface library device, register some callbacks with particular IDs that your host program should know, execute a method to enter an event loop, and then when the host program executes a command with the right ID the callback is run and you get data back. This gives you freedom to do whatever you want. The interface libraries primarily will deal with moving bytes across a link.

…

Next, for example purposes, I will have examples showing off the client (OpenMV Cam) and host side of the code for:

Face Detection
Person Detection
Qrcode Detection
AprilTag Detection
DataMatrix Detection
Barcode Detection
Color Tracking
Image Capture

As an example, on the Arduino, the data structure being moved will be described in C and then on the OpenMV Cam the data structure will be serialized using the struct module to match the Arduino structure. On the Pi, the struct will just be un-packed in python.

I feel that setting up things in this way is very generic and allows folks to do anything they need and makes the I/O part easy. That said, I understand it requires some understanding of data-structures. However, I think having 8 examples should give folks enough tools to copy from to do what they need.

Note that this library code is designed to make the OpenMV Cam a slave device on-purpose since that’s the best way for other processors to handle the data rate from the camera.

Thoughts?

… Regarding the VPC stuff. As time allows, I will work on features in the IDE to still be functional if the VPC port is used. Also, MicroPython technically enabled having two VPC ports now in the current firmware releases to it may be possible to have another VPC port that the IDE doesn’t have to share.

jcp13 · January 21, 2020, 3:44pm

This is awesome!! Thank you for working on this!

Questions:

Will the libraries also have APIs for the Flir Lepton IR camera?
Will you also support the Jetson Nano?

Cheers!

Nezra · January 21, 2020, 4:35pm

Just so I don’t go off on a tangent and i understand correctly.

The OpenMV cam is a ‘slave’ in this use case. it’s basically doing it’s own thing, but sends something to the host when queried? i say ‘slave’ because it’s not really designed to act as a slave; whereas this sounds more like a co-processor scenario.
The host is basically sending a command to the openmv cam, it does it’s operation, and returns the result to the host.

What ‘result’ is being transferred? Is it strictly numbers like the “Grove AI Hat for Edge Computing” or are we talking full blown images like a webcam?

If i understand the meaning, it’s much like the Wifi-Nina scenario with the Arduino and ESP32. The Nina core has a bunch of pre-programmed commands and an interpreter built in to act as a co-processor. It does it’s own thing, and shuttles data back and forth to the host. from the host perspective, you just ask for a webpage. From the co-processor perspective, it gets the request for a web-page, handles all of the internet related things and communication, along with any other processing it needs to, and returns the data as a result to the host. the host gets an object, either readable data or byte streams as an object and the user is left to do with it what they will. It’s not really a slave, more of a co-processor.

If this is the case I’m curious as to what an example would be, since i’d assume on the host side you’d send a command to the openmv cam to say check for an AprilTag. On the Cam side, you have the interface library set up a call back in a loop doing something that would tell the user code there to return something when the interrupt is triggered.

Or, am I completely misunderstanding and the goal here is to simply create a ‘standardized’ communication protocol between the Openmv Cam and the host that will in essence just transmit commands to engage the functions in openmv already?

Another item; does the Uart, SPI and I2C strictly need to be Arduino or can it be done that way via the RPI as well; specifically for python. something akin to what the Grove AI Hat for edge computing should have been but isn’t.

A further question from here would comedown to threading. Since the Openmv Cam didn’t support threading, how do you envision the callback or interrupt taking place ‘slave’ side? if the camera isn’t using an Async communication protocol like USB or Uart, and is busy processing something when it receives a request; won’t this cause an issue without having the FIFO buffer to fall back on? i’m not talking about the host needing to be programmed correctly to do a handshake and wait for data, but instead the cam acting as a ‘slave’ missing any of that initial handshake unless the Cam is sitting in a blocked state waiting for the host.

Or have i completely misunderstood.

kwagyeman · January 22, 2020, 3:02am

The goal is to make the OpenMV Cam into a co-processor that just does something when requested and then returns a result. Here’s a simple example:

On the Arduino I want to read a QR code string, so, I’ll be able to attach an OpenMV Cam to the Arduino and use the interface library to do this.

On the Arduino I call a method to activate a callback on the OpenMV Cam with some ID value that both the OpenMV Cam and the Arduino have set the same in code.
The camera receives the command and uses the ID value to lookup a callback to run. That callback runs, does whatever it needs to do taking as much time as it needs, and then returns some byte array result.
The Arduino receives the byte array result and the writes that to like a C struct so that you can use the result in your code.

So, effectively, the Arduino is able to ask the OpenMV Cam to do image processing for it.

The interface library will block the Arduino while the OpenMV Cam is working. Similarly, the OpenMV Cam will not do any work until it gets a command from the Arduino.

Given that you can transfer generic byte structs this means you can transfer images, objects, whatever. Also, you can edit the code running on the OpenMV Cam to do more than just return the result of find_blobs(), etc. You can do all the filtering on the OpenMV Cam and then just return the final result you think is good.

…

Regarding the Arduino/Pi, I can make the I/O library for both.

As for how I plan to handle SPI and I2C… well, it’s tricky. It’s not impossible to get slave mode working with SPI and I2C on the OpenMV Cam, it’s just hard. But, I will write the required code to make this work so you don’t need to think about it. Essentially, the master device will query the OpenMV Cam trying to get a response and it will keep trying until the OpenMV Cam has gotten to a point where it’s able to transmit data.

For example, with I2C, until I execute the I2C send method and then the OpenMV Cam starts waiting for a master device to clock I2C data out of it the master device will get NCKs on the I2C bus. So, the solution is to make the library poll the camera on the I2C bus until the camera responds. Similarly, you have to do the same thing for SPI. The value add of the library is that I write all this code and hand-shaking stuff for you and get it working nicely.

kwagyeman · January 22, 2020, 3:03am

Hi Jcp13,

There’s no API for anything specific, if you want to use the FLIR Lepton you can just use the interface library with it.
I would need community support to support the jetson. I’m just one person.

doodah · January 22, 2020, 7:31pm

This will be great to have. Writing a robust and reliable communication interface always seemed pretty daunting to me, so I appreciate the effort that will go into it.

It would be really nice if one of the libraries could allow for a single Arduino to accommodate multiple OpenMV’s as slave processors. Early in this forum, I’ve inquired about daisy chaining multiple OpenMV’s over UART and and an I2C mux to allow for multiple OpenMV’s on the I2C bus. I’d be happy with whatever makes the most sense between SPI, UART, and I2C.

Most of my work is with finding circles, so if you can throw in an example using that would be awesome.

Thanks!

kwagyeman · January 23, 2020, 5:10am

I’ll allow you to set the I2C bus address of the OpenMV Cam for this. As for a finding circles example… if I have time.

doodah · January 23, 2020, 3:44pm

Excellent. I’m sure your time is limited, so I completely understand.

I have no sense of how long all this would take. Would the I2C portion of it be done by the time the H7 plus is in stock? My project involves multiple OpenMV’s at a higher resolution, so I’m on hold until those are available. If the I2C interface is available then as well, that would be sweet.

Thanks!

kwagyeman · January 23, 2020, 6:54pm

Yeah, maybe. I will try. I have a bunch of driver firmware development to finish right now followed by clearing out bugs on GitHub, and ide release, and then this new library on my to-do list.

doodah · January 24, 2020, 3:17am

Seems ambitious, but this whole project/business is pretty ambitious so I guess it comes with the territory. Thanks!

jcp13 · March 10, 2020, 7:56pm

Hi kwagyeman,

I can give it a shot to build something for the Jetson Nano. I will need some guidance from you if you don’t mind. I may be able to start in early summer (I have a number of projects on my plate at this time). Is it ok to reach out when it gets closer to summer?

kwagyeman · March 10, 2020, 8:06pm

Hi, I’m almost to working on this. I have some other things involving drivers to finish up first but I expect to begin in the last two weeks of March.

kwagyeman · April 8, 2020, 3:42am

Hi all,

I’m almost done with the interface library:

github.com

kwagyeman/openmv/blob/kwabena/interface_library/scripts/libraries/mutex.py

import pyb, micropython, array, uctypes
micropython.alloc_emergency_exception_buf(100)

class MutexException(OSError):
    pass

class Mutex:
    @micropython.asm_thumb
    def _acquire(r0, r1):       # Spinlock: wait on the semaphore. Return on success.
        label(LOOP)
        ldr(r0, [r1, 0])        # Wait for lock to be zero
        cmp(r0, 0)
        bne(LOOP)               # Another process has the lock: spin on it
        cpsid(0)                # OK, we have lock  at this instant disable interrupts
        ldr(r0, [r1, 0])        # and re-check in case an interrupt occurred
        cmp(r0, 0)
        itt(ne)                 # if someone got in first re-enable ints
        cpsie(0)                # and start polling again
        b(LOOP)
        mov(r0, 1)              # We have an exclusive access

This file has been truncated. show original

github.com

kwagyeman/openmv/blob/kwabena/interface_library/scripts/libraries/rpc.py

# This file is part of the OpenMV project.
#
# Copyright (c) 2013-2020 Ibrahim Abdelkader <iabdalkader@openmv.io>
# Copyright (c) 2013-2020 Kwabena W. Agyeman <kwagyeman@openmv.io>
#
# This work is licensed under the MIT license, see the file LICENSE for details.

import gc, network, omv, pyb, select, socket, stm, struct

class rpc:

    _COMMAND_HEADER_PACKET_MAGIC = 0x1209
    _COMMAND_DATA_PACKET_MAGIC = 0xABD1
    _RESULT_HEADER_PACKET_MAGIC = 0x9021
    _RESULT_DATA_PACKET_MAGIC = 0x1DBA

    @micropython.viper
    def __def_crc_16(self, data, size : int) -> int: # private
        d = ptr8(data)
        crc = 0xFFFF

This file has been truncated. show original

https://github.com/kwagyeman/openmv/blob/kwabena/interface_library/scripts/examples/34-Remote-Control/as_the_remote_device.py
https://github.com/kwagyeman/openmv/blob/kwabena/interface_library/scripts/examples/34-Remote-Control/as_the_controller_device.py

The above 4 files form the library. The RPC module and mutex need to be copied to your OpenMV Cam disk. They will be compiled into the firmware in the next release. The as the remote device script should run on your OpenMV Cam and the controller script should run on a MicroPython board controlling the OpenMV Cam.

So far, I’ve debugged UART control and that works perfectly. The protocol is self-healing and can handle being disconnected and then reconnected in the middle of operation without anything crashing. All transfers are CRC16 checked too.

…

For Arduino support we have someone working on an Arduino interface library which will be released to the Arduino package repo for everyone along with creating a RPi python library too. These libraries will appear soon after the we’ve released the library for the OpenMV Cam.

Anyway, you can demo the library with UART control between two OpenMV Cams right now. If you want to do that save the mutex.py, rpc.py on both OpenMV Cams and then the as_the_remote_device.py script as main.py on an OpenMV Cam you want to remote control. Connect the UARTs (P4->P5 and P5->P4 and GND) between the two cameras. Then finally connect to the master camera and run the as_the_controller_device.py script.

For debugging both scripts if you decide to edit things, I recommend opening both example scripts in OpenMV IDE, connecting to the slave device in the main window, and then using the Open Terminal feature to connect to REPL of the master device and send it the master script. This is a little advanced to setup but works nicely.

NOTE: Only the UART works right now, I will debug and fix any issues with SPI, I2C, CAN, and USB in the coming days.