ESP32 AND OPENMV

Hello!

i am trying to stream live video using openmv3 m7 and Esp32. so i need some suggestions that which interface will be better for communication to openmv and if is there any person have done the same thing can you guys let me know.

Use the RPC library. We have an Arduino implementation of the library. Just connect that to an OpenMV Cam with an RPC script running and you can move data then.

Hello! I have been trying to do a similar thing and am running into an issue setting up the streaming interface. I am hoping you can help. I am using an Adafruit HUZZAH32 - ESP32 Feather and an OpemMV H7 with the newest firmware. It looks like the SPI interface is the fastest available over RPC for connecting these two, so that is the interface I am trying to use. I have attached the test files I am using. Note that your forum does not consider .ino as a valid file extension, so I renamed the Arduino file to .ino.txt so I could upload it.

TestStreaming.ino is run on my ESP32.
If I set the variable on line 9: streamTest = false, then TestStreaming.ino works with popular_features_spi_test.py (see attached) on the OpenMV H7. That works correctly, so I know my SPI setup is correct.
If I set the variable on line 9: streamTest = true, then TestStreaming.ino works with image_transfer_jpg_streaming.py (see attached) on the OpenMV H7. Here is where I run into issues. Both programs run, and I do not get the debug output from my Arduino program of ā€œFailed to start streamingā€, so it appears that the RPC streaming connection is set up correctly. However, I never get the debug output from my Arduino program of ā€œReceived video bytes: ā€, so the callback function for receiving data is never called.

In image_transfer_jpg_streaming.py (see attached), I have tried uncommenting lines 36 and 37 and commenting out line 39 to test significantly reduced data (since I see you mention that the full streaming rate overwhelms most MCUs). Even when testing that way, I never see the debug output from my Arduino program of ā€œReceived video bytes: ā€, so the callback function for receiving data is never called.

I am assuming I have set up something incorrectly for the RPC streaming to work properly, but I do not know what I am missing/doing incorrectly. Could you please help?

Thank you!
TestStreaming.ino.txt (1.4 KB)
popular_features_spi_test.py (10.8 KB)
image_transfer_jpg_streaming.py (2.05 KB)

Stream mode doesnā€™t really work for SPI since the SPI bus is half duplex. Use a UART if you want to stream data async like.

For SPI modeā€¦ thereā€™s the cut through option thatā€™s in between stream mode and normal RPC calls where you are just sending data in the raw after an RPC call completes that you want to use. The image transfer (non-streaming) scripts show how to do this.

Thank you! I followed the non-streaming scripts and it is working now. I misinterpreted the H7 marketing statement ā€œA SPI bus that can run up to 80Mbs allowing you to easily stream image data off the system to either the LCD Shield, the WiFi Shield, or another microcontroller.ā€ to mean that it can use the RPC streaming protocol.

Two follow-up questions:

  1. When you say half-duplex, I read on the STM32H743 datasheet that it supports full-duplex, half-duplex, and simplex SPI modes. Is the half-duplex a fixed H7 cam configuration? I was just surprised to hear half-duplex since the wiring seems to be typical for a full-duplex setup.
  2. With the current setup, I am getting approximately 2FPS. The compressed JPEG image sizes are <10000 Bytes/80000 bits. Given that the SPI bus should be able to move up to 20Mbs or more (2 orders of magnitude higher for the frame rate), I am assuming the bottleneck is either with the RPC calls or how the images are captured/sent on the H7 camera. Even when trying the streaming mode, my debug statements on the camera seemed to suggest that the capture/send was the slower part. Is there a better way for me to capture/send on the H7 camera that would speed things up? I do not need high rates, but I would ideally like to be above 10FPS.

Thank you!

  1. SPI is full duplex, but, it would be exceptionally hard to send data in both directions at the same time since both streams share the same clock. So, TX and RX are not separated. Contrast to the UART where RX and TX are independent.

I suppose it wouldnā€™t be impossible to get full duplex working. But, it would require you to deal with bit and byte framing in softwareā€¦ so, the CPU load to decode the data would be quite high.

Anyway, I donā€™t plan to support streaming mode for SPI and I2C.

  1. Itā€™s the RPC call overhead + snapshot overhead. If you check bus usage youā€™ll see itā€™s idle most of the time.

Can you use the UART? Itā€™s just easier. If not, then youā€™ll want to use the cut through mode to move data. This is where an RPC call is used to transfer the file header and then the data is transferred in one giant SPI write. The image transfer script shows this mode off. Thereā€™s no CRC protection on the data but it is much faster to move the image. You should instantly see the performance increase.

Iā€™m not sure if my Arduino examples show this mode off however. You can see it in the OpenMV Cam RPC scripts that have the master/slave role for the OpenMV Cam.

The reason cut through mode is not shown is the at when you use it the camera WILL send the data at whatever clock speed you set with NO break. Unless your MCU uses SPI with DMA your microcontroller will fall over trying to receive the data. From what Iā€™ve seen of Arduino wrappers for hardware like a SPIā€¦ DMA is not used. So, I suspect your Microcontroller will have trouble handing the data. But, you can try.

(Typically most low effort bus wrappers have the processor reading data from the SPI bus one byte at a time. This is not fast enough to handle receiving data when itā€™s coming in at 40 Mb/s).

I have an ESP32 board, and would prefer to use Micropython to transfer images over UART.
The RPC library doesnā€™t seem to work on the ESP32 because it does not support Pyb, but only Machine.
Are there some quick modifications I could make to transfer images over UART? Iā€™ve tried with very limited successā€¦

Just add another master/slave class to the library for machine.

The RPC library allows you to create new class and is weā€™ll documented.

Yes, this requires you to write python code using object oriented features of python. However, if you read the RPC code you can do this in less than 20 lines of code.

Happy forum anniversary. I am not quite savvy enough to do the programming quite yetā€¦ so Iā€™ve switched to Arduino. However, when I try to load the RPC image_transfer_jpg_as_the_controller_device_example.ino code onto my ESP32, I get errors.

I2C:

Arduino: 1.8.15 (Windows Store 1.8.49.0) (Windows 10), Board: "DOIT ESP32 DEVKIT V1, 80MHz, 921600, None"

C:\GitLab\openmv-arduino-rpc\examples\image_transfer_jpg_as_the_controller_device_example\image_transfer_jpg_as_the_controller_device_example.ino: In function 'void loop()':

image_transfer_jpg_as_the_controller_device_example:112:53: error: invalid conversion from 'const void*' to 'void*' [-fpermissive]

                        &jpeg_size, sizeof(jpeg_size))) {

                                                     ^

In file included from C:\GitLab\openmv-arduino-rpc\examples\image_transfer_jpg_as_the_controller_device_example\image_transfer_jpg_as_the_controller_device_example.ino:11:0:

C:\Users\[USER]\Documents\Arduino\libraries\OpenMV_Arduino_RPC\src/openmvrpc.h:155:10: note:   initializing argument 2 of 'bool openmv::rpc_master::call(const __FlashStringHelper*, void*, size_t, void*, size_t, bool, long unsigned int, long unsigned int)'

     bool call(const __FlashStringHelper *name,

          ^

image_transfer_jpg_as_the_controller_device_example:128:43: error: 'O_WRITE' was not declared in this scope

         File jpg_file = SD.open(filename, O_WRITE | O_CREAT | O_TRUNC);

                                           ^

image_transfer_jpg_as_the_controller_device_example:128:53: error: 'O_CREAT' was not declared in this scope

         File jpg_file = SD.open(filename, O_WRITE | O_CREAT | O_TRUNC);

                                                     ^

image_transfer_jpg_as_the_controller_device_example:128:63: error: 'O_TRUNC' was not declared in this scope

         File jpg_file = SD.open(filename, O_WRITE | O_CREAT | O_TRUNC);

                                                               ^

image_transfer_jpg_as_the_controller_device_example:153:109: error: invalid conversion from 'char**' to 'void**' [-fpermissive]

                 if (interface.call_no_copy(F("jpeg_image_read"), &arg, sizeof(arg), &jpg_data, &jpg_data_len)) {

                                                                                                             ^

In file included from C:\GitLab\openmv-arduino-rpc\examples\image_transfer_jpg_as_the_controller_device_example\image_transfer_jpg_as_the_controller_device_example.ino:11:0:

C:\Users\[USER]\Documents\Arduino\libraries\OpenMV_Arduino_RPC\src/openmvrpc.h:134:10: note:   initializing argument 4 of 'bool openmv::rpc_master::call_no_copy(const __FlashStringHelper*, void*, size_t, void**, size_t*, long unsigned int, long unsigned int)'

     bool call_no_copy(const __FlashStringHelper *name,

          ^

image_transfer_jpg_as_the_controller_device_example:157:62: error: invalid conversion from 'char*' to 'const uint8_t* {aka const unsigned char*}' [-fpermissive]

                     if (jpg_file.write(jpg_data, jpg_data_len) == jpg_data_len) {

                                                              ^

In file included from C:\Users\[USER]\Documents\ArduinoData\packages\esp32\hardware\esp32\1.0.6\libraries\SD\src/SD.h:17:0,

                 from C:\GitLab\openmv-arduino-rpc\examples\image_transfer_jpg_as_the_controller_device_example\image_transfer_jpg_as_the_controller_device_example.ino:10:

C:\Users\[USER]\Documents\ArduinoData\packages\esp32\hardware\esp32\1.0.6\libraries\FS\src/FS.h:55:12: note:   initializing argument 1 of 'virtual size_t fs::File::write(const uint8_t*, size_t)'

     size_t write(const uint8_t *buf, size_t size) override;

            ^

Multiple libraries were found for "SD.h"

 Used: C:\Users\[USER]\Documents\ArduinoData\packages\esp32\hardware\esp32\1.0.6\libraries\SD

 Not used: C:\Program Files\WindowsApps\ArduinoLLC.ArduinoIDE_1.8.49.0_x86__mdqgnx93n4wtt\libraries\SD

exit status 1

invalid conversion from 'const void*' to 'void*' [-fpermissive]



This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.

Hardware UART (same error as Software UART):

Arduino: 1.8.15 (Windows Store 1.8.49.0) (Windows 10), Board: "DOIT ESP32 DEVKIT V1, 80MHz, 921600, None"

image_transfer_jpg_as_the_controller_device_example:60:9: error: 'rpc_hardware_serial_uart_master' in namespace 'openmv' does not name a type

 openmv::rpc_hardware_serial_uart_master interface(115200);

         ^

C:\GitLab\openmv-arduino-rpc\examples\image_transfer_jpg_as_the_controller_device_example\image_transfer_jpg_as_the_controller_device_example.ino: In function 'void setup()':

image_transfer_jpg_as_the_controller_device_example:81:5: error: 'interface' was not declared in this scope

     interface.begin();

     ^

C:\GitLab\openmv-arduino-rpc\examples\image_transfer_jpg_as_the_controller_device_example\image_transfer_jpg_as_the_controller_device_example.ino: In function 'void loop()':

image_transfer_jpg_as_the_controller_device_example:110:9: error: 'interface' was not declared in this scope

     if (interface.call(F("jpeg_image_snapshot"),

         ^

image_transfer_jpg_as_the_controller_device_example:128:43: error: 'O_WRITE' was not declared in this scope

         File jpg_file = SD.open(filename, O_WRITE | O_CREAT | O_TRUNC);

                                           ^

image_transfer_jpg_as_the_controller_device_example:128:53: error: 'O_CREAT' was not declared in this scope

         File jpg_file = SD.open(filename, O_WRITE | O_CREAT | O_TRUNC);

                                                     ^

image_transfer_jpg_as_the_controller_device_example:128:63: error: 'O_TRUNC' was not declared in this scope

         File jpg_file = SD.open(filename, O_WRITE | O_CREAT | O_TRUNC);

                                                               ^

image_transfer_jpg_as_the_controller_device_example:157:62: error: invalid conversion from 'char*' to 'const uint8_t* {aka const unsigned char*}' [-fpermissive]

                     if (jpg_file.write(jpg_data, jpg_data_len) == jpg_data_len) {

                                                              ^

In file included from C:\Users\[USER]\Documents\ArduinoData\packages\esp32\hardware\esp32\1.0.6\libraries\SD\src/SD.h:17:0,

                 from C:\GitLab\openmv-arduino-rpc\examples\image_transfer_jpg_as_the_controller_device_example\image_transfer_jpg_as_the_controller_device_example.ino:10:

C:\Users\[USER]\Documents\ArduinoData\packages\esp32\hardware\esp32\1.0.6\libraries\FS\src/FS.h:55:12: note:   initializing argument 1 of 'virtual size_t fs::File::write(const uint8_t*, size_t)'

     size_t write(const uint8_t *buf, size_t size) override;

            ^

Multiple libraries were found for "SD.h"

 Used: C:\Users\[USER]\Documents\ArduinoData\packages\esp32\hardware\esp32\1.0.6\libraries\SD

 Not used: C:\Program Files\WindowsApps\ArduinoLLC.ArduinoIDE_1.8.49.0_x86__mdqgnx93n4wtt\libraries\SD

exit status 1

'rpc_hardware_serial_uart_master' in namespace 'openmv' does not name a type



This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.

Hi, it appears that the ESP32 compiler requirements changed.

Those errors are happening because the code is designed for the Atmel based Arduinos. However, Iā€™ve compiled the library for ESP32 and it used to work. Itā€™s been a year now I guess.

Are you able to fix these issues yourself? Or no?

The first error seemed easier to fix.
I tried defining the jpeg_size as a const, but then I realized the size of the jpeg may change for each image?
I am not sure where to go from hereā€¦ @kwagyeman

When installing OpenMV Arduino RPC library on ESP32, dependencies are needed and when select Install All the error ā€œno protocolā€ appears in lower left.

When installing only the OpenMV Arduino RPC library, the examples do not compile. The following error messages appear:

Arduino: 1.8.16 (Windows 10), Board: ā€œESP32S2 Dev Module, Disabled, Disabled, Disabled, UART0, Disabled, Default 4MB with spiffs (1.2MB APP/1.5MB SPIFFS), 240MHz (WiFi), QIO, 80MHz, 4MB (32Mb), 921600, Noneā€

arduino_to_arduino_streaming_comms_as_the_remote_device:81:9: error: ā€˜rpc_software_serial_uart_slaveā€™ in namespace ā€˜openmvā€™ does not name a type

openmv::rpc_software_serial_uart_slave interface(2, 3, 19200);

     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

C:\Users\Alan\Documents\Arduino\libraries\OpenMV_Arduino_RPC\examples\arduino_to_arduino_streaming_comms_as_the_remote_device\arduino_to_arduino_streaming_comms_as_the_remote_device.ino: In function ā€˜void setup()ā€™:

arduino_to_arduino_streaming_comms_as_the_remote_device:110:5: error: ā€˜interfaceā€™ was not declared in this scope

 interface.register_callback(F("setup_streaming_cb"), setup_streaming_cb);

 ^~~~~~~~~

C:\Users\Alan\Documents\Arduino\libraries\OpenMV_Arduino_RPC\examples\arduino_to_arduino_streaming_comms_as_the_remote_device\arduino_to_arduino_streaming_comms_as_the_remote_device.ino: In function ā€˜void loop()ā€™:

arduino_to_arduino_streaming_comms_as_the_remote_device:126:9: error: ā€˜interfaceā€™ was not declared in this scope

     interface.loop();

     ^~~~~~~~~

arduino_to_arduino_streaming_comms_as_the_remote_device:128:27: error: ā€˜interfaceā€™ was not declared in this scope

     streaming_state = interface.stream_writer_setup() ? STREAMING : NOT_STREAMING;

                       ^~~~~~~~~

arduino_to_arduino_streaming_comms_as_the_remote_device:130:27: error: ā€˜interfaceā€™ was not declared in this scope

     streaming_state = interface.stream_writer_loop(stream_writer_cb) ? STREAMING : NOT_STREAMING;

                       ^~~~~~~~~

exit status 1

ā€˜rpc_software_serial_uart_slaveā€™ in namespace ā€˜openmvā€™ does not name a type

This report would have more information with
ā€œShow verbose output during compilationā€
option enabled in File ā†’ Preferences.

Hi, not all interfaces work on the EPS32. You need to use an interface that exists. E.g. software serial is not supported. You have to use hardware serial.

E.g. see the H file: openmv-arduino-rpc/openmvrpc.h at master Ā· openmv/openmv-arduino-rpc Ā· GitHub

Itā€™s really hard to support all the different interfaces on Arduino boards since they arenā€™t standardized.

1 Like

Hi @kwagyeman I used this library for hardware serial UART but when I compiled it for ESP32 it keeps giving me this error too

Arduino: 1.8.15 (Windows Store 1.8.49.0) (Windows 10), Board: "DOIT ESP32 DEVKIT V1, 80MHz, 921600, None"

image_transfer_jpg_as_the_controller_device_example:60:9: error: 'rpc_hardware_serial_uart_master' in namespace 'openmv' does not name a type

 openmv::rpc_hardware_serial_uart_master interface(115200);

         ^

C:\GitLab\openmv-arduino-rpc\examples\image_transfer_jpg_as_the_controller_device_example\image_transfer_jpg_as_the_controller_device_example.ino: In function 'void setup()':

image_transfer_jpg_as_the_controller_device_example:81:5: error: 'interface' was not declared in this scope

     interface.begin();

     ^

C:\GitLab\openmv-arduino-rpc\examples\image_transfer_jpg_as_the_controller_device_example\image_transfer_jpg_as_the_controller_device_example.ino: In function 'void loop()':

image_transfer_jpg_as_the_controller_device_example:110:9: error: 'interface' was not declared in this scope

     if (interface.call(F("jpeg_image_snapshot"),

         ^

image_transfer_jpg_as_the_controller_device_example:128:43: error: 'O_WRITE' was not declared in this scope

         File jpg_file = SD.open(filename, O_WRITE | O_CREAT | O_TRUNC);

                                           ^

image_transfer_jpg_as_the_controller_device_example:128:53: error: 'O_CREAT' was not declared in this scope

         File jpg_file = SD.open(filename, O_WRITE | O_CREAT | O_TRUNC);

                                                     ^

image_transfer_jpg_as_the_controller_device_example:128:63: error: 'O_TRUNC' was not declared in this scope

         File jpg_file = SD.open(filename, O_WRITE | O_CREAT | O_TRUNC);

                                                               ^

image_transfer_jpg_as_the_controller_device_example:157:62: error: invalid conversion from 'char*' to 'const uint8_t* {aka const unsigned char*}' [-fpermissive]

                     if (jpg_file.write(jpg_data, jpg_data_len) == jpg_data_len) {

                                                              ^

In file included from C:\Users\[USER]\Documents\ArduinoData\packages\esp32\hardware\esp32\1.0.6\libraries\SD\src/SD.h:17:0,

                 from C:\GitLab\openmv-arduino-rpc\examples\image_transfer_jpg_as_the_controller_device_example\image_transfer_jpg_as_the_controller_device_example.ino:10:

C:\Users\[USER]\Documents\ArduinoData\packages\esp32\hardware\esp32\1.0.6\libraries\FS\src/FS.h:55:12: note:   initializing argument 1 of 'virtual size_t fs::File::write(const uint8_t*, size_t)'

     size_t write(const uint8_t *buf, size_t size) override;

            ^

Multiple libraries were found for "SD.h"

 Used: C:\Users\[USER]\Documents\ArduinoData\packages\esp32\hardware\esp32\1.0.6\libraries\SD

 Not used: C:\Program Files\WindowsApps\ArduinoLLC.ArduinoIDE_1.8.49.0_x86__mdqgnx93n4wtt\libraries\SD

exit status 1

'rpc_hardware_serial_uart_master' in namespace 'openmv' does not name a type

Hi, you have to change the interface name to match whatever the ESP32 has. I fixed the compile errors in the library.

It may be called hardware serial uart master 1 or something like that.

See: openmv-arduino-rpc/openmvrpc.h at master Ā· openmv/openmv-arduino-rpc Ā· GitHub

Our code basically tries to bind interfaces to everything the MCU has. However, it can be tricky to figure out which interface is there.

Do you mean I should change it within src file or the example? cos honestly Iā€™m kinda a newbie in this
Do you mean I should edit this? And what should I change it to?

#define RPC_HARDWARE_SERIAL_UART_MASTER(name, port) \
class rpc_hardware_serial##name##_uart_master : public rpc_master \
{ \
public: \
    rpc_hardware_serial##name##_uart_master(unsigned long baudrate=115200) : rpc_master(), __baudrate(baudrate) {} \
    ~rpc_hardware_serial##name##_uart_master() {} \
    virtual void _flush() override; \
    virtual bool get_bytes(uint8_t *buff, size_t size, unsigned long timeout) override; \
    virtual bool put_bytes(uint8_t *data, size_t size, unsigned long timeout) override; \
    virtual void begin() override { port.begin(__baudrate); } \
    virtual void end() override { port.end(); } \
private: \
    unsigned long __baudrate; \
    rpc_hardware_serial##name##_uart_master(const rpc_hardware_serial##name##_uart_master &); \
};

No, you have to determined what the serial port is named.

Arduino has like hardware Serial1/2/3 etc.

So, it may be called openmv::rpc_hardware_serial_uart_master1 or openmv::rpc_hardware_serial_uart_master2 or etc.

Esp32 has 3 serial portsā€¦uart0,uart1,uart2.
And from the class declaration:
rpc_hardware_serial##name##_uart_master
Is the instance not supposed to be
openmv::rpc_hardware_serial1_uart_master
or

openmv::rpc_hardware_serial2_uart_master

Not

openmv::rpc_hardware_serial_uart_master1 or 2

Yes, you are right.

Hi,
I have the same issue and errors as Petchu65.
fixing it to openmv::rpc_hardware_serial_uart_master interface(115200), openmv::rpc_hardware_serial1_uart_master interface(115200), and openmv::rpc_hardware_serial2_uart_master interface(115200) donā€™t work and produce the same errors.
Can anybody provide any suggestions to solve these errors and communicate between ESP32 and OpenMV (using UART)?
Also, why should I use the RPC library to send compressed JPEG images to the ESP32 (for live-streaming in an online server), instead of uart.write(sensor.snapshot().compress(90)) in a loop?
Thank you!

Error:

sketch_dec25a:71:9: error: 'rpc_hardware_serial_uart_master' in namespace 'openmv' does not name a type
 openmv::rpc_hardware_serial_uart_master interface(115200);
         ^