Thanks for the nice products. What I am using is the Genx320 event camera with RT1062 board. From the IDE, I can set the frequency to 200FPS, and I saw the streaming frequency for USB is about 18FPS from the downright corner of IDE. When I run pyopenmv_fb.py in my linux PC, the receiving frequency if also 18FPS which should be correct.
However, this is contrary to our purpose of using the event camera: we would rather use the high refresh rate feature of the event frame. I know this is limited by the USB tranfer rate (12Mb for full speed USB?), but I noticed the USB for RT1062 also support high speed (480Mb). Do you have any idea how to realize the high frequency tranfer? Thanks.
There’s a benchmark mode in pyopenmv_fb.py that could help with estimating the maximum USB throughput. Try something like:
python pyopenmv_fb.py --bench --poll 10
Keep in mind that none of these cameras are designed to just stream video over USB as fast as possible, it’s only there to help with development.
1 Like
Another issue is the JPEG compression. The RT1062 can’t compress images fast enough to send to the PC.
Can you do the processing on the RT1062? It’s not meant to be a frame grabber. If this is all you want the platform for, there’s a way to stream RAW uncompressed video over USB back to the PC. However, this will still be slower than the frame rate if you process the frames onboard the system.
1 Like
Can you please talk more about the way to stream RAW uncompressed video over USB to the PC? We need to do more complex operations for event frame, I guess the RT1062 is not enough for processing.
Hi, attached is a firmware for the RT1062 with RAW mode output enabled.
firmware.zip (1.4 MB)
I’ve enabled RAW mode output and removed the frame rate limit of 20 Hz.
I’ve also attached an updated version of the pyopenmv_fb script with an updated benchmark script to work for the GENX320.
pyopenmv.zip (4.1 KB)
You should get 90 FPS back to the PC. There are things that can be done to go faster… but they get into the single-purpose system optimization realm.
The diff of changes are below if you are comfortable editing the firmware and want to play with it yourself if you need to push it beyond 90 FPS. The USB interface could do up to 30 MB/s, but, it’s not easy to hit this.
diff --git a/src/omv/boards/OPENMV_RT1060/omv_boardconfig.h b/src/omv/boards/OPENMV_RT1060/omv_boardconfig.h
index ceade308..bf2a5a34 100644
--- a/src/omv/boards/OPENMV_RT1060/omv_boardconfig.h
+++ b/src/omv/boards/OPENMV_RT1060/omv_boardconfig.h
@@ -73,6 +73,11 @@
#define OMV_USB_PHY_TXCAL45DP (0x06U)
#define OMV_USB_PHY_TXCAL45DM (0x06U)
+// Enable RAW preview.
+#define OMV_RAW_PREVIEW_ENABLE (1)
+#define OMV_RAW_PREVIEW_WIDTH (512)
+#define OMV_RAW_PREVIEW_HEIGHT (320)
+
// Linker script constants (see the linker script template mimxrt10xx.ld.S).
// Note: fb_alloc is a stack-based, dynamically allocated memory on FB.
// The maximum available fb_alloc memory = FB_ALLOC_SIZE + FB_SIZE - (w*h*bpp).
diff --git a/src/omv/common/usbdbg.c b/src/omv/common/usbdbg.c
index bd8763c4..38f0fe92 100644
--- a/src/omv/common/usbdbg.c
+++ b/src/omv/common/usbdbg.c
@@ -233,7 +233,6 @@ void usbdbg_data_in(uint32_t size, usbdbg_write_callback_t write_callback) {
uint8_t byte_buffer[size];
memset(byte_buffer, 0, size);
uint32_t *buffer = (uint32_t *) byte_buffer;
- static uint32_t last_update_ms = 0;
// Set script running flag
if (script_running) {
@@ -247,8 +246,7 @@ void usbdbg_data_in(uint32_t size, usbdbg_write_callback_t write_callback) {
}
// Limit the frames sent over USB to 20Hz.
- if (ticks_diff_ms(last_update_ms) > 50 &&
- mutex_try_lock_alternate(&JPEG_FB()->lock, MUTEX_TID_IDE)) {
+ if (mutex_try_lock_alternate(&JPEG_FB()->lock, MUTEX_TID_IDE)) {
// If header size == 0 frame is not ready
if (JPEG_FB()->size == 0) {
// unlock FB
@@ -261,7 +259,6 @@ void usbdbg_data_in(uint32_t size, usbdbg_write_callback_t write_callback) {
buffer[1] = JPEG_FB()->w;
buffer[2] = JPEG_FB()->h;
buffer[3] = JPEG_FB()->size;
- last_update_ms = mp_hal_ticks_ms();
}
}
For even faster speeds you can increase the USB buffers from 512 bytes to 4096 bytes to max out the USB transfers.
firmware.zip (1.4 MB)
The updated firmware is attached. This changed required reducing the heap available on the board.
-#define OMV_GC_BLOCK1_SIZE (288K)
+#define OMV_GC_BLOCK1_SIZE (270K)
And in lib/micropython
diff --git a/shared/tinyusb/tusb_config.h b/shared/tinyusb/tusb_config.h
index 0cc5ef039..b51712722 100644
--- a/shared/tinyusb/tusb_config.h
+++ b/shared/tinyusb/tusb_config.h
@@ -75,6 +75,11 @@
#define CFG_TUD_MSC (0)
#endif
+ // CDC Endpoint transfer buffer size, more is faster
+ #define CFG_TUD_CDC_EP_BUFSIZE (4096)
+ #define CFG_TUD_CDC_RX_BUFSIZE (4096)
+ #define CFG_TUD_CDC_TX_BUFSIZE (4096)
+
// CDC Configuration
#if CFG_TUD_CDC
#ifndef CFG_TUD_CDC_RX_BUFSIZE
Any further speed increases would require substantial modifications to how the triple buffered frame buffer works and the IDE image buffer. Right now the system copies each frame to be sent to a buffer the IDE reads async which allows the python script running to continue operation without waiting on the IDE to read a frame.
1 Like
Note to the above. That was me running the benchmark script. Performance is different on the PC when viewing:
Note the --bench being removed.
The RT1062 is still running at 150 FPS onboard versus the PC viz which seems to be the limiter.
Thanks for your reply! This is amazing, now I can get 100FPS from USB, that’s enough for my project!
1 Like
Hi,
In this case, the board is still running at 150fps, but the pc only gets 52.08FPS. Does this mean there are some dropped frames? In the bench mode, the usb can handle 13.02MB/S, and it drops to 5.09MB/s in this case. May I know what limit the speed?
Bench mode is just the RAW transfer of the same frame over and over again to the PC. There’s no new frames. So, it just tests how much data the system can move over USB really.
As for the board being at 150 fps but the PC getting 50 FPS, the way the system works is that we have a triple-buffered frame buffer. Two back buffers receive frames all the time. Then there’s a front buffer the CPU is working off of to process a frame. Finally, when its done processing a frame it can copy the frame and put it into a buffer that the PC will then read async to pull video from the camera.
The 50 FPS is the speed at which the MCU is able to copy a new frame into the buffer the PC is using and then the PC pulling that frame while the camera is actively capturing new frames.
As mentioned above, I think the visualization on the PC side is the limiter here.