GENX320 - streaming raw event data over USB

OpenMV Boards
1

Hey everyone,

I’m using the OpenMV H7 Plus with the GENX320 event camera. I’m using scripts slated for the next firmware release in Github to stream the raw event data (genx320_event_mode_high_speed.py), but I’m having a hard time actually streaming the raw data over USB. I need to use the raw event data, as I’m trying to feed it through an event-based object detection model (RVT from the folks over at University of Zurich), and thus can’t process the events onboard. I tried making my own python script for interpreting the event data, but I often get broken streams, where only parts of events are streamed back over USB. The Micropython script I’m using to stream back to my laptop is linked below:

import csi
import time
# https://micropython-ulab.readthedocs.io/en/latest/index.html
from ulab import numpy as np

# Stores camera events
# Shape: (EVT_res, 6) where EVT_res is the event resolution
# EVT_res: must be a power of two between 1024 and 65536.
# Columns:
#   [0]  Event type
#   [1]  Seconds timestamp
#   [2]  Milliseconds timestamp
#   [3]  Microseconds timestamp
#   [4]  X coordinate 0 to 319 for GENX320
#   [5]  Y coordinate 0 to 319 for GENX320
events = np.zeros((2048, 6), dtype=np.uint16)

# Initialize the sensor.
csi0 = csi.CSI(cid=csi.GENX320)
csi0.reset()
csi0.ioctl(csi.IOCTL_GENX320_SET_MODE, csi.GENX320_MODE_EVENT, events.shape[0])

clock = time.clock()
t = time.ticks_us()

i = 0
while True:
    clock.tick()
    t1 = time.ticks_us()
    diff = time.ticks_diff(t1, t)
    t = t1
    i += 1

    # Reads up to 2048 events from the camera.
    # Returns the number of valid events (0-2048) or a negative error code.
    # Note that old events in the buffer are not cleared to save CPU time.
    event_count = csi0.ioctl(csi.IOCTL_GENX320_READ_EVENTS, events)
    new_events = events[:event_count]

    # Sub-sample the event rate output to not impact performance of event
    # data processing. The overhead of sending stats outputs to the IDE can
    # become significant at high event rates.
    if not i % 10:
        #print(event_count)
        for event in new_events:
            print(list(event))

My question is: are there any plans for extending pyopenmv.py to handle raw event data? If not, are there any scripts you have for reliably transferring raw event data over USB?

2

We are aware of this issue and have a new prototcol in the works which will resolve the issue: common: OpenMV Protocol V2 Implementation. by iabdalkader · Pull Request #2824 · openmv/openmv · GitHub

We are working on getting a release done now, once that is finished we will be updating the protocol over the next month in the firmware and IDE. The new protocol supports customs streams which can send the data without dropping it. This will allow data transfer to the PC over USB easily at high speeds.

In the mean-time, the H7 Plus is not suitable at all for sending the event data off camera. it only has 12 Mb/s USB. This is far below the event rate of 48MB/s. The RT1062 is much closer to being able to send the event data by printing in. However, for the best RAW transfer speed using the current firmware you have two options:

  1. Use the RT1062 with the PoE Shield and ethernet. You can then transmit the NDarray as a byte structure via UDP sockets at close to 100Mb/s. This is by far the easiest and most flexible way to stream the data to a PC.
  2. The current USB debug logic is very good at sending the frame buffer as a large transfer quickly at 480 Mb/s. However, it’s designed to sample images from the camera, not capture all data with perfect fidelity. However, I can make a hacked version of the firmware that transfers the captured image into the jpeg buffer for you per frame capture. This will let you transfer very large event captures with some reliability without data loss.

I’d recommend the Ethernet approach, though, as it doesn’t require anything custom.