Triggered video recording with output frame grab TTLs

I am attempting to record video using my OpenMV camera that can later be used for training in ML contexts. I would like to be able to trigger video recording start using a TTL input to the OpenMV camera from an external application (Matlab). I would also like to output TTLs from OpenMV to an external application (again, Matlab) at the time of each frame grab. Based on the examples I found online and the topics in the forum, I created the following code to do this:

import pyb
import sensor
import image
import time
import mjpeg
import machine
import os
from pyb import Pin, ExtInt

sensor.reset()  # Reset and initialize the sensor.
sensor.set_pixformat(sensor.GRAYSCALE)  # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA)  # Set frame size to QVGA (320x240)
sensor.set_windowing((121,101,100,80)) # Set windowing
sensor.skip_frames(time=2000)  # Wait for settings take effect.

led = machine.LED("LED_RED")

# Prepare the voltage output
pin0 = Pin('P0', Pin.OUT_PP)

# Set external trigger
trigger = 0
def triggerline (line):
    global trigger
    trigger = 1
ext = ExtInt(Pin('P7'), ExtInt.IRQ_RISING, Pin.PULL_DOWN, triggerline)
ext.enable()

# Set trigger pin
pin7 = Pin('P7', Pin.IN)

# led.on()
m = mjpeg.Mjpeg("openMVvideo.mjpeg")

clock = time.clock()  # Create a clock object to track the FPS.
while(True):
    if (trigger == 1):
        sensor.skip_frames(time=1000) # Wait for the trigger to reset
        trigger = 0
        while(trigger == 0):
            clock.tick()
            led.on()
            pin0.high()
            m.write(sensor.snapshot())
            time.sleep(1) # Included to slow pulses/flashes and make them visible
            pin0.low()
            led.off()
            print(clock.fps())
        break

m.close()
# led.off()

raise (Exception("Please reset the camera to see the new file."))

This code successfully starts recording following an external TTL and then stops recordings following a second TTL. However, I am not receiving an output TTL from P0 for some reason and the LED is not blinking. I would appreciate any suggestions on how to fix this problem.

Also, is there a more efficient way to record the video? As it is, the video is recorded in mjpeg format to the SD card on the OpenMV camera. The camera has to be reset to see the video file. Is there a way to automate the reset process or include it in the code?

Also, transfer of this file off the SD card to the computer is very slow. Can the video be directly written to the computer?

Also, I’m having a hard time working with the mjpeg format file. I see that it can be converted to mp4 in the IDE, but is there a way to automate this process or write to mp4 during the original recording? I’m also finding that following conversion to mp4, the video file is not the proper length.

I would greatly appreciate any advice people can provide. I attempted to look through the github examples and the forum to find answers to these issues. If I missed the answers, I apologize. Thank you!

~Nick

I was able to fix the problem with the lack of blinking LED (see updated code below) and the lack of a TTL from P0 (bad connector).

import pyb
import sensor
import image
import time
import mjpeg
import machine
import os
from pyb import Pin, ExtInt

sensor.reset()  # Reset and initialize the sensor.
sensor.set_pixformat(sensor.GRAYSCALE)  # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA)  # Set frame size to QVGA (320x240)
sensor.set_windowing((121,101,100,80)) # Set windowing
sensor.skip_frames(time=2000)  # Wait for settings take effect.

led = machine.LED("LED_RED")

# Prepare the voltage output
pin0 = Pin('P0', Pin.OUT_PP)

# Set external trigger
trigger = 0
def triggerline (line):
    global trigger
    trigger = 1
ext = ExtInt(Pin('P7'), ExtInt.IRQ_RISING, Pin.PULL_DOWN, triggerline)
ext.enable()

# Set trigger pin
pin7 = Pin('P7', Pin.IN)

# led.on()
m = mjpeg.Mjpeg("openMVvideo.mjpeg")

clock = time.clock()  # Create a clock object to track the FPS.
while(True):
    if (trigger == 1):
        sensor.skip_frames(time=1000) # Wait for the trigger to reset
        trigger = 0
        while(trigger == 0):
            clock.tick()
            led.on()
            pin0.high()
            m.write(sensor.snapshot())
            time.sleep(1) # Included to slow pulses/flashes to make them visible
            pin0.low()
            led.off()
            time.sleep(1) # Included to slow pulses/flashes to make them visible
            print(clock.fps())
        break

m.close()
# led.off()

raise (Exception("Please reset the camera to see the new file."))

However, I would still appreciate any suggestions about the questions I had above about how to record this video more efficiently and how to handle the resulting files. Thank you!

~Nick

Hi, sorry for not responding yesterday. Didn’t see this forum post.

Yeah, so, OpenMV IDE can just record the frame buffer directly. There’s no need to do this on the camera. However, I guess it won’t be triggered then. Either way, once you have the video OpenMV IDE can split this into JPG files using FFMPEG when converting the video if you specify the output is a %d.jpeg format. As for MP4, we don’t have H.264 on the current versions of the OpenMV Cam, historically, no Microcontrollers have had the hardware to encode this video format on them. That said, this will be changing soon. But, as of 2024 this is not a feature any mainstream ARM MCU has. We write the data via MJPEG as this is possible with software/hardware JPEG encoding support.

Anyway, as for making it automated and more efficient. You need to write code on the desktop.

Use this: openmv/tools/pyopenmv.py at master · openmv/openmv (github.com)

You can control the camera and pull frames from it using this. So, it’s possible then to send a script that detects the event, and then you can pull text/frames from the camera and send to the desktop where you can save the data.

Note that the RT1062 has USB-HS. So, you can transfer files much faster to the PC using it. Otherwise, if you still need to write to the disk just remove the uSD card and plug into the PC.

Hi @kwagyeman!

There’s no need to apologize! I really appreciate your help! Thank you for sharing that information and guidance on how to improve the workflow. I’ll let you know if I have any questions.

Unfortunately, I’ve run into another issue with this code related to starting and stopping recording using a TTL. (I’ll keep the question in this thread since it pertains the other overall purpose.) This process had been working properly, but then I changed some simple parts of the code and the wiring and then it stopped functioning properly. Here is the updated code:

import pyb
import sensor
import image
import time
import mjpeg
import machine
import os
from pyb import Pin, ExtInt

sensor.reset()  # Reset and initialize the sensor.
sensor.set_pixformat(sensor.GRAYSCALE)  # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA)  # Set frame size to QVGA (320x240)
sensor.set_windowing((121,101,100,80)) # Set windowing
sensor.skip_frames(time=2000)  # Wait for settings take effect.

# Use triggered mode for frame capture to ensure all pixels are recorded simultaneously
sensor.ioctl(sensor.IOCTL_SET_TRIGGERED_MODE, True)

led = machine.LED("LED_RED")

# Prepare the voltage output
pin0 = Pin('P0', Pin.OUT_PP)

# Set external trigger
trigger = 0
def triggerline (line):
    global trigger
    trigger = 1
ext = ExtInt(Pin('P7'), ExtInt.IRQ_RISING, Pin.PULL_DOWN, triggerline)
ext.enable()

# Set trigger pin
pin7 = Pin('P7', Pin.IN)

# led.on()

m = mjpeg.Mjpeg("openMVvideoFakeSubject.mjpeg") # <----UPDATE THIS FILE NAME!!!! -----

print(trigger)

clock = time.clock()  # Create a clock object to track the FPS.
while(True):
    print(trigger)
    time.sleep(1)
    print(trigger)
    if (trigger == 1):
        sensor.skip_frames(time=1000) # Wait for the trigger to reset
        trigger = 0
        while(trigger == 0):
            clock.tick()
            led.on()
            pin0.high()
            m.write(sensor.snapshot())
            # time.sleep(1) # Included to slow pulses/flashes to make them visible
            pin0.low()
            led.off()
            # time.sleep(1) # Included to slow pulses/flashes to make them visible
            # print(clock.fps())
        break

m.close()
# led.off()

raise (Exception("Please reset the camera to see the new file."))

The code should run until it waits for the value of trigger to change from 0 to 1 when an external TTL is input to pin 7. Then it should start recording until another TTL comes into pin 7. With the versions of the code in my previous posts, this worked just fine.

Now, the trigger mysteriously changes to 1 after first entering the main while(True) loop. The terminal output is:

0
0
1

I reversed the code changes and disconnected all pin connections from the camera in case I made some mistake rewiring the system, but the behavior persists. When I comment out the following lines, the behavior goes away:

def triggerline (line):
    global trigger
    trigger = 1
ext = ExtInt(Pin('P7'), ExtInt.IRQ_RISING, Pin.PULL_DOWN, triggerline)
ext.enable()

# Set trigger pin
pin7 = Pin('P7', Pin.IN)

So, it must be related to pin 7 and the TTL input processing. However, I didn’t change any of this in my recent updates.

I would appreciate any suggestions on why this is happening. Thank you!

~Nick

You don’t need an external input for what you are trying to do. Just make the trigger pin an input and read its state. By making it an external interrupt you have a race condition when setting it low again.

You’d only need the external input if you had do something immediately when the trigger happened. However, image processing methods cannot happen in interrupt context time so the best latency you can get is just by doing a tight loop checking for a trigger.

Hi @kwagyeman ,

Thank you for that suggestion! However, I don’t think it has fixed the problem. Here is the updated code I am running:

import pyb
import sensor
import image
import time
import mjpeg
import machine
import os
from pyb import Pin, ExtInt

sensor.reset()  # Reset and initialize the sensor.
sensor.set_pixformat(sensor.GRAYSCALE)  # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA)  # Set frame size to QVGA (320x240)
sensor.set_windowing((121,101,100,80)) # Set windowing
sensor.skip_frames(time=2000)  # Wait for settings take effect.

# Use triggered mode for frame capture to ensure all pixels are recorded simultaneously
sensor.ioctl(sensor.IOCTL_SET_TRIGGERED_MODE, True)

led = machine.LED("LED_RED")

# Prepare the voltage output
pin0 = Pin('P0', Pin.OUT_PP)

# Set external trigger
trigger = 0
def triggerline (line):
    global trigger
    trigger = 1
ext = ExtInt(Pin('P7'), ExtInt.IRQ_RISING, Pin.PULL_DOWN, triggerline)
ext.enable()

# Set trigger pin
pin7 = Pin('P7', Pin.IN)
print(pin7)
print(pin7.value())

# led.on()

m = mjpeg.Mjpeg("openMVvideoFakeSubject.mjpeg") # <----UPDATE THIS FILE NAME!!!! -----

print(trigger)

clock = time.clock()  # Create a clock object to track the FPS.
while(True):
    print(pin7.value())
    print(trigger)
    time.sleep(1)
#    if (trigger == 1):
#        sensor.skip_frames(time=1000) # Wait for the trigger to reset
#        trigger = 0
#        while(trigger == 0):
#            clock.tick()
#            led.on()
#            pin0.high()
#            m.write(sensor.snapshot())
#            # time.sleep(1) # Included to slow pulses/flashes to make them visible
#            pin0.low()
#            led.off()
            # time.sleep(1) # Included to slow pulses/flashes to make them visible
            # print(clock.fps())
#        break

m.close()
# led.off()

raise (Exception("Please reset the camera to see the new file."))

This produces the following terminal output (explanations added):

>>> Pin(Pin.cpu.D12, mode=Pin.IN)
0 (pin7.value() before while loop)
0 (trigger before while loop)
0 (pin7.value() in first pass of while loop)
0 (trigger in first pass of while loop)
1 (pin7.value() in second pass of while loop)
1 (trigger in second pass of while loop)
1
1

Both the trigger variable and pin7.value() start at 0 (as I would expect), but then go to 1 after one pass through the while loop even with no external inputs. (All pins are currently disconnected.) I’m also confused by why it was working with the external input interrupt previously and now it has stopped.

Thank you for your help and I’d appreciate any other suggestions.

~Nick

Not sure if it’s right to use extint and the pin input at the same time. I’d just have the pin input.

As for a tight loop which I was suggesting, you’d do a while loop checking for the pin not being low, then when it changes state do something, then another while loop to wait for it to go back to low.

The only behavior that explains what you are seeing is if the pin doesn’t have a pull down. If it’s floating then when pulled high once it would stay high.

When you declare the pin as an input without specificing pull down this overrides what the previous extint setting was. Please define the pin input with the pull down arg set for it.

Hi @kwagyeman !

Thank you for those suggestions! Frustratingly, I discovered that the problem was some wonky connections between the terminals and the wires. This code works properly with regards to the pin 7 trigger to start and stop recording and with pin 0 sending out a TTL for each frame grab:

import pyb
import sensor
import image
import time
import mjpeg
import machine
import os
from pyb import Pin, ExtInt

sensor.reset()  # Reset and initialize the sensor.
sensor.set_pixformat(sensor.GRAYSCALE)  # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA)  # Set frame size to QVGA (320x240)
sensor.set_windowing((121,101,100,80)) # Set windowing
sensor.skip_frames(time=2000)  # Wait for settings take effect.

# Use triggered mode for frame capture to ensure all pixels are recorded simultaneously
sensor.ioctl(sensor.IOCTL_SET_TRIGGERED_MODE, True)

led = machine.LED("LED_RED")

# Prepare the voltage output
pin0 = Pin('P0', Pin.OUT_PP)
pin0.low()

# Set external trigger
trigger = 0
def triggerline (line):
    global trigger
    trigger = 1
ext = ExtInt(Pin('P7'), ExtInt.IRQ_RISING, Pin.PULL_DOWN, triggerline)
ext.enable()

# Set trigger pin
#pin7 = Pin('P7', Pin.IN,  Pin.PULL_DOWN)
#print(pin7)
#print(pin7.value())

# led.on()

m = mjpeg.Mjpeg("openMVvideoFakeSubject.mjpeg") # <----UPDATE THIS FILE NAME!!!! -----

print(trigger)

clock = time.clock()  # Create a clock object to track the FPS.
while(True):
    #print(pin7.value())
    #print(trigger)
    #time.sleep(1)
    if (trigger == 1):
        sensor.skip_frames(time=1000) # Wait for the trigger to reset
        trigger = 0
        while(trigger == 0):
            clock.tick()
            led.on()
            pin0.high()
            m.write(sensor.snapshot())
            # time.sleep(1) # Included to slow pulses/flashes to make them visible
            pin0.low()
            led.off()
            time.sleep(0.003) # Included to give pin 0 voltage time to reset to 0
            # time.sleep(1) # Included to slow pulses/flashes to make them visible
            print(clock.fps())
        break

m.close()
# led.off()

raise (Exception("Please reset the camera to see the new file."))

I had to include a brief delay after resenting the pin 0 voltage to 0 in order to give time for the voltage to change and be detected by Matlab.

Thank you again for all your help!

~Nick