Higher FPS

General discussion about topics related to OpenMV.
User avatar
iabdalkader
Posts: 544
Joined: Sun May 24, 2015 3:53 pm

Higher FPS

Postby iabdalkader » Mon May 15, 2017 5:13 pm

Hi, I've been working on the sensor code lately, and it seems the sensor actually supports a much higher FPS, without going into too many details, the higher FPS enables the cam to capture frames when needed and run faster (unless it's running an slow algorithm). So, I'm attaching a firmware image if anyone's interested in testing. Please let me know if you find any problems (but keep in mind this is a development firmware image) you should see at least 42FPS when running helloworld.py (if you disable the FB it goes up to 84FPS :shock: and same for color blobs using a single blob)

FPS.gif
FPS.gif (1.55 MiB) Viewed 2792 times
Attachments
firmware.zip
(814.61 KiB) Downloaded 84 times
User avatar
sergeyzwezdin
Posts: 8
Joined: Wed May 24, 2017 11:56 pm
Location: Chelyabinsk, Russia
Contact:

Re: Higher FPS

Postby sergeyzwezdin » Thu May 25, 2017 12:12 am

I just flashed my OpenMV M7 with your firmware, but don't any changes in FPS. It's still around 16FPS with FB enabled and around 28-29FPS with FB disabled.

What I did to flash the device:
1. Connected OpenMV to Windows laptop
2. Opened OpenMV IDE
3. Without connecting to the device: Tools → Run Bootloader
4. Selected extracted firmware.bin file
5. Pressed Run with "erase internal FS" checked.

What I'm doing wrong?

1.png
1.png (230.08 KiB) Viewed 2714 times
2.png
2.png (230.58 KiB) Viewed 2714 times
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Thu May 25, 2017 12:20 am

Can you try a single color tracking script? The multi-blob one goes over the image 3 times for each color. I tested with the single color script and got 56 FPS or so.
Nyamekye,
User avatar
sergeyzwezdin
Posts: 8
Joined: Wed May 24, 2017 11:56 pm
Location: Chelyabinsk, Russia
Contact:

Re: Higher FPS

Postby sergeyzwezdin » Thu May 25, 2017 1:36 am

I tuned it a bit. I've switchted to grayscale and decreased framesize. I also decresed size of roi.
So now it's around 80FPS.

Thank you very much for your work, guys.

3.png
3.png (29.15 KiB) Viewed 2711 times
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Sat May 05, 2018 7:14 am

Hello gentlemen,

I tried to follow sergeyzwezdin install instructions, but it never makes it past the "if this takes more than 5 seconds please cancel it" message.

After I cancel it says it can't contact the boot-loader. The current firmware is at 2.7.0. This looks rather new, does it work, I mean it should erase the current firmware in the Arm.

I can see the green led doing it's boot-loader thing, but then it just hangs. Is there a work around for this?
Do I need to jump a pin to ground someplace?

Thanks for the help you can provide,

Robert
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Sat May 05, 2018 2:29 pm

Hmm, please wait for the latest IDE. It will come out this weekend.

We did an update of the bootloader and the old IDE can't handle it. If you updated your board's firmware using DFU recently then you'd have this issue.

That said, you may also use DFU to update your firmware. Please see the OpenMV.dfu file in the firmware folder.
Nyamekye,
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Sat May 05, 2018 10:59 pm

Yes, I did update the firmware from the IDE. And if so I have this problem. If the new IDE come out this weekend, then I don't need to worry about it..

In the worst case, I do have The Easy STM32 Pro development board and the debugger/programmer is built into the board. I have not looked at the pin-out of the openMV M7 cam, But I'm going to
guess, that the Programming pins can be accessed there.

If so I can just update the firmware with that too. But I think your boot loader would get erased by doing this. I'll just wait for the new IDE. :D
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Sun May 06, 2018 2:02 am

Alas I did look in that firmware folder that I downloaded from this topic, It only contains firmware.bin, there is no other file contained in it that I can find.

I did download openMV.dfu zip file from 2016 from this site. But I'm not sure if it's the most current, so I did not use it to update the firmware.
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Sun May 06, 2018 2:02 pm

What are you trying to do exactly?
Nyamekye,
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Sun May 06, 2018 8:00 pm

I'm building a robot, and it is moving a 1 meter per second, it needs to read a data matrix (10x10) at that speed.
The frames per second are around 6 to 7 FPS. I'm worrying, that it will not be able to read the data matrix at that speed.

So i'm trying to get the FPS as high as possible....

Has that IDE been released as of yet? Note, I'm doing this in a windows environment, can also do this in Linux. if it okay to use the DFU that was created in 2016 to place this firmware on the M7?
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Sun May 06, 2018 8:45 pm

Hi, I'm almost done with the release. I've just finishing up releasing for the Pi right now and will post it in a few hours.

Um, so, data matrix speed is limited by the CPU. Messing around with the camera sensor will not improve it. That said, you can change the effort level the data matrix algorithm uses. This might help.
Nyamekye,
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Sun May 06, 2018 9:13 pm

Hello there,

I just tried this out on my laptop with windows 10, the IDE placed the firmware on the M7 without any issues. However the windows 7 desktop seems to have an issue, I will make sure that both IDE's are the same version.

Once I did place this firmware on it and tried to run it did complain about my program, I will try it with the hello world one...

Thanks for your hard work,

Robert
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Sun May 06, 2018 11:28 pm

hello there, I'm using the 1.9.0 version now. Works perfect!

By adjusting the frame size to QQQVGA I can double the FPS count. However if I try the QQQQVGA it has an issue with FB How do I disable FB?

I'm not sure where this is located

Robert

P.S never mind the FB I found it from a search here....
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Mon May 07, 2018 1:53 am

Not really sure QQQQVGA works correctly. That's a 40x30 image right? Mmm, will fix.

Um, for a higher FPS play around with the sensor exposure control. You can go way higher if you lower it. But, to adjust you have to make the gain go way up which amplifies the noise. See the camera exposure control scripts.
Nyamekye,
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Mon May 07, 2018 2:08 am

Yes, that is the right image size for QQQQVGA, I can turn off the Frame buffer in the ide but when I save the script to the OpenMV M7 it still
gets frame buffer error in a serial application.

It does work in the IDE with it disabled, just can't seem to save it to the M7. I tried to do it manually in the code, but never got it to disable,
do to my newness of this ide. (not sure of how to write it in the code thus far.)

I will have a look at exposure time also, thanks for the tip,

Robert
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Mon May 07, 2018 2:35 am

Hi, you can't disable the frame buffer in the code. When you click the button in the IDE it just tells the camera it need not try to read a JPEG image from the camera. The camera will not produce another JPEG image if the current one has not been read.

Um, anyway, what's the error otherwise? Can you take a screen shot?
Nyamekye,
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Mon May 07, 2018 3:44 am

Hi there,
when using QQQQVGA I get a frame buffer collision error in the serial terminal. I'll list the exact error in a few mins.

Exact error message in terminal window, also in the IDE:

Traceback (most recent call last):
File "/main.py", line 22, in <module>
MemoryError: FB Alloc Collision!!!
MicroPython v1.9.3-4944-gd5396e0a4 on 2018-05-06; OPENMV3 with STM32F765
Type "help()" for more information.
>>>
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Mon May 07, 2018 4:09 am

And the code is:

Code: Select all

# OpenMV 10x10 data matrix decoder
# Added the Infrared LED'S to the project
# Removed the IR blocker on the camera lens, it can now read matrix codes in the dark
# Pi 3.14159265359

import sensor, image, time, math, pyb

from pyb import LED
red_led   = LED(1)
green_led = LED(2)
blue_led  = LED(3)
ir_led    = LED(4)

def led_control(x):

#    if (x&8)==8: ir_led.on()
    if (x&1)==1: red_led.on()
#    if (x&2)==2: green_led.on()
#    if (x&4)==4: blue_led.on()



sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE) # GrayScale runs faster
#sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QQQQVGA)
sensor.skip_frames(time = 2000)
sensor.set_auto_gain(False)  # must turn this off to prevent image washout...
sensor.set_auto_whitebal(False)  # must turn this off to prevent image washout...
clock = time.clock()

while(True):
    clock.tick()
    img = sensor.snapshot()
    img.lens_corr(1.8) # strength of 1.8 is good for the 2.8mm lens.
    for i in range(16):
        led_control(i)

    matrices = img.find_datamatrices()
    for matrix in matrices:
        img.draw_rectangle(matrix.rect(), color = (255, 0, 0))
#        print_args = (matrix.rows(), matrix.columns(), matrix.payload(), (180 * matrix.rotation()) / math.pi, clock.fps())
#        print("Matrix [%d:%d], Payload \"%s\", rotation %f (degrees), FPS %f" % print_args)
#        print("N"+ matrix.payload()+"X"+ str(matrix.x())+"Y"+ str(matrix.y())+"R"+ str(matrix.rotation()))
        Cx = matrix.x() + (matrix.w()/2) # get the center of the matrix "X" value
        Cy = matrix.y() + (matrix.h()/2) # get the center of the matrix "Y" value
        Xoff = (img.width()/2) - Cx # Determine how much if any error is present for the X axis
        Yoff = (img.height()/2) - Cy # Determine how much if any error is present for the Y axis

        print("N"+ str(matrix.payload())+"X"+ str(int(math.ceil(Xoff)))+"Y"+ str(int(math.ceil(Yoff)))+"R"+ str(int(math.ceil((180 * matrix.rotation() / math.pi)))))
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Mon May 07, 2018 11:50 am

Hmm, that's means you're out of RAM. Need to make that error more verbose.

Um, the resolution is really small however. This shouldn't happen.

I'll try your code out and see what's up. You should have no memory issues.
Nyamekye,
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Tue May 08, 2018 8:12 pm

Hi there,

Any new information yet?
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Wed May 09, 2018 1:07 am

Hi, sorry, this error is expected. I will update the error messages for this in the future. Just do this:

Code: Select all

# OpenMV 10x10 data matrix decoder
# Added the Infrared LED'S to the project
# Removed the IR blocker on the camera lens, it can now read matrix codes in the dark
# Pi 3.14159265359

import sensor, image, time, math, pyb

from pyb import LED
red_led   = LED(1)
green_led = LED(2)
blue_led  = LED(3)
ir_led    = LED(4)

def led_control(x):

#    if (x&8)==8: ir_led.on()
    if (x&1)==1: red_led.on()
#    if (x&2)==2: green_led.on()
#    if (x&4)==4: blue_led.on()



sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE) # GrayScale runs faster
#sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QQQVGA)
sensor.skip_frames(time = 2000)
sensor.set_auto_gain(False)  # must turn this off to prevent image washout...
sensor.set_auto_whitebal(False)  # must turn this off to prevent image washout...
clock = time.clock()

while(True):
    clock.tick()
    img = sensor.snapshot()
    img.lens_corr(1.8) # strength of 1.8 is good for the 2.8mm lens.
    for i in range(16):
        led_control(i)

    matrices =[]
    try:
        matrices = img.find_datamatrices()
    except OSError:
        pass
    for matrix in matrices:
        img.draw_rectangle(matrix.rect(), color = (255, 0, 0))
#        print_args = (matrix.rows(), matrix.columns(), matrix.payload(), (180 * matrix.rotation()) / math.pi, clock.fps())
#        print("Matrix [%d:%d], Payload \"%s\", rotation %f (degrees), FPS %f" % print_args)
#        print("N"+ matrix.payload()+"X"+ str(matrix.x())+"Y"+ str(matrix.y())+"R"+ str(matrix.rotation()))
        Cx = matrix.x() + (matrix.w()/2) # get the center of the matrix "X" value
        Cy = matrix.y() + (matrix.h()/2) # get the center of the matrix "Y" value
        Xoff = (img.width()/2) - Cx # Determine how much if any error is present for the X axis
        Yoff = (img.height()/2) - Cy # Determine how much if any error is present for the Y axis

        print("N"+ str(matrix.payload())+"X"+ str(int(math.ceil(Xoff)))+"Y"+ str(int(math.ceil(Yoff)))+"R"+ str(int(math.ceil((180 * matrix.rotation() / math.pi)))))
As of right now, the errors for running out of memory are all the same. If you look at a datamatrix code... you get the fb collision error when you don't have any temporary allocated heap space left. The data matrix code is not particular amazing and allocates memory somewhat wastefully.
Nyamekye,
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Wed May 09, 2018 8:51 am

Hi,

Thanks, nice lil trick...

What about April tags, Can it do these faster, without running out of memory?

I just tried out the April tags, it reads these really fast as opposed to the 10x10 data matrix. and Did not generate a FB out of mem error...

I'm checking with my boss now, as it's not to late to change the tags already placed on the floor.

Robert
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Wed May 09, 2018 12:04 pm

Um, the AprilTag code has the same issue internally, but, much harder to hit. You should always wrap your script with the try and catch stuff in general and handle any errors that are thrown.

Um, the largest AprilTag dictionary I provide is 2k sysmbols. However, if you put two tags right next to each other then you can have 2k^2 symbols. You may also make your own new AprilTag dictionary and edit our firmware for even more tags. However, it's far easier to just have two codes or more in the same area.

Assuming you are locating a robot in a room... Use one tag for X and one for Y. Note that you could make the tags different colors and use binary to select for one or the other. Or, just use multiple OpenMV Cam's and place the two tags in physically different areas. The cameras are cheap. No point in trying to make one do everything. Maybe a third camera to follow a line?
Nyamekye,
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Fri May 11, 2018 3:21 am

Hello there,

I'm now using the April tags and reading them at 85 FPS! I have this connected to the O droidXU4 running a program
Created in Xojo. Tonight I will perform an actual speed test on the camera in light and darkness and test just how fast it can read the April Codes. I'm hoping all the way up too maybe 3 meters pes second...

If successful, I can't wait to get my hands on the H7! I ordered two more of the M7's that makes 3 M7's in total
for just the one bot...

I'll post the information back here after the test. ;)

Robert
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Fri May 11, 2018 11:08 am

Hi, how were you able to achieve that speed?
Nyamekye,
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Fri May 11, 2018 11:42 am

Hello there,

Testing is still ongoing at this time, but I dare say it can see better in the dark than in the day! it's performance excels at night, right where it needs to be. It will be on the bottom of the robot with a lot less light than normal....

As to how I got to that speed, just a fault on my part, an accident that works well.... Also in the default code it turns gain and exposure to false, mine reads it better with fine adjustments to the
exposure and gain settings...

Just by toying around with those and the resolution, I was able to hit that number. As I speak right now from an hour of testing the camera, it can support 3 meters per second travel speed with April
tags.

This is much higher than the spec given to me by the company! what a nice camera! I would make a suggestion, make the IR leds adjustable via pwm. they are way to bright without the filter in place.

I'll post more later...

Robert
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Fri May 11, 2018 12:07 pm

Hi, there's no timer on the LED pins (it's used to generate the clock for the camera). Maybe the M7/H7 have a timer now there though. I can ask Ibrahim to check. The original M4 did not have one to spare.

Anyway, if you're willing please post your code for others to use. Or, if you feel this a valuable advantage don't. I ask because quite a few other customers have asked to do this exact same thing and I've been unable to meet their needs. This example code would make them happy.
Nyamekye,
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Fri May 11, 2018 12:26 pm

Hi there, I have no problem with posting that code, But since it was developed while working with this company, even by mistake I have to run it by the company boss first. I'm in China, so I'm going to say its a big no!

But no worries, It can come out as freeware? My email address is ... a private address they know nothing about. Send me an email to reply to, I will send it to you... don't share it, just make some changes with it to tweak the program... This way nobody comes out on the short end of the stick!

Robert

Anybody else asking for this will be a flat out no, so don't bother...

Cheers,

Robert
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Fri May 11, 2018 12:51 pm

Okay, that's fine. I don't actually need it. Anyone else reading this thread can work out what to do themselves if they put the time in to get the same performance.

I appreciate that you've actually put effort in to use the camera versus trying to run the algorithm at the highest resolution and saying it's too slow. Everyone wants to run everything at crazy high resolutions and then complains when it's slow. Not many folks actually try to work out what resolution they need to hit their target and then work from there.

Note, I removed your email address from the above post. Don't publicly post that...
Nyamekye,
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Fri May 11, 2018 1:07 pm

Well it's still a work in progress, but yes, that just how it was played out, what can this camera do?

It works well enough to get the April tags at three meters per second on my end. giving the testing that was performed on my behalf... given that was not an in company test and performed at my lab in my house, but non the less valid.... (it does work)

Now to integrate this into the, beating out of the major manufactures of Beijing here in China. your camera kills them, they just don't know it as of yet!

Keep up the great work that you are doing, I will use your camera any day of the week, versus theirs!

Robert
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Fri May 11, 2018 1:12 pm

Another thing to think about, even if we just produce 1000 of the things, that's still 9000 M7 needed to fill the bill, are you ready for this?

Robert
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Fri May 11, 2018 2:32 pm

Please email us offline about production. If you are serious about building we have a CM in China called Etonnet who's able to build these things. Making 10k for this is not a problem. They actually do all of OmniVisions dev boards so that have this magic ability to get the camera chips.

Large orders potentially like yours are why they even entertain us. So, they been waiting for someone to build a commercial product using the cam. They will be happy to hear about what you are doing.

Note, since you'd be paying for a production order the prices will be a lot lower than retail.
Nyamekye,
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Sat May 12, 2018 7:41 am

That's awesome, great to hear.

It's Saturday night here, I won't be able to mess with this on the bot until Maybe Tuesday of this coming week. They want it running right away, but I still have to get the Beijing camera out of it, and install the OpenMV M7 into it, then change the programming to read it over the serial port...

The coolest thing about the OpenMV M7 is that we have control over how the camera works as far as programming. This is not the case with just purchasing one off the shelf from a major suppler
of higher end cameras...

They really like the fact it can be used in different ways.

Robert
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Sat May 12, 2018 9:24 am

Woot- Woot,

I'm at 3.5 meters per second 90.9 FPS! This camera has a lot of untapped potential!

I'm not to sure how much farther I can push it, but I will keep trying...

Man I love this M7!

You guys have made my whole plants day! We were shooting for 3.0 meters per second, but I think I can still squeeze a lil bit more from it.
User avatar
robert_d1968
Posts: 29
Joined: Thu May 03, 2018 9:44 am

Re: Higher FPS

Postby robert_d1968 » Sat May 12, 2018 11:00 am

Hello there,

what email address should I use to contact you for the initial order?
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Sat May 12, 2018 11:21 am

Nyamekye,
User avatar
kwagyeman
Posts: 2283
Joined: Sun May 24, 2015 2:10 pm

Re: Higher FPS

Postby kwagyeman » Sat Jun 09, 2018 1:54 pm

Hi, any news?
Nyamekye,

Return to “General Topics”

Who is online

Users browsing this forum: No registered users and 2 guests