OpenMV

Posted: **Wed Aug 02, 2017 8:06 am**

Hey guys,

I´m currently working on an image stream from openMV to PC by using LabVIEW. I´m trying to achieve, to import the snapshots from the cam to LabVIEW as png images. I noticed that the Image Class does not support png. So I am transfering a jpg stream. I am able to get that stream in LabVIEW and write it to a jpg file. That´s working perfectly. But I have two questions:

1. I´m currently using this code on the openMV module:

Code: Select all

import time, pyb, sensor

#some init code

while(usb.isconnected()):
    img = sensor.snapshot()
    img.save("test.jpg")
    with open("test.jpg", mode='rb') as file:
        fileContent = file.read()
        usb.send(data=fileContent)
    time.sleep(200)

So as you can see, I´m creating a jpg file on the openMV module just to read it out and stream the data to the PC. Is there a solution to convert the data stored in img to jpg (or even png) without writing a jpg file?

2. Does anybody know the algorithm to convert jpg data to png? I mean the binary data. Otherwise I receive the jpg stream on the PC, create a jpg file and have to convert that file to png..

Best case scenario would be to get a png stream to my LabVIEW application without creating any file. Just take a snapshot, convert the data to png and transfer the png data to PC. Or take a snapshot, convert the data to jpg, transfer the jpg data to PC and convert it to png there.

Thank you in advance!

Posted: **Wed Aug 02, 2017 9:51 am**

Allright, so I found a great DLL to convert the JPEG stream into an image in LabVIEW. Awesome. Taking a snapshot, sending it to the PC and show it on my GUI is taking about 25 ms now. That would be fast enough for my application. But I´m still curious if there´s a way to generate that JPEG string without creating a jpg file on the openMV memory.

Posted: **Wed Aug 02, 2017 10:39 am**

Hi, you can get jpg images via the USB debug protocol. Would this work for you? It's a lot of code though in C to use that... Not sure labview can do.

Posted: **Wed Aug 02, 2017 11:26 am**

kwagyeman wrote: ↑
Wed Aug 02, 2017 10:39 am
Hi, you can get jpg images via the USB debug protocol. Would this work for you? It's a lot of code though in C to use that... Not sure labview can do.

Thank´s for your answer. I have no idea what you mean by USB debug protocol and was not able to find it searching the web.
And you´re saying c and LabVIEW - does that mean I would send the sensor.snapshot()) data to PC and convert it to jpg there?
Best solution would be to convert the object sensor.snapshot() returns to a jpg string on the openMV controller. Isn´t that possible without writing a jpg file and read it binary to get that string?

Posted: **Wed Aug 02, 2017 1:28 pm**

The USB debug protocol is what OpenMV IDE uses to pull images off the OpenMV Cam at near the FPS the camera is running at. It's not documented except in code.

We have two files which you can use to pull the jpeg frame buffer. There's an openmv.py script in the usr directory on our github that does the protocol and is still valid. Please see that script and make frame buffer calls like it does in labview using a serial port to pull the jpeg compressed frame buffer at near native speed.

Posted: **Wed Aug 02, 2017 2:14 pm**

Aaaah, I tried to find out how the IDE gets the image data using a USB sniffer, but couldn´t make it.
Thank you so much for the suggestion. That sounds really great and like a much better solution than my ugly workaround. I´ll definitely check that out tomorrow.

Posted: **Thu Aug 03, 2017 7:31 am**

All right.

1. Using an aquivalent to the fb_size() function results in getting an Array [320, 240, ~3000]. Is ~3000 a plausible value for the frame buffer size? On my working alternative the jpg files have about 4000 Bytes.

2. When I use a function that represents the fb_dump() of the openmv.py script I get binary data (see attached screenshot) which cannot be interpreted as jpg. In the original fb_dump() function there´s one code line to decode the buffer data:
buff = np.asarray(Image.frombuffer("RGB", size[0:2], buff, "jpeg", "RGB", ""))
But I don´t think I can create a LabVIEW function to replace that line. So that protocol does not seem to work for me. Except I call a python script from LabVIEW that contains that line. But I guess that slows down the application to much. Thank you for your help anyway!

Posted: **Thu Aug 03, 2017 11:49 am**

Mmm, well, look at the guys of OpenMV IDE and you'll be able to see the protocol in action using C. There's a lot of code though...

Posted: **Thu Aug 03, 2017 12:32 pm**

Thank´s but due to a deadline I´ll stick to my working code. I can get my head around a smarter alternative if there´s time at the end of the project.

Posted: **Thu Aug 03, 2017 8:53 pm**

WolleSmith wrote: ↑
Thu Aug 03, 2017 12:32 pm
Thank´s but due to a deadline I´ll stick to my working code. I can get my head around a smarter alternative if there´s time at the end of the project.

Hi, the Python script is outdated. The camera only sends JPEG images now, which is what you need, right ?

Update:

Although the script is outdated, I think it still works, it just assumes there are other formats sent from the camera.

This line decodes JPEG to RGB888:

buff = np.asarray(Image.frombuffer("RGB", size[0:2], buff, "jpeg", "RGB", ""))

Posted: **Fri Aug 04, 2017 1:53 am**

I don´t really need RGB8888. I just need the jpg data. So sending the __USBDBG_FRAME_DUMP command should get me that data, right? If you take a look at the screenshot I attached in an earlier reply, you see the data I get back when sending that command. But that data is not jpg format. When I log that binary string as a jpg file I cannot open it.

Posted: **Fri Aug 04, 2017 2:12 am**

What script is running?

Posted: **Fri Aug 04, 2017 2:27 am**

The hello world script. Off course it´s running as main.py on the camera. Like mentioned the commands seem to work. The __USBDBG_FRAME_SIZE command returns data that looks legit. But the binary data returned when sending the __USBDBG_FRAME_DUMP command does not.

Posted: **Fri Aug 04, 2017 2:36 am**

This is how it looks like in LabVIEW. Pretty easy to read I guess.

Posted: **Fri Aug 04, 2017 2:38 am**

Here's the actual code openmv ide uses:

https://github.com/openmv/qt-creator/bl ... uginio.cpp

https://github.com/openmv/qt-creator/bl ... alport.cpp

The IO layer is the transaction layer and the serial port layer is another thread which executes commands and returns the response.

Note that the IDE does... a lot more... to be compatible on a lot of systems. You don't need to handle quite all the stuff it does.

Posted: **Fri Aug 04, 2017 2:41 am**

Um, so, the protocol that the openmv.py script uses hasn't changed. So, wondering if there's some byte scrambling problem for you.

Posted: **Fri Aug 04, 2017 2:42 am**

Um, can you verify the first command to get the frame size works correctly? As in it returns a sane frame size?

Posted: **Fri Aug 04, 2017 2:55 am**

The first command Returns 12 Bytes. You guys Interpret this 12 Bytes as 3 integers with 4 bytes each:

Code: Select all

__serial.write(struct.pack("<BBI", __USBDBG_CMD, __USBDBG_FRAME_SIZE, __FB_HDR_SIZE))
struct.unpack("III", __serial.read(12))

I do the same and get the integers 320, 240 and something about 3000. 320x240 is the size of the Image. This third integer should be the frame size. Now you take that third integer and use it in fb_dump() like this:

Code: Select all

num_bytes = size[2]
__serial.write(struct.pack("<BBI", __USBDBG_CMD, __USBDBG_FRAME_DUMP, num_bytes))
buff = __serial.read(num_bytes)

That´s what I am doing, too. 'buff' in your script equals 'frame buf read' in my application. But that is no jpg data. It contains the same line over and over again.

Posted: **Fri Aug 04, 2017 3:05 am**

Can labview handle the data rate? It's at 12 MBps. We've seen issues with this kinda thing before. Older Macs require the frame buffer to be dumped in MTU sized chunks.

Posted: **Fri Aug 04, 2017 4:54 am**

That is not a LabVIEW issue. I also look at the data using an usb sniffer. Looks the same..

Posted: **Fri Aug 04, 2017 9:46 am**

Can you attach the data you get ?

Posted: **Fri Aug 04, 2017 11:53 am**

Hi, what I mean by MTU sized chunks is that on older MACs we've seen the OS drop bytes. Like, literally fom receiveing a USB packet to then giving it to the user application the OS drops bytes because it doesn't expect that much data to go through a serial port at USB speeds.

Posted: **Fri Aug 04, 2017 11:55 am**

Oh, okay, so you see the same data with a USB sniffer... Um, question, are all the bytes between 0x80 and 0xBF?

Posted: **Fri Aug 04, 2017 2:41 pm**

Even if the driver is dropping bytes, the data won't be the same . I'm interested in a hexdump maybe it will give us a clue to what the cam is sending.

Update:
Or maybe you're getting the header (which should be the same) anyway a hex dump will help a lot.

Posted: **Sat Aug 05, 2017 8:47 am**

My guess ist that I´ve sent wrong commands. The constant 12 that I send with the USB_FRAME_Size is a string. Means the two characters '1' and '2' are sent and not a 4 Byte integer. I do the same at the USB_FRAME_DUMB command. I just cast the frame size to a string which results in sending a string like "3459" instead of an integer 3459. I will change that in my code on monday, try again and forward the results to you guys.
Thank´s for the help so far. I really appreciate it.

Posted: **Mon Aug 07, 2017 3:17 am**

All right. I´ll attach some files with the binary data I got back from the cam. I tried a couple of different settings. That´s why the content is different. I noticed one thing: When I use a baud rate of 115200 I often get back the FPS instead of the Frame buf. Or let´s say I believe it´s the FPS (values of about 28.xxx).

Posted: **Mon Aug 07, 2017 3:24 am**

There was an error uploading the files. Here they are.

Posted: **Mon Aug 07, 2017 3:43 am**

Guys what about sending the data sensor.snapshot() returns to the pc. Is there a way to convert that data to a jpg or png or whatever? For example by using the PIL modul python provides. I can start python scripts using LabVIEW.

Posted: **Mon Aug 07, 2017 3:58 am**

Yeah, you can do that directly. Um, just do:

Code: Select all

img = sensor.snapshot()
print(img.compressed_for_ide())

This will come out of whatever serial port the thing is attached too. Here's how to decode the data stream:

https://github.com/openmv/qt-creator/bl ... o.cpp#L781

...

alternatively, for a raw jpeg file create a serial usb vcp object, I think you did this before and just write:

Code: Select all

serial_vcp.write(sensor.snapshot().compressed())

The compressed() method compressed the image and returns a jpeg version of it. The print() statement won't print the data normally unless it's been compressed for the IDE which will the compressed image and then update the FB. To just send raw compressed jpeg data you have to send it out through the uart/vcp/spi with write().

Posted: **Mon Aug 07, 2017 4:00 am**

Remember the camera supports REPL. You can command it via a script from the PC if you want. MicroPython probably has a way to send the camera a file to execute... look around in their repo... and get the serial responses.

Posted: **Mon Aug 07, 2017 4:23 am**

Thank you so much. Jesus, that '.compressed()' is the thing I was looking for the whole time.

Posted: **Mon Aug 07, 2017 12:01 pm**

Always remember to search the docs.

Posted: **Mon Aug 07, 2017 2:14 pm**

Haha, you´re absolutely right. I did and I saw the function but somehow I misunderstood its functionality. If I had just tried it. But no worries. I´m just glad that everything works fine now. Even though the image stream is slower than I expected. I get an image every 80 ms. In my opinion that´s way so slow ain´t it?

Code: Select all

...	#init stuff
while(True):
    try:
        while(not usb.isconnected()):
            streamFlg = False
            time.sleep(100)
        while(streamFlg and usb.isconnected()):
            usb.send(data=sensor.snapshot().compressed())
            usb.send(data=10)							#Termination character
            usb.send(data="END")
            usb.send(data=10)
    except KeyboardInterrupt:
        streamFlg = not streamFlg[\Code]

Posted: **Mon Aug 07, 2017 2:55 pm**

So... We use really low compression quality to the GUI. Try lowering the compression quality and if you can... Please log where time is spent. I have a feeling though that it's spending more time compressing. Compress has a quality arg. Lower it.

Posted: **Mon Aug 07, 2017 3:46 pm**

Here's a hex dump of "frame buf.txt" looks like you're getting the JPEG header over and over again.

Code: Select all

00000000  20 20 20 20 f0 20 20 20  16 07 20 20 6a 70 65 ea  |    .   ..  jpe.|
00000010  24 1d 99 5d b3 6d f0 39  aa 7c 82 08 2d 01 c7 7f  |$..].m.9.|..-...|
00000020  8e 77 bd c5 5e d1 6f 62  ff 09 5b d9 08 bd bc c4  |.w..^.ob..[.....|
00000030  32 20 61 e0 21 71 ac 0d  a6 83 76 95 bd 14 9d e6  |2 a.!q....v.....|

00000040  20 20 20 20 f0 20 20 20  16 07 20 20 6a 70 65 ea  |    .   ..  jpe.|
00000050  24 1d 99 5d b3 6d f0 39  aa 7c 82 08 2d 01 c7 7f  |$..].m.9.|..-...|
00000060  8e 77 bd c5 5e d1 6f 62  ff 09 5b d9 08 bd bc c4  |.w..^.ob..[.....|
00000070  32 20 61 e0 21 71 ac 0d  a6 83 76 95 bd 14 9d e6  |2 a.!q....v.....|

00000080  20 20 20 20 f0 20 20 20  16 07 20 20 6a 70 65 ea  |    .   ..  jpe.|
00000090  24 1d 99 5d b3 6d f0 39  aa 7c 82 08 2d 01 c7 7f  |$..].m.9.|..-...|
000000a0  8e 77 bd c5 5e d1 6f 62  ff 09 5b d9 08 bd bc c4  |.w..^.ob..[.....|
000000b0  32 20 61 e0 21 71 ac 0d  a6 83 76 95 bd 14 9d e6  |2 a.!q....v.....|
000000c0  20 20 20 20 f0 20 20 20                           |    .   |

Posted: **Mon Aug 21, 2017 4:17 am**

iabdalkader wrote: ↑

Mon Aug 07, 2017 3:46 pm

Here's a hex dump of "frame buf.txt" looks like you're getting the JPEG header over and over again.

Code: Select all

00000000  20 20 20 20 f0 20 20 20  16 07 20 20 6a 70 65 ea  |    .   ..  jpe.|
00000010  24 1d 99 5d b3 6d f0 39  aa 7c 82 08 2d 01 c7 7f  |$..].m.9.|..-...|
00000020  8e 77 bd c5 5e d1 6f 62  ff 09 5b d9 08 bd bc c4  |.w..^.ob..[.....|
00000030  32 20 61 e0 21 71 ac 0d  a6 83 76 95 bd 14 9d e6  |2 a.!q....v.....|

00000040  20 20 20 20 f0 20 20 20  16 07 20 20 6a 70 65 ea  |    .   ..  jpe.|
00000050  24 1d 99 5d b3 6d f0 39  aa 7c 82 08 2d 01 c7 7f  |$..].m.9.|..-...|
00000060  8e 77 bd c5 5e d1 6f 62  ff 09 5b d9 08 bd bc c4  |.w..^.ob..[.....|
00000070  32 20 61 e0 21 71 ac 0d  a6 83 76 95 bd 14 9d e6  |2 a.!q....v.....|

00000080  20 20 20 20 f0 20 20 20  16 07 20 20 6a 70 65 ea  |    .   ..  jpe.|
00000090  24 1d 99 5d b3 6d f0 39  aa 7c 82 08 2d 01 c7 7f  |$..].m.9.|..-...|
000000a0  8e 77 bd c5 5e d1 6f 62  ff 09 5b d9 08 bd bc c4  |.w..^.ob..[.....|
000000b0  32 20 61 e0 21 71 ac 0d  a6 83 76 95 bd 14 9d e6  |2 a.!q....v.....|
000000c0  20 20 20 20 f0 20 20 20                           |    .   |

Thank you very much. Do you have any idea why this is happening? Sorry for the late reply. I´ve been working on something else for quite a while now.

Posted: **Mon Aug 21, 2017 11:51 am**

I'm sorry, I don't. We've demonstrated that our code works well for this. I don't know how NI's stuff operates. Is it possible for Lab View to call a system process that invokes our python script to save a file with the image data?

Posted: **Mon Aug 21, 2017 12:05 pm**

Allright. Then I will stick to my current version. Thank you both for the kind support

Posted: **Mon Aug 21, 2017 2:53 pm**

When time allows I will create some python scripts for interfacing to the OpenMV Cam using OpenMV IDEs robust comm stack. This should allow others to get stuff working nicely.

Posted: **Thu Aug 24, 2017 6:08 pm**

WolleSmith wrote: ↑
Mon Aug 21, 2017 4:17 am
Thank you very much. Do you have any idea why this is happening? Sorry for the late reply. I´ve been working on something else for quite a while now.

Hi, I think you're not reading the whole frame, just the first packet. Note in the hex dump, you're reading exactly 64 byte packets (the max size of the USB FS end point). Not sure how your code works, but you should request the whole frame size from the cam, then the cam will keep sending 64bytes packets plus the remainder/last packet <= 64 bytes.

OpenMV

Image Stream

Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream

Re: Image Stream