Page 1 of 1
Image Stream
Posted: Wed Aug 02, 2017 8:06 am
by WolleSmith
Hey guys,
I´m currently working on an image stream from openMV to PC by using LabVIEW. I´m trying to achieve, to import the snapshots from the cam to LabVIEW as png images. I noticed that the Image Class does not support png. So I am transfering a jpg stream. I am able to get that stream in LabVIEW and write it to a jpg file. That´s working perfectly. But I have two questions:
1. I´m currently using this code on the openMV module:
Code: Select all
import time, pyb, sensor
#some init code
while(usb.isconnected()):
img = sensor.snapshot()
img.save("test.jpg")
with open("test.jpg", mode='rb') as file:
fileContent = file.read()
usb.send(data=fileContent)
time.sleep(200)
So as you can see, I´m creating a jpg file on the openMV module just to read it out and stream the data to the PC. Is there a solution to convert the data stored in img to jpg (or even png) without writing a jpg file?
2. Does anybody know the algorithm to convert jpg data to png? I mean the binary data. Otherwise I receive the jpg stream on the PC, create a jpg file and have to convert that file to png..
Best case scenario would be to get a png stream to my LabVIEW application without creating any file. Just take a snapshot, convert the data to png and transfer the png data to PC. Or take a snapshot, convert the data to jpg, transfer the jpg data to PC and convert it to png there.
Thank you in advance!
Re: Image Stream
Posted: Wed Aug 02, 2017 9:51 am
by WolleSmith
Allright, so I found a great DLL to convert the JPEG stream into an image in LabVIEW. Awesome. Taking a snapshot, sending it to the PC and show it on my GUI is taking about 25 ms now. That would be fast enough for my application. But I´m still curious if there´s a way to generate that JPEG string without creating a jpg file on the openMV memory.
Re: Image Stream
Posted: Wed Aug 02, 2017 10:39 am
by kwagyeman
Hi, you can get jpg images via the USB debug protocol. Would this work for you? It's a lot of code though in C to use that... Not sure labview can do.
Re: Image Stream
Posted: Wed Aug 02, 2017 11:26 am
by WolleSmith
kwagyeman wrote: ↑Wed Aug 02, 2017 10:39 am
Hi, you can get jpg images via the USB debug protocol. Would this work for you? It's a lot of code though in C to use that... Not sure labview can do.
Thank´s for your answer. I have no idea what you mean by USB debug protocol and was not able to find it searching the web.
And you´re saying c and LabVIEW - does that mean I would send the sensor.snapshot()) data to PC and convert it to jpg there?
Best solution would be to convert the object sensor.snapshot() returns to a jpg string on the openMV controller. Isn´t that possible without writing a jpg file and read it binary to get that string?
Re: Image Stream
Posted: Wed Aug 02, 2017 1:28 pm
by kwagyeman
The USB debug protocol is what OpenMV IDE uses to pull images off the OpenMV Cam at near the FPS the camera is running at. It's not documented except in code.
We have two files which you can use to pull the jpeg frame buffer. There's an openmv.py script in the usr directory on our github that does the protocol and is still valid. Please see that script and make frame buffer calls like it does in labview using a serial port to pull the jpeg compressed frame buffer at near native speed.
Re: Image Stream
Posted: Wed Aug 02, 2017 2:14 pm
by WolleSmith
Aaaah, I tried to find out how the IDE gets the image data using a USB sniffer, but couldn´t make it.
Thank you so much for the suggestion. That sounds really great and like a much better solution than my ugly workaround. I´ll definitely check that out tomorrow.
Re: Image Stream
Posted: Thu Aug 03, 2017 7:31 am
by WolleSmith
All right.
1. Using an aquivalent to the fb_size() function results in getting an Array [320, 240, ~3000]. Is ~3000 a plausible value for the frame buffer size? On my working alternative the jpg files have about 4000 Bytes.
2. When I use a function that represents the fb_dump() of the openmv.py script I get binary data (see attached screenshot) which cannot be interpreted as jpg. In the original fb_dump() function there´s one code line to decode the buffer data:
buff = np.asarray(Image.frombuffer("RGB", size[0:2], buff, "jpeg", "RGB", ""))
But I don´t think I can create a LabVIEW function to replace that line. So that protocol does not seem to work for me. Except I call a python script from LabVIEW that contains that line. But I guess that slows down the application to much. Thank you for your help anyway!
Re: Image Stream
Posted: Thu Aug 03, 2017 11:49 am
by kwagyeman
Mmm, well, look at the guys of OpenMV IDE and you'll be able to see the protocol in action using C. There's a lot of code though...
Re: Image Stream
Posted: Thu Aug 03, 2017 12:32 pm
by WolleSmith
Thank´s but due to a deadline I´ll stick to my working code. I can get my head around a smarter alternative if there´s time at the end of the project.
Re: Image Stream
Posted: Thu Aug 03, 2017 8:53 pm
by iabdalkader
WolleSmith wrote: ↑Thu Aug 03, 2017 12:32 pm
Thank´s but due to a deadline I´ll stick to my working code. I can get my head around a smarter alternative if there´s time at the end of the project.
Hi, the Python script is outdated. The camera only sends JPEG images now, which is what you need, right ?
Update:
Although the script is outdated, I think it still works, it just assumes there are other formats sent from the camera.
This line decodes JPEG to RGB888:
buff = np.asarray(Image.frombuffer("RGB", size[0:2], buff, "jpeg", "RGB", ""))
Re: Image Stream
Posted: Fri Aug 04, 2017 1:53 am
by WolleSmith
I don´t really need RGB8888. I just need the jpg data. So sending the __USBDBG_FRAME_DUMP command should get me that data, right? If you take a look at the screenshot I attached in an earlier reply, you see the data I get back when sending that command. But that data is not jpg format. When I log that binary string as a jpg file I cannot open it.
Re: Image Stream
Posted: Fri Aug 04, 2017 2:12 am
by kwagyeman
What script is running?
Re: Image Stream
Posted: Fri Aug 04, 2017 2:27 am
by WolleSmith
The hello world script. Off course it´s running as main.py on the camera. Like mentioned the commands seem to work. The __USBDBG_FRAME_SIZE command returns data that looks legit. But the binary data returned when sending the __USBDBG_FRAME_DUMP command does not.
Re: Image Stream
Posted: Fri Aug 04, 2017 2:36 am
by WolleSmith
This is how it looks like in LabVIEW. Pretty easy to read I guess.
Re: Image Stream
Posted: Fri Aug 04, 2017 2:38 am
by kwagyeman
Here's the actual code openmv ide uses:
https://github.com/openmv/qt-creator/bl ... uginio.cpp
https://github.com/openmv/qt-creator/bl ... alport.cpp
The IO layer is the transaction layer and the serial port layer is another thread which executes commands and returns the response.
Note that the IDE does... a lot more... to be compatible on a lot of systems. You don't need to handle quite all the stuff it does.
Re: Image Stream
Posted: Fri Aug 04, 2017 2:41 am
by kwagyeman
Um, so, the protocol that the openmv.py script uses hasn't changed. So, wondering if there's some byte scrambling problem for you.
Re: Image Stream
Posted: Fri Aug 04, 2017 2:42 am
by kwagyeman
Um, can you verify the first command to get the frame size works correctly? As in it returns a sane frame size?
Re: Image Stream
Posted: Fri Aug 04, 2017 2:55 am
by WolleSmith
The first command Returns 12 Bytes. You guys Interpret this 12 Bytes as 3 integers with 4 bytes each:
Code: Select all
__serial.write(struct.pack("<BBI", __USBDBG_CMD, __USBDBG_FRAME_SIZE, __FB_HDR_SIZE))
struct.unpack("III", __serial.read(12))
I do the same and get the integers 320, 240 and something about 3000. 320x240 is the size of the Image. This third integer should be the frame size. Now you take that third integer and use it in fb_dump() like this:
Code: Select all
num_bytes = size[2]
__serial.write(struct.pack("<BBI", __USBDBG_CMD, __USBDBG_FRAME_DUMP, num_bytes))
buff = __serial.read(num_bytes)
That´s what I am doing, too. 'buff' in your script equals 'frame buf read' in my application. But that is no jpg data. It contains the same line over and over again.
Re: Image Stream
Posted: Fri Aug 04, 2017 3:05 am
by kwagyeman
Can labview handle the data rate? It's at 12 MBps. We've seen issues with this kinda thing before. Older Macs require the frame buffer to be dumped in MTU sized chunks.
Re: Image Stream
Posted: Fri Aug 04, 2017 4:54 am
by WolleSmith
That is not a LabVIEW issue. I also look at the data using an usb sniffer. Looks the same..
Re: Image Stream
Posted: Fri Aug 04, 2017 9:46 am
by iabdalkader
Can you attach the data you get ?
Re: Image Stream
Posted: Fri Aug 04, 2017 11:53 am
by kwagyeman
Hi, what I mean by MTU sized chunks is that on older MACs we've seen the OS drop bytes. Like, literally fom receiveing a USB packet to then giving it to the user application the OS drops bytes because it doesn't expect that much data to go through a serial port at USB speeds.
Re: Image Stream
Posted: Fri Aug 04, 2017 11:55 am
by kwagyeman
Oh, okay, so you see the same data with a USB sniffer... Um, question, are all the bytes between 0x80 and 0xBF?
Re: Image Stream
Posted: Fri Aug 04, 2017 2:41 pm
by iabdalkader
Even if the driver is dropping bytes, the data won't be the same . I'm interested in a hexdump maybe it will give us a clue to what the cam is sending.
Update:
Or maybe you're getting the header (which should be the same) anyway a hex dump will help a lot.
Re: Image Stream
Posted: Sat Aug 05, 2017 8:47 am
by WolleSmith
My guess ist that I´ve sent wrong commands. The constant 12 that I send with the USB_FRAME_Size is a string. Means the two characters '1' and '2' are sent and not a 4 Byte integer. I do the same at the USB_FRAME_DUMB command. I just cast the frame size to a string which results in sending a string like "3459" instead of an integer 3459. I will change that in my code on monday, try again and forward the results to you guys.
Thank´s for the help so far. I really appreciate it.
Re: Image Stream
Posted: Mon Aug 07, 2017 3:17 am
by WolleSmith
All right. I´ll attach some files with the binary data I got back from the cam. I tried a couple of different settings. That´s why the content is different. I noticed one thing: When I use a baud rate of 115200 I often get back the FPS instead of the Frame buf. Or let´s say I believe it´s the FPS (values of about 28.xxx).
Re: Image Stream
Posted: Mon Aug 07, 2017 3:24 am
by WolleSmith
There was an error uploading the files. Here they are.
Re: Image Stream
Posted: Mon Aug 07, 2017 3:43 am
by WolleSmith
Guys what about sending the data sensor.snapshot() returns to the pc. Is there a way to convert that data to a jpg or png or whatever? For example by using the PIL modul python provides. I can start python scripts using LabVIEW.
Re: Image Stream
Posted: Mon Aug 07, 2017 3:58 am
by kwagyeman
Yeah, you can do that directly. Um, just do:
Code: Select all
img = sensor.snapshot()
print(img.compressed_for_ide())
This will come out of whatever serial port the thing is attached too. Here's how to decode the data stream:
https://github.com/openmv/qt-creator/bl ... o.cpp#L781
...
alternatively, for a raw jpeg file create a serial usb vcp object, I think you did this before and just write:
Code: Select all
serial_vcp.write(sensor.snapshot().compressed())
The compressed() method compressed the image and returns a jpeg version of it. The print() statement won't print the data normally unless it's been compressed for the IDE which will the compressed image and then update the FB. To just send raw compressed jpeg data you have to send it out through the uart/vcp/spi with write().
Re: Image Stream
Posted: Mon Aug 07, 2017 4:00 am
by kwagyeman
Remember the camera supports REPL. You can command it via a script from the PC if you want. MicroPython probably has a way to send the camera a file to execute... look around in their repo... and get the serial responses.
Re: Image Stream
Posted: Mon Aug 07, 2017 4:23 am
by WolleSmith
Thank you so much. Jesus, that '.compressed()' is the thing I was looking for the whole time.
Re: Image Stream
Posted: Mon Aug 07, 2017 12:01 pm
by kwagyeman
Always remember to search the docs.

Re: Image Stream
Posted: Mon Aug 07, 2017 2:14 pm
by WolleSmith
Haha, you´re absolutely right. I did and I saw the function but somehow I misunderstood its functionality. If I had just tried it. But no worries. I´m just glad that everything works fine now. Even though the image stream is slower than I expected. I get an image every 80 ms. In my opinion that´s way so slow ain´t it?
Code: Select all
... #init stuff
while(True):
try:
while(not usb.isconnected()):
streamFlg = False
time.sleep(100)
while(streamFlg and usb.isconnected()):
usb.send(data=sensor.snapshot().compressed())
usb.send(data=10) #Termination character
usb.send(data="END")
usb.send(data=10)
except KeyboardInterrupt:
streamFlg = not streamFlg[\Code]
Re: Image Stream
Posted: Mon Aug 07, 2017 2:55 pm
by kwagyeman
So... We use really low compression quality to the GUI. Try lowering the compression quality and if you can... Please log where time is spent. I have a feeling though that it's spending more time compressing. Compress has a quality arg. Lower it.
Re: Image Stream
Posted: Mon Aug 07, 2017 3:46 pm
by iabdalkader
Here's a hex dump of "frame buf.txt" looks like you're getting the JPEG header over and over again.
Code: Select all
00000000 20 20 20 20 f0 20 20 20 16 07 20 20 6a 70 65 ea | . .. jpe.|
00000010 24 1d 99 5d b3 6d f0 39 aa 7c 82 08 2d 01 c7 7f |$..].m.9.|..-...|
00000020 8e 77 bd c5 5e d1 6f 62 ff 09 5b d9 08 bd bc c4 |.w..^.ob..[.....|
00000030 32 20 61 e0 21 71 ac 0d a6 83 76 95 bd 14 9d e6 |2 a.!q....v.....|
00000040 20 20 20 20 f0 20 20 20 16 07 20 20 6a 70 65 ea | . .. jpe.|
00000050 24 1d 99 5d b3 6d f0 39 aa 7c 82 08 2d 01 c7 7f |$..].m.9.|..-...|
00000060 8e 77 bd c5 5e d1 6f 62 ff 09 5b d9 08 bd bc c4 |.w..^.ob..[.....|
00000070 32 20 61 e0 21 71 ac 0d a6 83 76 95 bd 14 9d e6 |2 a.!q....v.....|
00000080 20 20 20 20 f0 20 20 20 16 07 20 20 6a 70 65 ea | . .. jpe.|
00000090 24 1d 99 5d b3 6d f0 39 aa 7c 82 08 2d 01 c7 7f |$..].m.9.|..-...|
000000a0 8e 77 bd c5 5e d1 6f 62 ff 09 5b d9 08 bd bc c4 |.w..^.ob..[.....|
000000b0 32 20 61 e0 21 71 ac 0d a6 83 76 95 bd 14 9d e6 |2 a.!q....v.....|
000000c0 20 20 20 20 f0 20 20 20 | . |
Re: Image Stream
Posted: Mon Aug 21, 2017 4:17 am
by WolleSmith
iabdalkader wrote: ↑Mon Aug 07, 2017 3:46 pm
Here's a hex dump of "frame buf.txt" looks like you're getting the JPEG header over and over again.
Code: Select all
00000000 20 20 20 20 f0 20 20 20 16 07 20 20 6a 70 65 ea | . .. jpe.|
00000010 24 1d 99 5d b3 6d f0 39 aa 7c 82 08 2d 01 c7 7f |$..].m.9.|..-...|
00000020 8e 77 bd c5 5e d1 6f 62 ff 09 5b d9 08 bd bc c4 |.w..^.ob..[.....|
00000030 32 20 61 e0 21 71 ac 0d a6 83 76 95 bd 14 9d e6 |2 a.!q....v.....|
00000040 20 20 20 20 f0 20 20 20 16 07 20 20 6a 70 65 ea | . .. jpe.|
00000050 24 1d 99 5d b3 6d f0 39 aa 7c 82 08 2d 01 c7 7f |$..].m.9.|..-...|
00000060 8e 77 bd c5 5e d1 6f 62 ff 09 5b d9 08 bd bc c4 |.w..^.ob..[.....|
00000070 32 20 61 e0 21 71 ac 0d a6 83 76 95 bd 14 9d e6 |2 a.!q....v.....|
00000080 20 20 20 20 f0 20 20 20 16 07 20 20 6a 70 65 ea | . .. jpe.|
00000090 24 1d 99 5d b3 6d f0 39 aa 7c 82 08 2d 01 c7 7f |$..].m.9.|..-...|
000000a0 8e 77 bd c5 5e d1 6f 62 ff 09 5b d9 08 bd bc c4 |.w..^.ob..[.....|
000000b0 32 20 61 e0 21 71 ac 0d a6 83 76 95 bd 14 9d e6 |2 a.!q....v.....|
000000c0 20 20 20 20 f0 20 20 20 | . |
Thank you very much. Do you have any idea why this is happening? Sorry for the late reply. I´ve been working on something else for quite a while now.
Re: Image Stream
Posted: Mon Aug 21, 2017 11:51 am
by kwagyeman
I'm sorry, I don't. We've demonstrated that our code works well for this. I don't know how NI's stuff operates. Is it possible for Lab View to call a system process that invokes our python script to save a file with the image data?
Re: Image Stream
Posted: Mon Aug 21, 2017 12:05 pm
by WolleSmith
Allright. Then I will stick to my current version. Thank you both for the kind support

Re: Image Stream
Posted: Mon Aug 21, 2017 2:53 pm
by kwagyeman
When time allows I will create some python scripts for interfacing to the OpenMV Cam using OpenMV IDEs robust comm stack. This should allow others to get stuff working nicely.
Re: Image Stream
Posted: Thu Aug 24, 2017 6:08 pm
by iabdalkader
WolleSmith wrote: ↑Mon Aug 21, 2017 4:17 am
Thank you very much. Do you have any idea why this is happening? Sorry for the late reply. I´ve been working on something else for quite a while now.
Hi, I think you're not reading the whole frame, just the first packet. Note in the hex dump, you're reading exactly 64 byte packets (the max size of the USB FS end point). Not sure how your code works, but you should request the whole frame size from the cam, then the cam will keep sending 64bytes packets plus the remainder/last packet <= 64 bytes.