RTSP in Nicla Vision

sata · January 20, 2023, 8:20am

Will nicla vision be able to run the rtsp example from openmv?

I see that the device is able to connect to my router and I’m able to ping the ip, but cannot get it to start streaming in VLC

I also changed the pixel format to rgb565 but I was expecting to fail at a later stage due to encoding issues? but the rtsp stream function loop seems to be stuck in checking

github.com

openmv/openmv/blob/master/scripts/libraries/rtsp.py#L214


      
                          img_data_len = len(img_data)
                          data_len = self.__send(rtp_header + jpeg_header + img_data)
                          if not data_len: break
                          i += img_data_len
                          l -= img_data_len
                      if l: self.__close_socket()
                  except OSError: self.__close_socket()
          
          
def stream(self, image_callback, quality=90): # public
              while True:
                  if self.__valid_tcp_socket():
                      try:
                          self.__tcp__socket.settimeout(0.1)
                          while True:
                              data = self.__tcp__socket.recv(1400)
                              if data and len(data): self.__parse_rtsp_request(data)
                              if self.__playing: self.__send_rtp(image_callback, quality)
                      except OSError: self.__close_tcp_socket()

The ip that nicla is printing is something like 192.168.xx.xx:554
Trying to stream in vlc using rtsp://192.168.xx.xx:554
But it doesn’t seem to be working, I also tried with different wifi routers and mobile hotspots

Did anyone here try streaming rtsp in nicla vision?

Did anyone manage to successfully stream feed via rtsp?

Any help would be much appreciated. Thanks!

kwagyeman · January 23, 2023, 1:02am

Hi, it won’t work on that camera since it only seems to work with JPEGs from the OV5640 camera module. However, you appear to have a different error. Not sure why.

sata · January 23, 2023, 6:42am

Right? Thank you for the reply, will see if I can provide more info on this.

kwagyeman · January 23, 2023, 4:56pm

A wire shark trace helps of the IP communication.

sata · February 8, 2023, 8:52am

Thank you for the suggestion @kwagyeman , I tried tracing it using wire shark and the request is going from vlc to the nicla vision board.

I edited the rtsp module code and added a few debugging statements.

It is throwing an out of memory error, code 12 on this website:
https://mpython.readthedocs.io/en/master/library/pythonStd/uerrno.html

So I guess, nicla vision just doesn’t have enough space on board to handle it?

kwagyeman · February 8, 2023, 4:53pm

Hi, which line of code is it throwing the error on?

sata · February 9, 2023, 8:29am

Hi, thank you for the reply.

github.com

openmv/openmv/blob/master/scripts/libraries/rtsp.py#L21


      
          
          
def __valid_tcp_socket(self): # private
              if self.__tcp__socket is None:
                  try:
                      s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
                      s.bind(self.__myaddr)
                      s.listen(0)
                      s.settimeout(5)
                      self.__tcp__socket, self.__client_addr = s.accept()
                      s.close()
                  except OSError: self.__tcp__socket = None
              return self.__tcp__socket is not None
          
          
def __close_tcp_socket(self): # private
              self.__tcp__socket.close()
              self.__tcp__socket = None
          
          
def __valid_udp_socket(self): # private
              if self.__udp__socket is None:
                  try:
                      self.__udp__socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

This is where it fails, I printed the exception here and it fails multiple times before finally showing os error 12

I edit the code a bit to see what exactly vlc sends to the board. These are the commands that I found

DESCRIBE rtsp://192.168.105.221:554 RTSP/1.0 CSeq: 3
User-Agent: LibVLC/3.0.16 (LIVE555 Streaming Media v2016.11.28)
Accept: application/sdp

This works with out any error and the device sent:

3 Content-Type: application/sdp
Content-Length: 98

After this, I got OSError(98,) at the checkpoint, which is Address already in use according to uerrno – System Error Code — mPython board 2.2.2 documentation

This repeated for about 5 times before I either get a OSError(12,) or sometimes I get a connection from VLC again

VLC Sent:

SETUP rtsp://192.168.105.221:554 RTSP/1.0 
CSeq: 0
Transport: RTP/AVP;unicast;client_port=9256-9257

This also worked I guess, not sure about it though

The device continued to raise OSError 12 memory error at this point
I printed the free memory using gc.mem_free() and it shows 159808, which I hope is sufficient?

It continued for some time before the VLC raised the error that it’s failing to connect with the message:
VLC is unable to open the MRL ‘rtsp://192.168.105.221:554’ Check the log for details.

My guess is the socket continues to be in use even after s.close() ?, not sure about it though.

Any help would be much appreciated.

Thank you!

kwagyeman · February 9, 2023, 7:56pm

What RTSP program are you using to talk to the camera? That’s the easiest for me to use to debug with. I can’t say the example code I wrote is universally spec compliant.

sata · February 10, 2023, 12:10pm

Hi,

Okay, got it. I saw in one instance the nicla board was sending 400 bad request.

I’m trying to run this example on nicla vision

github.com

openmv/openmv/blob/master/scripts/examples/08-RPC-Library/36-Web-Servers/rtsp_video_server_wlan.py

# RTSP Video Server
#
# This example shows off how to stream video over RTSP with your OpenMV Cam.
#
# You can use a program like VLC to view the video stream by connecting to the
# OpenMV Cam's IP address.

import network, omv, rtsp, sensor, time

# RTP MJPEG streaming works using JPEG images produced by the OV2640/OV5640 camera modules.
# Not all programs (e.g. VLC) implement the full JPEG standard for decoding any JPEG image
# in RTP packets. Images JPEG compressed by the OpenMV Cam internally may not display.

# FFPLAY will correctly handle JPEGs produced by OpenMV software.

sensor.reset()

sensor.set_pixformat(sensor.JPEG) # Only supported by the OV2640/OV5640.
sensor.set_framesize(sensor.UXGA)

This file has been truncated. show original

And I’m using VLC media player v3.0.16 to connect with the board

Do you think VLC might be the problem here?

I tried with OpenCV as well to check if it can open the stream and it also failed and made no connection with the board.

Not sure if it helps but the MJPEG streaming example (https://github.com/openmv/openmv/blob/master/scripts/examples/10-Arduino-Boards/Nicla-Vision/03-WiFi/mjpeg_streamer.py) is working perfectly well.

Thank you!

kwagyeman · February 10, 2023, 3:35pm

Yeah, I have never got it working with VLC. I don’t get an error though but just a blank image as VLC doesn’t decide the jpegs we send.

sata · February 13, 2023, 7:37am

Hi,

Okay, understood.

So, if not VLC, could you please suggest any players which would be able to stream the rtsp cam from openmv.

Not specifically for Nicla Vision board, OpenMV rtsp module compatibility in general.

Thank you!

kwagyeman · February 13, 2023, 5:50pm

Hi, I’ll see what I can do to update the code for this and get some fixes in this week.

sata · February 14, 2023, 10:09am

Awesome, thank you!

kwagyeman · February 15, 2023, 5:15am

Okay, something is broken inside of the WiFi driver. I can’t get sockets to accept connections. Also, they seem to block indefinitely even though there’s a timeout.

This works with the WiFi shield on our other cameras but I guess there’s an issue with the Nicla WiFi.

…

Anyway, the issue is lower than RTSP. The WiFi driver just doesn’t accept socket connections correctly.

@iabdalkader

Needs some help getting this fixed.

kwagyeman · February 15, 2023, 5:21am

kwagyeman · February 20, 2023, 11:08pm

Okay, I have a fix up for UDP streaming mode. scripts/libraries: Fix RSTP library for the winc. by kwagyeman · Pull Request #1786 · openmv/openmv · GitHub

Once this is merged the development firmware should work well enough.

TCP mode (which is not really used but should work too on principle) needs drivers to be updated.

kwagyeman · February 20, 2023, 11:11pm

sata · February 21, 2023, 10:36am

hi @kwagyeman ,

This is Amazing!!

Thank you so much for providing the fixes.

I built the latest master branch from openmv and tried the rtsp-wlan example using my nicla board now over wi-fi.

I used:

ffplay -rtsp_transport udp rtsp://192.168.xx.xx:554

It connected but unfortunately it was failing after a couple of frames.

It shows:

Could you please share the example script you’re running? Maybe I could cross check if there’s anything wrong here?

Thank you!

kwagyeman · February 21, 2023, 3:33pm

The PR fix is not merged yet.

Eliza · February 21, 2023, 10:00pm

Hi, I would appreciate if someone can help me. How could modified the library rtsp.py? I cant find the file. Am I missing something?