Creating command and control via MQTT

Project Discussion
1

Hi all,

I have a question about my project and I hope someone has some experience on this.

I have put the OpenMV cam + WiFi shield on a Elecfreaks CuteBot (powered by a Micro:bit) in order to make a remote controlled robot (with autopilot based on images). My goal is first to control the robot via my browser while seeing everything it does. That requires the board to do two things:

  1. Show video over the network
  2. Accept commands from the network
  3. Conttinuously control the robot

My first attempt was creating a HTTP server that would have a GET command for getting images and a POST for commands but this was not satisfactory as the HTTP server would need to (blocking) listen on a port for connections and hence couldn’t do anything else in between. This meant that (3) was not possible (and also HTTP was not always reliable).

So I decided MQTT might be a better choice here as the publish/subscribe model seems to be make more sense. However, I run into performance/reliability problems and I’m not sure if what I want is out of the ordinary or if I’m doing something wrong.

My setup is now: MQTT server (mosquitto) hosted locally, my laptop with webapp connecting via websocket to mqtt server, openMV that has two connections to the server via the usual port. I have pasted some of the code below.

First, the reason I am using two separate client connections is that I use one for sending images and one for receiving commands. I tried doing it over one connection but I quickly ran into problems that check_msg would fail when sending multiple (large) messages. Having two connections seems to fix the problem.

The problem I run into now is that the cam misses messages sent by the browser (client2.check_msg) if I send messages too quickly from the cam (client.publish). I have used a loop variable to artificially tone down on the FPS that is sent to the browser which seems to work around the problem a bit. If I disable the sending of image (cam = False) everything works perfectly.

The images that are sent are only about 12kb in size and I get a decent framerate, meaning that the check_msg should be called on a very regular basis.

What should I do to minimise the chance of missing commands while maximizing the FPS sent to the browser?

from mqtt import MQTTClient

<sensor setup>

client = MQTTClient("openmv-publish", <server>, port=1883)
client2 = MQTTClient("openmv-listen", <server>, port=1883)
client.connect()
client2.connect()

def callback(topic, msg):
    global command
    command = msg.decode('utf-8')
    print(topic, msg.decode('utf-8'));

# must set callback first
client2.set_callback(callback)
client2.subscribe("commands")

loop = 0
while True:
    try:
        loop = loop + 1
        # Check for messages
        client2.check_msg()

        # Handle command
        if command:
            # handle commands i got

        # Do video stuff
        objects, frame = do_video()

        if camon and loop % 5 == 0:
            msg = ubinascii.b2a_base64(frame.compress(quality=35)).decode('utf-8').strip()
            client.publish("videostream", msg);
    except Exception as e:
        print(e)
        raise
2

Have you tried our updated RTSP code? With the latest development firmware it’s very fast and lets you achieve the maximum frame rate while being able to connect to the camera remotely via FFPMEG.

As for MQTT. The library is probably fine but if we didn’t put effort into make it non-blocking and it’s still just really a demo you may need to take the python script that has the code in it and edit it to be more performant: openmv/mqtt.py at master · openmv/openmv · GitHub

For an example of how to write better socket code see our rtsp.py code: openmv/rtsp.py at master · openmv/openmv · GitHub

You’ll notice that in the RTSP code I aggressively handle all types of errors, recreate sockets, have low timeouts, and etc. The code only handles one connection at a time. It could be expanded to handle multiple. But, it runs smoothly now.

3

Hi Kwabena,

first of all, thank you for your reply - I am really impressed with how much effort you put in trying to answer each question that comes in :slight_smile:

I have not checked out the latest rtsp codd (I think), just the one in the examples. The issue that I experienced is that if I use the (current) RTSP streamer, I do get high video rate to get video but the HTTP server is unable to accept additional requests.

I will have a look at your links later this week to see if I find some inspiration to make the MQTT a little better.

What I am looking for is something like:

while True:
	
	read_sensor()

	make_decision_based_on_video()

	look_for_new_commands_by_user()

	send_video_if_requested()

Since there is no multithreading yet (I think?) , this code will not work because I would need to listen for new commands (over HTTP for example) and this currently blocks.

Ideally you would have one thread handling incoming connections and the other handling the read_sensor and make_decision functions.

But I will take your code pointers as inspiration to see where it gets me :smiley: Thanks!

4

Hi, there’s actually a WiFi debug protocol that’s more apt for this which works based on interrupts. However, I think it only works in the winc right now. We need to upgrade it to work on the 1DX unit.

Yeah, so, what you are saying is that you want the main thread to do work without waiting on someone to connect.

I understand, the best recommendation is to take the rstp library on our GitHub and just edit it. This is very easy to do. You can then insert all kinds of things you want in its main loop code.

1 Like