Obstacle Detection and Avoidance

OpenMV related project discussion.
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Obstacle Detection and Avoidance

Postby mjs513 » Sun Apr 30, 2017 10:21 pm

Real newbie here with machine vision/image processing. Just received my board the other day with the hopes of using it to detect obstacles and avoid them. Right now I am using multiple sonar and IR sensors. Was doing a bunch of research on methods to use monocular vision with OpenCV and it appears totally doable but not sure how to implement on OpenMV.

For instance I found this project, https://bigfacerobotics.wordpress.com/2 ... ng-opencv/ that uses Opencv with a webcam and several functions from OpenCV but I don't think they are all currently supported. Also was interest in an approach the detects the ground (floor) plane through segmentation using Eigen transform, think it was the cv function svd. Guess it is essentially for texture mapping.

Again just learning here so forgive if I got my terminology wrong.

Any help in getting started would be appreciated.

thanks
Mike
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Sun Apr 30, 2017 11:21 pm

Hi, so, are you looking to code this in C? Or use the python interface?

If in the python interface you just need to blur the image via the mean() method and then call the canny method on an image to get the image into line segments. Once you do this you can manually look at images pixels in python to implement the rest of the algorithm described in the link.

While accessing the image in python is slow... you don't have to check every pixel, just every 10-20 pixels. So, things should run quickly. Once you've created a list of image heights from the floor you then just test the difference between two measurements and threshold to get an idea of where obstacles are.

Alternatively, if you know the color of the floor you can just find_blobs equal to not the floor color.

Lot's of different ways to tackle this... all have their trades offs.
Nyamekye,
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Mon May 01, 2017 2:30 pm

@kwagyeman Thanks for the reply. I finally hooked up the camera and its great I love it. Examples are plentiful illustrating the use of the commands. Anyway, I want to implement everything using the python interface. I started using the median filter and edge detection methods and beginning to get a rudimentary understanding of how everything works.

Thanks to your comments it pointed me in the right direction for the commands :). Anyway if I pick up a good book on opencv would that be of help for a newbie like me. Last time I did anything with image processing it was on a ibm360 about 35 years ago, so anything I learned has been forgotten.

Thanks
Mike
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Mon May 01, 2017 4:37 pm

Cool, also, let me know if you need me to make anything in particular "faster". I plan to improve a lot of the library code... but, like anything. I have finite time and focus on fighting fires people tell me about first.

I'll be improving the line detection code next.
Nyamekye,
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Mon May 01, 2017 7:47 pm

Me again. I started coding the method but ran into a problem with memory allocation error. The exact error is MemoryError: memory allocation failed, allocating 16384 bytes.

Here is the code that I am using, any suggestions would be appreciated. Do have a couple of questions, how can I overlay the captured image with the edge detected image or the results of the edgearray?

Code: Select all

# Untitled - By: CyberPalin - Mon May 1 2017


import sensor, image, time, array

sensor.reset() # Initialize the camera sensor.
sensor.set_pixformat(sensor.GRAYSCALE) # or sensor.RGB565
sensor.set_framesize(sensor.QQVGA) # or sensor.QVGA (or others)
sensor.skip_frames(30) # Let new settings take affect.
sensor.set_gainceiling(8)
clock = time.clock() # Tracks FPS.

StepSize = 8

EdgeArray = []

while(True):
    clock.tick() # Track elapsed milliseconds between snapshots().
    img = sensor.snapshot() # Take a picture and return the image.

    # The only argument is the kernel size. N coresponds to a ((N*2)+1)^2
    # kernel size. E.g. 1 == 3x3 kernel, 2 == 5x5 kernel, etc. Note: You
    # shouldn't ever need to use a value bigger than 2.
    img.mean(1)

    # Use Canny edge detector
    img.find_edges(image.EDGE_CANNY, threshold=(50, 80))

    imagewidth = img.width() - 1
    imageheight = img.height() - 1

    print(imagewidth)
    print(imageheight)

    for j in range(0,imagewidth, StepSize):    #for the width of image array
        for i in range(imageheight-5,0,-1):    #step through every pixel in height of array from bottom to top
                                               #Ignore first couple of pixels as may trigger due to undistort
            if img.get_pixel(i,j) == 255:      #check to see if the pixel is white which indicates an edge has been found
                EdgeArray.append((j,i))        #if it is, add x,y coordinates to ObstacleArray
                break                          #if white pixel is found, skip rest of pixels in column
            else:                              #no white pixel found
                EdgeArray.append((j,0))        #if nothing found, assume no obstacle. Set pixel position way off the screen to indicate
                                               #no obstacle detected


    #for x in range (len(EdgeArray)-1):      #draw lines between points in ObstacleArray
    #    cv2.line(img, EdgeArray[x], EdgeArray[x+1],(0,255,0),1)
    #for x in range (len(EdgeArray)):        #draw lines from bottom of the screen to points in ObstacleArray
    #    cv2.line(img, (x*StepSize,imageheight), EdgeArray[x],(0,255,0),1)


    #print(clock.fps()) # Note: Your OpenMV Cam runs about half as fast while
    # connected to your computer. The FPS should increase once disconnected.
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Mon May 01, 2017 9:21 pm

Hi,

Which line are you running out of memory on? EdgeArray something? It looks like you're doing a double loop on that. So, it's growing too big. The camera has enough RAM for you to do one pass on the image, not w*h passes. For each horizontal position you should walk up from the bottom pixel until you hit a white line and record the distance walked. You're current code doesn't do that... but instead is trying to store most of the image in the small few KB of heap space for your micropython program.

As for overlaying the image... this is tricky. So, because the OpenMV Cam is a microcontroller we don't have enough RAM to keep two large copies of an image in RAM really. If you really want to do an image overlay you have to save the captured image to disk, then read it back in:

Code: Select all

img = sensor.snapshot()
img.save("/temp/bg.bmp")
# modify image
img.blend("temp/bg.bmp", alpha=127)
There are other functions than blend which can be used for better masking affects.

...

Yes, I know the above really isn't ideal ;). That said, in mission operation you don't need to do this so it's not that big of a deal when you deploy the system.
Nyamekye,
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Tue May 02, 2017 7:27 pm

Hi Nyamekye

Thanks for getting back to me. Right now I am just trying to implement the code as written in the link. But I think your way is the way to go in the end. You are right about the overlaying of images. Again just trying to see how he implemented the process, what the results are and the overlay is really just for debugging.

Running into another problem right now that has to do with python programming vs image the camera. The EdgeArray variable I think is a series of tuples with each set of coordinate pairs. What I have to do is break each down pair down into separate x0, y0, x1, y1 for draw_line and can not seem to get it through my head how to do that.

This is where I run into problems:

Code: Select all

    for x in range (len(EdgeArray)-1):      #draw lines between points in ObstacleArray
        #img.draw_line(EdgeArray[x], EdgeArray[x+1],color = 0)
        #img.draw_line(EdgeArray[([x][0])], EdgeArray[([x][1])], EdgeArray[([x+1][0])], EdgeArray[([x+1][1])], color=0)
        print(EdgeArray()[[x]])
I got around the memory issue by selectively sensor.snapshot() and also emptying the EdgeArray by redefining it after the loop finishes, in other words doing EdgeArray = [].

Thanks for your help
Mike
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Tue May 02, 2017 8:51 pm

Try this out:

Code: Select all

import sensor, image, time

sensor.reset()                      # Reset and initialize the sensor.
sensor.set_pixformat(sensor.GRAYSCALE) # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QQVGA)   # Set frame size to QVGA (320x240)
sensor.skip_frames(10)              # Wait for settings take effect.
clock = time.clock()                # Create a clock object to track the FPS.

while(True):
    clock.tick()                    # Update the FPS clock.
    img = sensor.snapshot()         # Take a picture and return the image.
    img.mean(1)
    img.find_edges(image.EDGE_CANNY)
    img.erode(1, threshold = 2)
    
    EdgeArray = []
    
    for x in range(img.width()):
        
        flag = False
        for y in range(img.height()):
            z = img.height() - 1 - y
            if img.get_pixel(x, z):
                EdgeArray.append((x, z))
                flag = True
                break
                
        if not flag:
            EdgeArray.append((x, 0))
        
    old = None 
    for i in range(len(EdgeArray)):
        if old != None:
            img.draw_line((old[0], old[1], EdgeArray[i][0], EdgeArray[i][1]), color = 127)
        old = EdgeArray[i]

    print(clock.fps())
             
This basically does what you want. You can see the contour of the edges in light gray. The next step is then to derive some insight from the points around the bottom edge of the image.

Note that scanning through the the pixels one pixel at time kills the performance. That's why this runs at about 3 FPS on the M7. If you skip multiple horizontal pixels the speed will increase by a lot.
Nyamekye,
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Tue May 02, 2017 9:00 pm

For finding a symbol just use the "find_apriltag" method. This finds any apriltag in the image irregardless of lighting, scale, shear, rotation, etc.
Nyamekye,
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Tue May 02, 2017 11:43 pm

Nyamekye

Wow, thanks for thanking the time to adjust my code. Great learning for me. Will give it a try first thing in the morning.

Thanks again
mike
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Wed May 03, 2017 9:51 am

Nyamekye

Been playing with it for a couple hours now and it works great. I changed the stepsize to 8 and got a 7.5 fps update rate. I also added the vertical lines in the image (actually helps. Been moving around obstacles to see what it gives me. Rather fun. Now the fun stuff starts with determining obstacle criteria based on the image data.

Will keep you posted on how it goes.

Mike
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Fri May 05, 2017 3:05 pm

Nyamekye

Thought you would like to see some preliminary results. I took the data and put into an excel spreadsheet so I could determine the best approach to the calculations. Here is a image and the results. Sorry the image is so big. Next time smaller.
Attachments
Picture1.png
Picture1.png (286.76 KiB) Viewed 7800 times
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Fri May 05, 2017 4:36 pm

Cool, looks like you have a hard problem ahead of you though for actually detecting the objects. Seems like you can just threshold the difference between two points to determine object start and stop. A large negative difference is a start, and a large positive difference is a stop. Just toggle a variable when either condition happens.
Nyamekye,
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Fri May 05, 2017 11:54 pm

That's pretty close to what I am doing. I added a few more tests to avoid where there is are spikes in the data. Got it coded up now in python and drawing circles where the obstacles edges are. Just one question, is there easy way to put it back in color mode and then draw circles where the bottom edge of a potential obstacle is. I was able to put it back into color mode but when I went to draw the circles wouldn't work.

There is also one draw back, if is a wall method doesn't work. Planning on adding a time of flight sensor or add a laser light hitting an object along the los of the lens :) Haven't got that far yet.

Thanks
Mike
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Sat May 06, 2017 12:15 am

If you switch the pixformat to RGB565 and then snapshot() the image will be in color. You can then draw in color again. Note that you're frame rate will suffer doing this. Only enable this for debug.
Nyamekye,
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Sat May 06, 2017 12:04 pm

Hi
I did that but what happens is the data points are not persistent, in others, draws one circle than draws the next circle but the first circle gets erased.

Mike
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Sat May 06, 2017 12:18 pm

Yeah, that will happen switching between color spaces.

Um, when I get around to it I'll make canny accept an RGB565 image. Until then, please us that function with just grayscale.
Nyamekye,
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Sat May 06, 2017 12:56 pm

Nyamekye,

Not a problem. No rush. Using it with grayscale.

Thanks
Mike

PS. Just bought the protoshield, wifi shield and wide angle lens. Having fun now.
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Mon May 15, 2017 4:41 pm

In prep for sending data for further processing I noticed that if I do a print to the serial terminal of two arrays back to back, i.e.,

Code: Select all

print(ObsArray3)
print(EdgeArray)
the will only partially print and only the last print:

Code: Select all

 152, 0), (156, 68)]
 0), (156, 73)]
(156, 0)]
If I put a 20 ms delay between the two it will work. This is what should be printed:

Code: Select all

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
[(0, 0), (4, 0), (8, 9), (12, 0), (16, 0), (20, 0), (24, 0), (28, 0), (32, 0), (36, 0), (40, 0), (44, 0), (48, 0), (52, 0), (56, 0), (60, 0), (64, 0), (68, 0), (72, 0), (76, 0), (80, 0), (84, 0), (88, 0), (92, 0), (96, 0), (100, 0), (104, 0), (108, 113), (112, 112), (116, 0), (120, 0), (124, 0), (128, 0), (132, 0), (136, 0), (140, 0), (144, 0), (148, 0), (152, 0), (156, 0)]
Is there a way around this or is this a bug?
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Mon May 15, 2017 7:14 pm

It's not a bug. When printing via the USB debug feature built-in to the IDE a maximum number of bytes can be printed per time unit. The IDE services the print buffer every 20 Ms or so. Print less data not as quickly and you'll be able to see it all.

Um, for large debug data sets you may wish to print to an SD card. Or hook up a serial to USB cable to the OpenMV Cams serial port and print via that.

To fix this issue we'd need to make the print function block when there's no buffer space available... Thus reducing FPS...

Question, do you think that would be better or worse, making print block on text output if there's no space available because the IDE hasn't read the text yet from a USB buffer?
Nyamekye,
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Mon May 15, 2017 7:33 pm

Think it would make it worse to do the blocking. Typically I do prints for testing and debugging purposes and the frame rate would not be critical. As long as we know that this is the case I can adjust the timing of large data prints. When I actually get this going I wouldn't be sending the data via SPI or wifi.

By the way when will UDP be implemented? If it is already do you have an example? As long as I got you I have the new M7 with the standard lens and just picked up the wide angles as well. Do you have the specs on these lens (FOV, lens dia, etc). Do you have any specs on the CCD as well - looking for dimensions? Going to try an experiment.

Thanks
Mike
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Mon May 15, 2017 8:01 pm

Lens specs are on the website on the lens product pages. As for UDP, that's Ibrahim's domain. He needs to be bugged about it. I'll let him know it's something to be fixed net at a higher priority.
Nyamekye,
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Mon May 15, 2017 8:32 pm

Thanks appreciate it. I actually have another question. When I run the edge detection example it shows the edge of the frame as a detected edge. How do you get around that?
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Mon May 15, 2017 10:49 pm

Um, please read up on the edge detection code. Use canny edge detection which does the proper thing. The fast basic one detects the edge of the frame as an edge because the convolution uses zeros for pixels off the edge of the image.
Nyamekye,
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Tue May 23, 2017 7:39 am

Sorry for not getting back here sooner but am in the processes of putting together a test bot and tweaking the code some more. Yep, read up on it and for some dumb reason thought the fast basic one was using canny as well. Think my eyes were getting crossed. By the way I also tested sending data over serial to one of the serial lines on a teensy 3.5 (tested at 115200) and works like a charm.

Mike
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Wed May 31, 2017 6:28 pm

Finally got the code working for at least some baseline testing. The final output that I am sending to a Teensy 3.5 over the uart is the estimated angle of perceived gaps. So anyway, thought I would share the code if anyone is interested (note: not the most efficient). There's also an option to send the image over wifi for testing purposes. I also have a vl53l0x sensor attached to the teensy. Wanted to use it directly with the OpenMv but ran into memory problems.

Enjoy
Mike

UPDATE: Here is a picture of bot.
20170604_211451.png
20170604_211451.png (358.75 KiB) Viewed 7455 times
Attachments
ObstacleAvoidanceMV.py
(7.91 KiB) Downloaded 106 times
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Sat Sep 02, 2017 1:19 pm

ok. Finally got everything at least baselined and tweaked to a point where it does work depending on a few factors I have to work on. I made a little video of it:

https://photos.app.goo.gl/JIc3378SOzJR2bSn1

Cheers.

PS> next version how about lots more memory and higher res camera :) know its a wish list. :)
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Sat Sep 02, 2017 3:46 pm

Hi, it looks like I'll have the weekend free so I'll get that code done for you. The next version will use the STM32H7 which has the same horse power as the raspberry pi 1. It will be able to work on VGA res stuff with CV. Also, we're working on making the camera module switchable and offering a global shutter cam. Resolution will not increase much beyond VGA but to be honest... VGA is pretty much a good res for CV stuff. Even the highest end deep learning systems only work on about 200x200 pixels at a time.

If you'd like to share a write up of the project please send a PR to the OpenMV projects repo on github. You can summit stuff you've done there.
Nyamekye,
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Sat Sep 02, 2017 7:01 pm

Thank you. How detailed do you need the write up?
User avatar
kwagyeman
Posts: 2286
Joined: Sun May 24, 2015 2:10 pm

Re: Obstacle Detection and Avoidance

Postby kwagyeman » Sat Sep 02, 2017 7:49 pm

Whatever you feel comfortable with. You've found the repo correct? Just add another folder at the top level and send a PR with whatever you've done.
Nyamekye,
mjs513
Posts: 74
Joined: Sun Apr 30, 2017 12:52 pm

Re: Obstacle Detection and Avoidance

Postby mjs513 » Sat Sep 02, 2017 8:10 pm

Yep. Saw the donkey car so I figured it was the right one :).

9/3/17 = created pull request :)

Return to “Project Discussion”

Who is online

Users browsing this forum: No registered users and 1 guest