Finding AprilTags at a distance

Discussion related to "under the hood" OpenMV topics.
User avatar
kwagyeman
Posts: 2461
Joined: Sun May 24, 2015 2:10 pm

Finding AprilTags at a distance

Postby kwagyeman » Tue Jun 05, 2018 2:28 am

By using image pre-filtering you can find apriltags at a distance:

Code: Select all

# Find Small Apriltags
#
# This script shows off how to use blob tracking as a pre-filter to
# finding Apriltags in the image using blob tracking to find the
# area of where the tag is first and then calling find_apriltags
# on that blob.

# Note, this script works well assuming most parts of the image do not
# pass the thresholding test... otherwise, you don't get a distance
# benefit.

import sensor, image, time, math, omv

# Set the thresholds to find a white object (i.e. tag border)
thresholds = (150, 255)

sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE)
if omv.board_type() == "H7": sensor.set_framesize(sensor.VGA)
elif omv.board_type() == "M7": sensor.set_framesize(sensor.QVGA)
else: raise Exception("You need a more powerful OpenMV Cam to run this script")
sensor.skip_frames(time = 200) # increase this to let the auto methods run for longer
sensor.set_auto_gain(False) # must be turned off for color tracking
sensor.set_auto_whitebal(False) # must be turned off for color tracking
clock = time.clock()

# The apriltag code supports up to 6 tag families which can be processed at the same time.
# Returned tag objects will have their tag family and id within the tag family.
tag_families = 0
tag_families |= image.TAG16H5 # comment out to disable this family
tag_families |= image.TAG25H7 # comment out to disable this family
tag_families |= image.TAG25H9 # comment out to disable this family
tag_families |= image.TAG36H10 # comment out to disable this family
tag_families |= image.TAG36H11 # comment out to disable this family (default family)
tag_families |= image.ARTOOLKIT # comment out to disable this family

while(True):
    clock.tick()
    img = sensor.snapshot()

    # First, we find blobs that may be candidates for tags.
    box_list = []

    # AprilTags may fail due to not having enough ram given the image sie being passed.
    tag_list = []

    for blob in img.find_blobs([thresholds], pixels_threshold=100, area_threshold=100, merge=True):
        # Next we look for a tag in an ROI that's bigger than the blob.
        w = min(max(int(blob.w() * 1.2), 20), 100) # Not too small, not too big.
        h = min(max(int(blob.h() * 1.2), 20), 100) # Not too small, not too big.
        x = min(max(int(blob.x() - (w * 0.1)), 0), img.width()-1)
        y = min(max(int(blob.y() - (h * 0.1)), 0), img.height()-1)

        box_list.append((x, y, w, h)) # We'll draw these later.

        # Since we constrict the roi size apriltags shouldn't run out of ram.
        # But, if it does we handle it...
        try:
            tag_list.extend(img.find_apriltags(roi=(x,y,w,h), families=tag_families))
        except (MemoryError, OSError): # Don't catch all exceptions otherwise you can't stop the script.
            pass

    for b in box_list:
        img.draw_rectangle(b)
    # Now print out the found tags
    for tag in tag_list:
        img.draw_rectangle(tag.rect())
        img.draw_cross(tag.cx(), tag.cy())
        for c in tag.corners():
            img.draw_circle(c[0], c[1], 5)
        print("Tag:", tag.cx(), tag.cy(), tag.rotation(), tag.id())
Note that there's a bug in the apriltag code related to the minimum roi. Please don't make the ROI too small. It causes crashes if you do.
Nyamekye,
sexton
Posts: 21
Joined: Sat Jun 09, 2018 1:08 pm

Re: Finding AprilTags at a distance

Postby sexton » Fri Jun 22, 2018 12:08 am

In my applications I don't need to see the images and mainly use the IDE just for testing and troubleshooting. My apriltag routines follow your examples pretty closely. I am curious if removing img.draw_ commands will save memory of any significant amount. Is it something to consider?
User avatar
kwagyeman
Posts: 2461
Joined: Sun May 24, 2015 2:10 pm

Re: Finding AprilTags at a distance

Postby kwagyeman » Fri Jun 22, 2018 12:27 am

Hi, I just updated the AprilTag code in the next firmware release. You can do 200x200 on the M7 now and also 100x400 or 400x100 if you know if the tag is vertically or horizontally centered. Can you wait for the new release or do you need it now?

Theses reses are without any pre-processing.
Nyamekye,
sexton
Posts: 21
Joined: Sat Jun 09, 2018 1:08 pm

Re: Finding AprilTags at a distance

Postby sexton » Fri Jun 22, 2018 12:44 am

My son and I have some testing planned for around July 2nd. It would be great to test the new firmware. When do you plan to release it? Thanks.
User avatar
kwagyeman
Posts: 2461
Joined: Sun May 24, 2015 2:10 pm

Re: Finding AprilTags at a distance

Postby kwagyeman » Fri Jun 22, 2018 1:01 am

Try the zoom example with windowing set to 200x200.

Note... there's some issue with set windowing I have to fix. It looks like it only works on sizes that are multiples of 8.
Attachments
firmware.zip
(1.75 MiB) Downloaded 19 times
Nyamekye,
sexton
Posts: 21
Joined: Sat Jun 09, 2018 1:08 pm

Re: Finding AprilTags at a distance

Postby sexton » Fri Jun 22, 2018 9:56 am

Thanks. I will give it a try. Windowing at 250x160 should also work right?
User avatar
kwagyeman
Posts: 2461
Joined: Sun May 24, 2015 2:10 pm

Re: Finding AprilTags at a distance

Postby kwagyeman » Fri Jun 22, 2018 11:21 am

Um, make sure the res for w and h and a multiple of 8. There's a bug with set windowing that causes issues if they aren't I need to fix. Otherwise, yes, it should work.
Nyamekye,
sexton
Posts: 21
Joined: Sat Jun 09, 2018 1:08 pm

Re: Finding AprilTags at a distance

Postby sexton » Fri Jun 22, 2018 12:35 pm

I am doing some bench testing now.
windowing of
200x200
250x160
160x250
400x100
100x400
are all 40000 so match the current multiple of 8 rule.
I think QVGA will work with the best balance for our application but will also try VGA. We need from around 30" to as far as possible. Caregivers are different heights so we also have to balance that with the windowing. So we will see what combination gives the best results.

Thanks.
sexton
Posts: 21
Joined: Sat Jun 09, 2018 1:08 pm

Re: Finding AprilTags at a distance

Postby sexton » Fri Jun 22, 2018 2:34 pm

In my initial testing zoom example on M7 with your firmware showing version 3.0.0 I see the following :
1. Baseline VGA with 160x120 windowing works well
2. Baseline QVGA with 160x120 windowing works well
3. VGA with 200x200 windowing works but not as frequently picking up the tag as 160x120
4. QVGA with 200x200 windowing doesn't work
5. 250x160 windowing doesn't work with either VGA or QVGA

I will try to setup more tests but will take some time. If you improve the firmware in the meantime I will load later versions.
User avatar
kwagyeman
Posts: 2461
Joined: Sun May 24, 2015 2:10 pm

Re: Finding AprilTags at a distance

Postby kwagyeman » Fri Jun 22, 2018 5:13 pm

Hi, increasing the resolution results in less detects to do memory limits. There's not much I can do about this. At 200x200 you're at more or less peak memory usage. The firmware fix however greatly improves detection at 160x120 since it frees up a lot of RAM.

I was able to get 200x200 to work staring at a large tag on a screen. However, in a natural environment like the outdoors you have more edges which generates more points in the image which take more RAM to process.
Nyamekye,
sexton
Posts: 21
Joined: Sat Jun 09, 2018 1:08 pm

Re: Finding AprilTags at a distance

Postby sexton » Fri Jun 22, 2018 5:37 pm

any ideas why QVGA didn't work?
User avatar
kwagyeman
Posts: 2461
Joined: Sun May 24, 2015 2:10 pm

Re: Finding AprilTags at a distance

Postby kwagyeman » Fri Jun 22, 2018 5:43 pm

Was it in grayscale or RGB565? Yes, I agree, there should be no problem with QVGA I'm grayscale. I'll look into this. I only tried set windowing with VGA when I was testing.
Nyamekye,
sexton
Posts: 21
Joined: Sat Jun 09, 2018 1:08 pm

Re: Finding AprilTags at a distance

Postby sexton » Fri Jun 22, 2018 6:02 pm

RGB565. Since I am only processing AprilTags I guess I should be using grayscale. I will try to redo some of my tests.
User avatar
kwagyeman
Posts: 2461
Joined: Sun May 24, 2015 2:10 pm

Re: Finding AprilTags at a distance

Postby kwagyeman » Fri Jun 22, 2018 7:32 pm

RGB565 images take double the space of a Grayscale image. Athe 200x200 res you're right at the RAM limit for the M7. (You may wish to actually handle memory exhaustion exceptions at this res).

With the H7 there's no problem with this.
Nyamekye,
sexton
Posts: 21
Joined: Sat Jun 09, 2018 1:08 pm

Re: Finding AprilTags at a distance

Postby sexton » Fri Jun 22, 2018 9:31 pm

I redid the tests but using GRAYSCALE AprilTag detection. .
1. Much better performance for VGA with 200x200 windowing.
2. QVGA with 200x200 windowing works.
3. VGA with 250x160 windowing works.
4. QVGA with 250x160 windowing does NOT work.
5. QVGA with 160x120 windowing has really good performance.
6. non-zoomed QQVGA performance seems slightly improved vs RGB565. Hard to say if firmware or just switching to grayscale.
User avatar
kwagyeman
Posts: 2461
Joined: Sun May 24, 2015 2:10 pm

Re: Finding AprilTags at a distance

Postby kwagyeman » Tue Jun 26, 2018 1:56 am

Cool, with the H7 you can hit 240x240 more or ess and with double the FPS. So, for a commercial product you may be okay.
Nyamekye,
sexton
Posts: 21
Joined: Sat Jun 09, 2018 1:08 pm

Re: Finding AprilTags at a distance

Postby sexton » Tue Jun 26, 2018 5:33 pm

We have the M7 working pretty well and look forward to trying the H7.
bsack
Posts: 1
Joined: Fri Jul 13, 2018 3:33 pm

Re: Finding AprilTags at a distance

Postby bsack » Fri Jul 13, 2018 3:36 pm

Hello!

I'm new to OpenMV so I apologize if this is the wrong place for this question, but I see that the program returns the x and y coordinates. Is it possible for Apriltags to return a z coordinate as well? (i.e. the distance between the camera and tag.)

Thanks!
Brady
User avatar
kwagyeman
Posts: 2461
Joined: Sun May 24, 2015 2:10 pm

Re: Finding AprilTags at a distance

Postby kwagyeman » Sat Jul 14, 2018 3:05 pm

Yes, it returns that as well. Please see the documentation: http://docs.openmv.io/library/omv.image ... e.apriltag
Nyamekye,

Return to “Technical Discussion”

Who is online

Users browsing this forum: No registered users and 6 guests