multi coies of an image

Hi there I am new to the OpenMV cam but have done a couple of projects with OpenCV on RPi.

First of all a big thanks to the creators of this project for all the work they have done to create such an awesome product.

I have a OpenMV M7 cam and I want to be able to take a RBG565 image and take a copy of it so that I can turn the copy into a binary image and find what I am looking for then draw a circle around it on the original RGB565 image.

Here is a bit of an example of me doing it on the RPi with OpenCV see at 0:40 Mactwist Rubik's cube Solver - YouTube

I made a few copies of the image then inverse binary thresholds of each copy different to make different colours black then used binary and to combine all the binary images then used find rectangle. Then rejected and rectangles too big or too small then rejected any rectangles that didn’t have other close rectangles to them.

Hi, you can do this via the alloc_fb method. As we’re a microcontroller we really don’t have a lot of RAM onboard… so, things are slightly weird. But, I’ve been extending the functional of the system to meet customer requests. So, please excuse the limitations. Anyway, checkout the in memory frame differencing script:

# In Memory Basic Frame Differencing Example
#
# This example demonstrates using frame differencing with your OpenMV Cam. It's
# called basic frame differencing because there's no background image update.
# So, as time passes the background image may change resulting in issues.

import sensor, image, pyb, os, time

TRIGGER_THRESHOLD = 5

sensor.reset() # Initialize the camera sensor.
sensor.set_pixformat(sensor.RGB565) # or sensor.GRAYSCALE
sensor.set_framesize(sensor.QVGA) # or sensor.QQVGA (or others)
sensor.skip_frames(time = 2000) # Let new settings take affect.
sensor.set_auto_whitebal(False) # Turn off white balance.
clock = time.clock() # Tracks FPS.

# Take from the main frame buffer's RAM to allocate a second frame buffer.
# There's a lot more RAM in the frame buffer than in the MicroPython heap.
# However, after doing this you have a lot less RAM for some algorithms...
# So, be aware that it's a lot easier to get out of RAM issues now. However,
# frame differencing doesn't use a lot of the extra space in the frame buffer.
# But, things like AprilTags do and won't work if you do this...
extra_fb = sensor.alloc_extra_fb(sensor.width(), sensor.height(), sensor.RGB565)

print("About to save background image...")
sensor.skip_frames(time = 2000) # Give the user time to get ready.
extra_fb.replace(sensor.snapshot())
print("Saved background image - Now frame differencing!")

while(True):
    clock.tick() # Track elapsed milliseconds between snapshots().
    img = sensor.snapshot() # Take a picture and return the image.

    # Replace the image with the "abs(NEW-OLD)" frame difference.
    img.difference(extra_fb)

    hist = img.get_histogram()
    # This code below works by comparing the 99th percentile value (e.g. the
    # non-outlier max value against the 90th percentile value (e.g. a non-max
    # value. The difference between the two values will grow as the difference
    # image seems more pixels change.
    diff = hist.get_percentile(0.99).l_value() - hist.get_percentile(0.90).l_value()
    triggered = diff > TRIGGER_THRESHOLD

    print(clock.fps(), triggered) # Note: Your OpenMV Cam runs about half as fast while
    # connected to your computer. The FPS should increase once disconnected.

You can also find this under examples → frame differencing. Basically, this shows you how to alloc another frame buffer and then copy images over to it. Note that you’ll want to reduce the res to fit both images. Also, I don’t believe the code for snapshot() right now checks to see if it’s overwriting the alloced fb (this is a todo to fix). So, if things randomly crash reduce the res. That said, the alloced fb is part of the same data structure used by our imaging methods, so, if you try to use a method that needs too much ram you’ll get an error in that case. Just not if snapshot() overwrites the alloced fb.

http://docs.openmv.io/library/omv.sensor.html?highlight=alloc_ex#sensor.sensor.alloc_extra_fb

OK that for your reply. I have started to play with Micropython on the ESP8266 and now ESP32 with psRAM so have started to get used to the confined RAM of MicroPython compared to RPi.

The ESP32 has an option for psRAM (i use a 4mb psRAM board myself) and this allows for image manipulation that isn’t possible on ESP32 board without the psRAM not sure if this is an option you can look into for future generations of your cam.

In the end the copies of the image I need to turn into binary images anyway. At the moment I have to make a RGB565 copy then use a method on the copy to turn it into a binary image, is it possible to add that I can make a binary copy as this will reduce the needed memory to 1/16th.

Maybe something like this

red_threshold = (0,255,   10,127,   0,100) # L A B
img = sensor.snapshot()
img2 = img.binary([red_threshold])

or maybe like this

red_threshold = (0,255,   10,127,   0,100) # L A B
img = sensor.snapshot()
img2 = binary_copy(img, [red_threshold])

OK I have used sensor.alloc_extra_fb to create a copy of the image for processing see below code.

I commonly use thresholding binary images to find what I am looking for in images for my robotics projects. I am quite often looking for a number of things in the image so want to create a number of binary images. e.g this cube solving robot needs to find all the coloured tiles on cube face so I use 4 threshold binary images (high brightness to find white and yellow, high red to find orange and red, high blue and dark to find blue and high green to find green tiles) then I combine all the images together using or and look for squares and I will find tiles on the face then based upon these found squares I can read the colour from the original image. To make 4 binary images by creating RGB565 buffers first will require lots of RAM but to create 4 binary images requires little RAM.

Here is the simple OpenMV script that I got to work that will only find red and orange squres on Rubix Cube face.

import sensor, image, time

sensor.reset()
sensor.set_pixformat(sensor.RGB565) # grayscale is faster (160x120 max on OpenMV-M7)
sensor.set_framesize(sensor.QQVGA)
sensor.skip_frames(time = 2000)
clock = time.clock()
extra_fb = sensor.alloc_extra_fb(sensor.width(), sensor.height(), sensor.RGB565)
red_threshold = (0,255,   10,127,   0,100) # L A B

while(True):
    clock.tick()

    img = sensor.snapshot()
    extra_fb.replace(img)
    extra_fb.binary([red_threshold])
    extra_fb.erode(1)

    for r in extra_fb.find_rects(threshold = 5000):
        img.draw_circle(r.x()+r.w()//2, r.y()+r.h()//2, r.w()//2, color = (0, 255, 0))

    print("FPS %f" % clock.fps())

Hi, we actually have binary image support coming very soon. I have been silently adding it to almost every method. Binary images take up 1/8th the space of grayscale images. However, I’m not done adding to every method so I didn’t document it. That said… there’s a secret .to_bitmap(copy=False) method that will convert an image to a binary image for use with almost every method that accepts a mask input.

https://github.com/openmv/openmv/blob/master/src/omv/py/py_image.c#L743

Pass copy=True to get a copy created on the heap. Or, copy=False to bitmap an image in place. Note that you need to binary() the image first followed by to_bitmap() to shrink the image.

This feature is in v2.9.0. But, not documented… so, you may find bugs.

Note - jpeg support is not implemented for the bitmap images. This is why I didn’t document them. So, do not try to view the binary images. Neither is file saving support.

… so, yeah, that kinda makes them useless right now. However, you can view the affect of applying one as a mask to any method that takes a mask= parameter.

I have just had a quick play with .to_bitmap(copy=True) but am unable to understand the correct syntax to use it.

Seeing my short script in earlier post how would I go about implementing .to_bitmap(copy=True) to achieve the same outcome?

Like so:

It’s going to be kinda hard to debug though since you can’t view the image after “.to_bitmap(copy=True)” is called. But, it’s valid to look at before that.

Note that the image is allocated on the heap when copy=True is called. So, as long as you have micropython heap you call allocate as many as you like. At QQVGA each image is 160*120/8 = 2.4KB

The “.to_bitmap(copy=True)” copies the image from the frame buffer to the heap. So, if you’d like to view the image just keep calling everything up to that method in a loop.

import sensor, image, time

sensor.reset()
sensor.set_pixformat(sensor.RGB565) # grayscale is faster (160x120 max on OpenMV-M7)
sensor.set_framesize(sensor.QQVGA)
sensor.skip_frames(time = 2000)
clock = time.clock()

red_threshold = (0,255,   10,127,   0,100) # L A B

mask = sensor.snapshot().binary([red_threshold]).erode(1).to_bitmap(copy=True)

while(True):
    clock.tick()

    img = sensor.snapshot()

    for r in mask.find_rects(threshold = 5000):
        img.draw_circle(r.x()+r.w()//2, r.y()+r.h()//2, r.w()//2, color = (0, 255, 0))

    print("FPS %f" % clock.fps())

Ok the adding the .to_bitmap(copy=True) to the end doesn’t do what I need because as well as making a copy it still alters the original image.

This is what I want to do: have the original RGB565 image then make binary copies of the image with different thresholds but leave the original RGB565 image unchanged. i.e not use a method on the original image but rather use a function that returns a new binary image.

The following script of course won’t work but you can see what I want to do.

import sensor, image, time, mjpeg

sensor.reset()
sensor.set_pixformat(sensor.RGB565) # grayscale is faster (160x120 max on OpenMV-M7)
sensor.set_framesize(sensor.QQVGA)
sensor.skip_frames(time = 2000)
clock = time.clock()

red_threshold = (17, 70, 29, 73, -25, 50) # L A B

#add other needed threshods here

while True:
    clock.tick()
    img = sensor.snapshot()
    mask_red_oragne = img.binary([red_threshold]).erode(1).to_bitmap(copy=True)
    mask_white_yellow = img.binary([light_threshold]).erode(1).to_bitmap(copy=True)
    mask_blue = img.binary([green_threshold]).erode(1).to_bitmap(copy=True)
    mask_green = img.binary([blue_threshold]).erode(1).to_bitmap(copy=True)
    

    combined_mask  = mask_red_oragne.b_or(mask_white_yellow).b_or(mask_blue).b_or(mask_green)    

    for r in combined_mask.find_rects(threshold = 100):
      if r.w()<25:
        img.draw_circle(r.x()+r.w()//2, r.y()+r.h()//2, r.w()//2, color = (0, 255, 0))
      
    print("FPS %f" % clock.fps())

I see. Then you need to allocate a new frame buffer. Copy the image to that and then binarize it and then convert it to a bitmap per image you want to work on.

As in, do replace, then binary, then errode, then to bitmap for every image you want to make.

You can deallocate the extra FB afterwards to free up stack ram.

I see. Then you need to allocate a new frame buffer. Copy the image to that and then binarize it and then convert it to a bitmap per image you want to work on.

Yes this is the only way it can be done with current firmware. The trouble with doing it like this is it require a lot of memory because I first have to allocate RGB565 frame buffer then binarize it. So I have to use 16 times more memory than is needed. If I want to create 4 binary copies with different threshold I need to first make 4 RGB565 frame buffers. Having to create a RGB565 FB to store a binary image is very inefficient.

At the moment you can only binarize an image and write it to the same FB the original image was stored in. I want to be able to perform binary threshold without destroying the original image.

I assume it would not be a huge amount change to your current firmware to be able add this feature as you already have all the code to do the threshold processing just to chnage that it is returned as a new binary FB rather than written over the top of the RGB565 FB.

Here is a short video of it in action just using 1 binary copy with just red threasholding see - YouTube

and here is the code that was running to make that video

import sensor, image, time, mjpeg

sensor.reset()
sensor.set_pixformat(sensor.RGB565) # grayscale is faster (160x120 max on OpenMV-M7)
sensor.set_framesize(sensor.QQVGA)
sensor.skip_frames(time = 2000)
clock = time.clock()
extra_fb = sensor.alloc_extra_fb(sensor.width(), sensor.height(), sensor.RGB565)
red_threshold = (17, 70, 29, 73, -25, 50) # L A B

while True:
    clock.tick()
    img = sensor.snapshot()
    extra_fb.replace(img)
    extra_fb.binary([red_threshold])
    extra_fb.erode(1)

    for r in extra_fb.find_rects(threshold = 100):
      if r.w()<25:
        img.draw_circle(r.x()+r.w()//2, r.y()+r.h()//2, r.w()//2, color = (0, 255, 0))
    print("FPS %f" % clock.fps())

Hi, you only need 1 extra frame buffer as far as I understand. You just copy the source image to that buffer. Binarize it. Then convert it to a bitmap. Then copy the source in back in there again and do the process again for the next color threshold.

That said, yes, agree this is not optimal. So, I can make to_bitmap() do this for you. It’s not hard to make to act like binary. However, I have quite a few time demands on me and I’m not exactly sure when I can get this done. Is just using one extra FB and repeatedly copying into it not enough to for now? You should have enough memory for this at QVGA RGB565.

Hi, you only need 1 extra frame buffer as far as I understand. You just copy the source image to that buffer. Binarize it. Then convert it to a bitmap. Then copy the source in back in there again and do the process again for the next color threshold.

Ahh yes I never though fo doing that :slight_smile:

That said, yes, agree this is not optimal. So, I can make to_bitmap() do this for you. It’s not hard to make to act like binary. However, I have quite a few time demands on me and I’m not exactly sure when I can get this done. Is just using one extra FB and repeatedly copying into it not enough to for now?

I will use repeated coping to RGB565 FB for the moment as it will get the job done that I need and I will wait till when you have to time to add the more efficient way :slight_smile:

Thanks for all your hard work and support on your product and I look forward to the release of the OpenMV H7 :slight_smile:

It seems that coping the RGB565 image to new fb then doing binary threashold then to_bitmap and make another copy lowers the fps a lot.

    extra_fb.replace(img)
    red = extra_fb.binary([red_threshold]).erode(1).to_bitmap(copy=True)

The above code does many steps:

  1. makes a copy of the RGB565 fb
  2. does binary thresholding
  3. converts this binary back to a B&W RGB565 then writes it over the original RGB565 fb
  4. performs erode on a B%W RGB565
  5. converts it back to a binary fb
  6. copies the binary fb to a new binary fb

All this can be done in 2 steps:

1.binary threshold the RGB565 and store the otuput in new binary fb
2. perform erode on a binary fb


For me this is what doesn’t make sense

new_binary_fb = img.binary([threshold]).to_bitmap(copy=True)

This line of code does binary thresholding to create a binary image then it converts the binary image back to a RGB565 fb then it converts the RGB565 fb back to a binary fb then copies it.

I can’t understand why we would ever want to convert binary images back to RGB565 fb. Something like this would make much more sense to me

new_binary_fb = img.binary([threshold], new_fb=True)

When new_fb is set to True then it does not go the long way around it just does what would normally be needed a new binary fb is made to hold the new binary image created and the original RGB565 is left unchanged as it is a RGB565 fb not a binary fb

Hi, binay image functionality has not really been implemented yet. I’ve been adding it but it isn’t really fully there yet. Only RGB565 and Grayscale have full support.

I will add two parametes to the method to do what you want.

Thanks a lot.

You guys obviously work very hard to provide this amazing support for your users.

Okay, this firmware implements jpeg compression for bitmaps and adds the feature you wanted to binary.

Binary now has a to_bitmap key word arg that when true turns the image to a bitmap. Note that you have to pass copy also at the same time to copy the image somewhere else. The method will not allow you to run in place.

Load either file (any will work) via run bootloader. The firmware.bin is the easier one to load.
firmware.zip (1.74 MB)

Thank you very much. I will sit down tonight and have a play with it :slight_smile:

I have uploaded new firmware :slight_smile:

I don’t quite understand the syntax for using this.

I tried this but it got an error OS can’t convert bitmap in place

new_binary = img.binary([white_threshold], to_bitmap=True)