find_blobs returns boundry boxes within boundry boxes

Hi,

I’m currently using the find_blobs function with merge=True to find hot spots in a thermal image but in some circumstances it returns a boundary box within another boundary box inside. The temperature value displayed doesn’t match with the temperature value I send back to the raspberry pi to be stored in a database. Initially I thought setting merge=True would solve this but having looked at the documentation, is it the margin parameter that I need to optimize. More specifically should I be setting the margin parameter to a negative value to force it to merge boundary boxes that appear like this. The image I have attached shows the problem I’m having. If you look at the top and bottom lines of the boundary box you will notice they are quick thick. This is not some functionality I have implemented and I have checked this against other images I have taken where this does not happen. The boundary boxes are either slightly within one another or exactly on top of one another. The fact that multiple temperature strings are not shown is also puzzling.


I cannot debug this via the IDE as I am using the FLIR Lepton 3.5 on a raspberry pi 3B which I communicate with via the USB_VCP class. Could you point me to the source code for the find_blobs function if you are unsure what my issue might be caused by?

Thank you,
G

Just make the merge margin greater than 1. This will merge the blobs then. It’s quite weird for that to happen however since this would require a region of pixels in-between the blobs to fail being thresholded.

If you pass the thresholds to binary() you can see what the thresholded image looks like.

Blobs source is here: https://github.com/openmv/openmv/blob/master/src/omv/img/blob.c

Thanks for the reply, Yes I was a bit unsure how it happened myself although I thought some implementation of the margin parameter would fix it. The image below would probably have been a better illustration of the issue I was having.


.

I still don’t understand why the temperature is not being displayed for the inner box but I will check the source code and see if I can find anything.

Can you attach the code and the image without the blob detection, I can run it on the image offline and see what’s happening.

I am currently working on the device so I cannot attach an image yet as I will need to put it back in the server room to gather images. The code that generated that image are as follows:

import sensor, image, time, math, ustruct, pyb
from pyb import USB_VCP


# Color Tracking Thresholds (Grayscale Min, Grayscale Max)
min_rgb = 1
max_rgb = 255
threshold_list = [(min_rgb, max_rgb)]# only bright grey colours will get tracked.

# Set the target temp range here
min_temp_in_celsius = 15.0
max_temp_in_celsius = 95.0
pixel_thresh=200
area_thresh=200


# These settings are applied on reset
sensor.reset()
sensor.ioctl(sensor.IOCTL_LEPTON_SET_MEASUREMENT_MODE, True) #This command forces the lepton to return each pixel value as a temperature
sensor.ioctl(sensor.IOCTL_LEPTON_SET_MEASUREMENT_RANGE, min_temp_in_celsius, max_temp_in_celsius) #sets the temperature range in celcius for mapping
#image will then be clamped between these values and scaled to between 0  and 255

#print("Lepton Res (%dx%d)" % (sensor.ioctl(sensor.IOCTL_LEPTON_GET_WIDTH),sensor.ioctl(sensor.IOCTL_LEPTON_GET_HEIGHT)))
#print("Radiometry Available: " + ("Yes" if sensor.ioctl(sensor.IOCTL_LEPTON_GET_RADIOMETRY) else "No"))

sensor.set_pixformat(sensor.GRAYSCALE) #sets the number of bits per pixel, greyscale is 8
sensor.set_framesize(sensor.QQVGA) #sets the image frame size QQVGA is 160x120
sensor.skip_frames(time=5000) #camera is given 5000ms to stabilize
clock = time.clock()

usb = USB_VCP() #Create a USB virtual communication port object.


# Only blobs with more pixels than "pixel_threshold" and more area than "area_threshold" are
# returned by "find_blobs" below. Change "pixels_threshold" and "area_threshold" if you change the
# camera resolution. "merge=True" merges all overlapping blobs in the image.

def map_g_to_temp(g):
    #this function takes a scaled pixel value and converts it back to a temperature value
    return ((g * (max_temp_in_celsius - min_temp_in_celsius)) / 255.0) + min_temp_in_celsius

def get_temperatures(img):
    temperatures = []
    for blob in img.find_blobs(threshold_list, pixel_threshold=pixel_thresh, area_threshold=area_thresh, margin=5, merge=True):

        stats = img.get_statistics(thresholds=threshold_list, roi=blob.rect()) #get statistics for each blob where the blob is the roi.
        temperatures.append(map_g_to_temp(stats.mean()))

    return temperatures


def add_details(img):
    for blob in img.find_blobs(threshold_list, pixel_threshold=pixel_thresh, area_threshold=area_thresh, margin=5, merge=True):

        img.draw_rectangle(blob.rect()) #draws a rectangle (x, y, w, h) around blob for bounding box
        stats = img.get_statistics(thresholds=threshold_list, roi=blob.rect()) #get statistics for each blob where the blob is the roi.
        img.draw_string(blob.x(), blob.y() - 10, "%.2f C" % map_g_to_temp(stats.mean()), mono_space=False)

    return img
    

def send_temp(temperature_list):
    cmd = usb.recv(4, timeout=5000)
    if cmd == b'temp':
        if len(temperature_list) >=1:
            time.sleep(1)
            usb.send(str(max(temperature_list))) #need to send string to avoid object buffer protocol error
            
            
        else:
            usb.send('-100') #if no hotspot is detected return -100 as the temperature


def campture_img():
    img = sensor.snapshot() #take an image
    temperature_list = get_temperatures(img)
    img = add_details(img)

    #img.to_rainbow(color_palette=sensor.PALETTE_IRONBOW) #show hot areas as colours
   
    img = img.compress(quality=100) #compress the image keeping as much detail as possible
    usb.send(ustruct.pack('<L', img.size()))
    usb.send(img)
    pyb.LED(3).off()
    
    send_temp(temperature_list)


def configure(configuration):
    global min_rgb
    global max_rgb
    global threshold_list
    global min_temp_in_celsius
    global max_temp_in_celsius
    global pixel_thresh
    global area_thresh
    
    configs = configuration.split(':')[1]
    values = configs.split(',')
    
    min_rgb = int(values[0])
    max_rgb = int(values[1])
    
    threshold_list = [(min_rgb, max_rgb)]
    
    pixel_thresh = int(values[2])
    area_thresh = int(values[3])

    min_temp_in_celsius = int(values[4])
    max_temp_in_celsius = int(values[5])
    
    sensor.ioctl(sensor.IOCTL_LEPTON_SET_MEASUREMENT_RANGE, min_temp_in_celsius, max_temp_in_celsius)
    sensor.skip_frames(time=5000)

    usb.send(b'Configured') 
    pyb.LED(2).off()
    


while(True):
    clock.tick()
    data = usb.read() #Read all available bytes
    if data != None: #check that data is not a None object if the read function returned nothing
        #data = data.decode('utf-8')
        
        if len(data) >= 4: # NEED TO HANDLE AT Commands
            
            if data[:4] == b'snap':
                pyb.LED(3).on()
                campture_img()
                
                
                
            if data[:4] == b'conf':
                data = data.decode('utf-8')
                pyb.LED(2).on()
                configure(data)
                
            else:
                continue
            
        else:
            continue
        
    else:
        continue



I have since removed the get_temperatures function and incorporated it into the add_details function. Also is there a way to force synchronisation when communicating over usb_vcp? I was having another issue where on occasion the maximum temperature returned by the get_temperatures function did not match the maximum temperature denoted on any of the bounding boxes. It was like it had gone out of sync but was not generating an error make it quite difficult to debug.

1 Like

No you’ll have to implement that somehow. When you attach the image I’ll run the same blob detection and drawing code and see if this issue can be reproduced.