Run TFLite on (public) training images in post

darrask · December 31, 2021, 5:21am

Directly on the H7+ board, I am trying to run TFLite models on their own training images (obtained from public image datasets) that were were used to build these models on EdgeImpulse, for diagnostics and cross-checks. I use the 4.1.5 development firmware. However, with many images it fails.

For instance, when loading these attached JPEGs:
image_03.zip (133.9 KB)

…the board either disconnects and reconnects, or starts flashing the RGB LED in quick succession and logs a MemManage error:
VID_20211231_120600.zip (150.3 KB)

This is my code:

#import libraries
import image, os, tf, pyb
#set constants
GREEN_LED_PIN = 2
#import mobilenet model and labels
mobilenet = "trained.tflite"
labels = [line.rstrip('\n') for line in open("labels.txt")]

#scan jpegs on card
files=os.listdir()
jpegs=[files for files in files if "jpg" in files]

# create detection file and "out" folder
if(not 'detections.csv' in os.listdir()):
        with open('detections.csv', 'a') as detectionlog:
            detectionlog.write("picture" +  ',' + "label1" + ',' "confidence1" + ',' + "x_top_left" + ',' + "y_top_left" + ',' + "width" + ',' + "height" +  ',' + "label2" + ',' "confidence2" + '\n')
#if not "out" in os.listdir(): os.mkdir("out")

#open and classify each jpeg
for jpeg in jpegs:
    print("Loading:",jpeg)
    img=image.Image(jpeg,copy_to_fb=True)
    img.to_rgb565()
    #starting classification
    pyb.LED(GREEN_LED_PIN).on()
    print("LED on: classifying image", jpeg, "with tensorflow lite...")
    for obj in tf.classify(mobilenet, img, min_scale=1, scale_mul=0.5, x_overlap=0.5, y_overlap=0.5):
        predictions_list = list(zip(labels, obj.output()))
        print(predictions_list)
        with open('detections.csv', 'a') as detectionlog:
            detectionlog.write(str(jpeg) + ',' + str(predictions_list[1][0]) + ',' + str(predictions_list[1][1]) + ',' + str(obj.rect()[0]) + ',' + str(obj.rect()[1]) + ',' + str(obj.rect()[2]) + ',' + str(obj.rect()[3]) + ',' + str(predictions_list[0][0]) + ',' + str(predictions_list[0][1]) + '\n')
    pyb.LED(GREEN_LED_PIN).off()

kwagyeman · December 31, 2021, 5:22am

Since loading images produced by the board works fine I suppose its and issues in Larry’s jpeg code.

kwagyeman · December 31, 2021, 5:23am

Larry’s been alerted.

darrask · December 31, 2021, 5:26am

Good. Note that with the newest firmware, the board seems to be able to produce TFLite confidence scores slightly above 1 (such as 1.00391) with our model (that previously didn’t).

kwagyeman · December 31, 2021, 7:36am

Some of the scaling issues were fixed in the TF processing code. I need more guidance to fix all of them from Edge Impulse.

kurileo · January 4, 2022, 7:50am

Hi @kwagyeman ,

To locate the bug, I have a question that whether there’s a custom implementation of dialect for MLIR on OpenMV’s runtime for tflite for microcontroller? As the model from Edge Impluse shows that it was applied MLIR optimization.

Thanks in advance!

Leo

kwagyeman · January 4, 2022, 5:26pm

Hi, oh, you are saying the confidence scores above 1 are a bug…

They may not be. We are now scaling the output as TensorFlow expects us to do. Before we were not.

However, I need edge impulse folks to tell me what to do as the way TensorFlow does scaling doesn’t make sense always.

Unless the output was softmaxed there’s no reason it should be less than 1.

darrask · January 18, 2022, 1:40am

Hello, this is the current status with me trying to cross-validate the EdgeImpulse model results by classifying RGB888 images (sourced from public datasets, used for training the model on EI) directly on the board with TFLite.
Even though the 4.2.1 firmware addresses " * Fix all issues with ImageIO to support all modes and older files.", I still have the same problem: with many images (~ 50%) used in my training dataset, the board will crash.

Depending on the particular picture, there are two possible types of crash:

board hangs up (unresponsive) with a rapidly blinking whitish LED, saves a MemManage fatal error log on the card. Only pulling the USB cable can help.
it stops during script execution, disconnects and then reconnects by itself. Does not save a MemManage fatal error log on the card.

Loading these public images (not images taken with the board) always works fine with my images.

However, for those images that are not supported, I tried different combinations and it appears the line img.to_rgb565 will crash the board, and running the TFLite line will also crash the board. I cannot catch the error either with a generic try/except.

Currently, the only solution for me is a very manual one: running the script until it crashes, manually removing the images that don’ t work, and repeating many times just to test a few dozen images.

As a first step, it would be good that the img.to_rgb565 conversion works with any image. That way, we could train better models because we believe that the color depth mismatch between RGB888 training images and RGB565 images captured by the board leads to low performance.

kwagyeman · January 18, 2022, 5:49am

Hi, Previous files you created with ImageIO during the time it was broken are corrupted. Additionally, the IDE has not been updated to handle the new file types.

I don’t have a solution for previous files that were created while the library was bugged.

Can you re record?

darrask · January 18, 2022, 6:33am

Sorry, maybe I was unclear.
I meant that almost half of the images from a public dataset (Oxford 102 flowers) cannot be converted to RGB565 and/or analysed with TFLite on the H7+ board.

The images taken by the board can be classified just fine (and don’t need conversion).

kwagyeman · January 18, 2022, 7:24pm

Ah, this is related to the JPEGs being corrupted on load. Yes, I haven’t heard back from Larry yet on this. Let me ping him.

darrask · January 21, 2022, 6:57am

Hi, any way I can skip this error for now? I do not understand why my try-except calls are not jumping these errors:

for jpeg in jpegs:
    print("Loading:",jpeg)
    img=image.Image(jpeg,copy_to_fb=True)
    print("Loaded:",jpeg)
    try:
        print("Converting to RGB565:",jpeg)
        img.to_rgb565()
    except:
        print("Error with conversion to RGB565")
    print("Converted to RGB565:",jpeg)
    #starting classification
    pyb.LED(GREEN_LED_PIN).on()
    print("LED on: classifying image", jpeg,"\n**************")
    try:
        for obj in tf.classify(mobilenet, img, min_scale=1, scale_mul=0.5, x_overlap=0.5, y_overlap=0.5):
            predictions_list = list(zip(labels, obj.output()))
            print(predictions_list)
    except:
        print("Error with TFLite execution")
    pyb.LED(GREEN_LED_PIN).off()

darrask · January 21, 2022, 8:32am

It’s weird because all of the images that are loaded and you say get corrupted display just fine in the viewer.
I can also convert them to_bitmap() just fine (to_grayscale results in blocks at left border though). But to_rgb565 fails.

kwagyeman · January 21, 2022, 6:13pm

It’s because it’s a C level error in the decoder corrupting system state.

We are working on fixing this right now.

darrask · January 24, 2022, 5:15am

I see. Is there any intermediary firmware you’d like me to test?

darrask · February 17, 2022, 5:37am

Hi @kwagyeman , any updates on this?

kwagyeman · February 17, 2022, 7:39pm

Yes, I think we just fixed the last bug. There will be another firmware release. We just did one which fixed the crashing issue. However, then there was a scaling issues. We now have that fixed too.

darrask · February 18, 2022, 12:40am

excellent, thanks! Let me know if I should test it with an intermediary firmware.

kwagyeman · February 20, 2022, 12:31pm

The latest release is fixed.

darrask · February 21, 2022, 1:59am

With firmware 4.2.3, the same bug still occurs. By the way, I had to manually install it as my IDE isn’t updating by itself.