Run TFLite on (public) training images in post

OpenMV Boards
1

Directly on the H7+ board, I am trying to run TFLite models on their own training images (obtained from public image datasets) that were were used to build these models on EdgeImpulse, for diagnostics and cross-checks. I use the 4.1.5 development firmware. However, with many images it fails.

For instance, when loading these attached JPEGs:
image_03.zip (133.9 KB)

…the board either disconnects and reconnects, or starts flashing the RGB LED in quick succession and logs a MemManage error:
VID_20211231_120600.zip (150.3 KB)

This is my code:

#import libraries
import image, os, tf, pyb
#set constants
GREEN_LED_PIN = 2
#import mobilenet model and labels
mobilenet = "trained.tflite"
labels = [line.rstrip('\n') for line in open("labels.txt")]

#scan jpegs on card
files=os.listdir()
jpegs=[files for files in files if "jpg" in files]

# create detection file and "out" folder
if(not 'detections.csv' in os.listdir()):
        with open('detections.csv', 'a') as detectionlog:
            detectionlog.write("picture" +  ',' + "label1" + ',' "confidence1" + ',' + "x_top_left" + ',' + "y_top_left" + ',' + "width" + ',' + "height" +  ',' + "label2" + ',' "confidence2" + '\n')
#if not "out" in os.listdir(): os.mkdir("out")

#open and classify each jpeg
for jpeg in jpegs:
    print("Loading:",jpeg)
    img=image.Image(jpeg,copy_to_fb=True)
    img.to_rgb565()
    #starting classification
    pyb.LED(GREEN_LED_PIN).on()
    print("LED on: classifying image", jpeg, "with tensorflow lite...")
    for obj in tf.classify(mobilenet, img, min_scale=1, scale_mul=0.5, x_overlap=0.5, y_overlap=0.5):
        predictions_list = list(zip(labels, obj.output()))
        print(predictions_list)
        with open('detections.csv', 'a') as detectionlog:
            detectionlog.write(str(jpeg) + ',' + str(predictions_list[1][0]) + ',' + str(predictions_list[1][1]) + ',' + str(obj.rect()[0]) + ',' + str(obj.rect()[1]) + ',' + str(obj.rect()[2]) + ',' + str(obj.rect()[3]) + ',' + str(predictions_list[0][0]) + ',' + str(predictions_list[0][1]) + '\n')
    pyb.LED(GREEN_LED_PIN).off()
2

Since loading images produced by the board works fine I suppose its and issues in Larry’s jpeg code.

3

Larry’s been alerted.

4

Good. Note that with the newest firmware, the board seems to be able to produce TFLite confidence scores slightly above 1 (such as 1.00391) with our model (that previously didn’t).

6

Some of the scaling issues were fixed in the TF processing code. I need more guidance to fix all of them from Edge Impulse.

7

Hi @kwagyeman ,

To locate the bug, I have a question that whether there’s a custom implementation of dialect for MLIR on OpenMV’s runtime for tflite for microcontroller? As the model from Edge Impluse shows that it was applied MLIR optimization.

image

Thanks in advance!

Leo

8

Hi, oh, you are saying the confidence scores above 1 are a bug…

They may not be. We are now scaling the output as TensorFlow expects us to do. Before we were not.

However, I need edge impulse folks to tell me what to do as the way TensorFlow does scaling doesn’t make sense always.

Unless the output was softmaxed there’s no reason it should be less than 1.

9

Hello, this is the current status with me trying to cross-validate the EdgeImpulse model results by classifying RGB888 images (sourced from public datasets, used for training the model on EI) directly on the board with TFLite.
Even though the 4.2.1 firmware addresses " * Fix all issues with ImageIO to support all modes and older files.", I still have the same problem: with many images (~ 50%) used in my training dataset, the board will crash.

Depending on the particular picture, there are two possible types of crash:

  1. board hangs up (unresponsive) with a rapidly blinking whitish LED, saves a MemManage fatal error log on the card. Only pulling the USB cable can help.
  2. it stops during script execution, disconnects and then reconnects by itself. Does not save a MemManage fatal error log on the card.

Loading these public images (not images taken with the board) always works fine with my images.

However, for those images that are not supported, I tried different combinations and it appears the line img.to_rgb565 will crash the board, and running the TFLite line will also crash the board. I cannot catch the error either with a generic try/except.

Currently, the only solution for me is a very manual one: running the script until it crashes, manually removing the images that don’ t work, and repeating many times just to test a few dozen images.

As a first step, it would be good that the img.to_rgb565 conversion works with any image. That way, we could train better models because we believe that the color depth mismatch between RGB888 training images and RGB565 images captured by the board leads to low performance.

10

Hi, Previous files you created with ImageIO during the time it was broken are corrupted. Additionally, the IDE has not been updated to handle the new file types.

I don’t have a solution for previous files that were created while the library was bugged.

Can you re record?

11

Sorry, maybe I was unclear.
I meant that almost half of the images from a public dataset (Oxford 102 flowers) cannot be converted to RGB565 and/or analysed with TFLite on the H7+ board.

The images taken by the board can be classified just fine (and don’t need conversion).

12

Ah, this is related to the JPEGs being corrupted on load. Yes, I haven’t heard back from Larry yet on this. Let me ping him.

13

Hi, any way I can skip this error for now? I do not understand why my try-except calls are not jumping these errors:

for jpeg in jpegs:
    print("Loading:",jpeg)
    img=image.Image(jpeg,copy_to_fb=True)
    print("Loaded:",jpeg)
    try:
        print("Converting to RGB565:",jpeg)
        img.to_rgb565()
    except:
        print("Error with conversion to RGB565")
    print("Converted to RGB565:",jpeg)
    #starting classification
    pyb.LED(GREEN_LED_PIN).on()
    print("LED on: classifying image", jpeg,"\n**************")
    try:
        for obj in tf.classify(mobilenet, img, min_scale=1, scale_mul=0.5, x_overlap=0.5, y_overlap=0.5):
            predictions_list = list(zip(labels, obj.output()))
            print(predictions_list)
    except:
        print("Error with TFLite execution")
    pyb.LED(GREEN_LED_PIN).off()
14

It’s weird because all of the images that are loaded and you say get corrupted display just fine in the viewer.
I can also convert them to_bitmap() just fine (to_grayscale results in blocks at left border though). But to_rgb565 fails.

15

It’s because it’s a C level error in the decoder corrupting system state.

We are working on fixing this right now.

1 Like
16

I see. Is there any intermediary firmware you’d like me to test?

17

Hi @kwagyeman , any updates on this?

18

Yes, I think we just fixed the last bug. There will be another firmware release. We just did one which fixed the crashing issue. However, then there was a scaling issues. We now have that fixed too.

19

excellent, thanks! Let me know if I should test it with an intermediary firmware.

20

The latest release is fixed.

21

With firmware 4.2.3, the same bug still occurs. By the way, I had to manually install it as my IDE isn’t updating by itself.