Your code is completely wrong… Please read what the network outputs. I will post an update with changes to your code in a it.
Hi,
Yes, I had developed that code based on the network output I was getting.
For e.g. printing just the “out” variable in above code gets the following output like:
[{“x”:0, “y”:0, “w”:128, “h”:128, “output”:[0.5294118, 0.8745098, 0.627451, 0.5803922, 0.5254902, 0.682353, 0.5372549, 0.6352942, 0.5058824, 0.5294118]}]
Code with outputs is attached below
But sure, I’ll be happy to learn from your updated code.
Thanks in advance.
Regards,
IM
Hi, another user brought an issue up and the problem is likely related to the fact that I don’t do uint8_t->int8_t input/output conversion. Right now the image is fed to the net as a uint8_t image.
Can you tell me what type of image data you trained on and what you expect the output to be? I.e is the image input -128 to 127? Or 0 to 127? And is the output -128 to 127? or 0 to 127? I have to fix this in the C code for things to work.
As for your code being wrong:
- out = net.classify(img.copy().binary([( 0, 1)], invert=True))
score = max(out[0][4]) * 100
max_idx = out[0][4].index(max(out[0][4]))
if (score < 50):
score_str = “??:??%”
else:
score_str = "%s:%d%% "%(labels[max_idx], score)
img.draw_string(0, 0, score_str)
I was just thrown off by this. I think this is from some other code older code. The tensorflow example doesn’t show how to print class labels this way.
Hi,
I trained on the MNIST images(uint8) , then quantized it with full integer quantization, making the inputs and outputs as int8 type ( screenshot attached).
Okay, I’ll make a version of the firmware that fixes the offset and post it tonight. This should fix the problem.
The firmware will check the network input type and then fix the data signedness automatically.
…
That said, it would be helpful to know the range if possible. I’m going to assume I just need to subtract 128 on the input and add 128 on the output. I hope that is correct.
Hi,
The scikit-image website says the range of int8 is -128 to 127.
However, I think the your approach should work. Similar thing is suggested in this comment too: Quantization problem while reproducing person detection example · Issue #37347 · tensorflow/tensorflow · GitHub
Hi, try this binary out. It will subtract 128 from the input unsigned image data if the data is signed and add 128 to the output if signed to make it unsigned again. The input and output are done independently. So, you are free to mix and match.
I’ve verified that our unsigned person detector network works.
…
Please let me know if this works and then I will send in the branch to be merged.
firmware.zip (1.16 MB)
Hi,
Thanks for providing the binary. I gave it a try but the same situation still continues. The outputs are still following the same pattern as I mentioned earlier.
the output is generating same result (say any particular number for e.g. ‘five’) no matter what the feed is…
So basically, the output changes with change of model files but is constant for a particular model with a slight variation in scores here & there.
I am attaching the full integer quantized model with this.
mnistconv_int8.zip (62.1 KB)
Okay, I need this from you. Can you provide me the final model output and the training data set that you used for it. If I can load these images onto the camera via the SD card then I can actually debug this and get it working.
I won’t be able to debug your model unless I know what type of input it needs.
Okay, Here’s is a Jupyter notebook which trains the model and performs full integer quantization.
Open it up with Google Colaboratory and run in the “Playground” mode after that.
Running the entire notebook (by clicking on “Runtime” → “Run all” option in Colab ) will get you the Final Model (“.tflite”) output file which you can easily download from the “Files” section on left side of Colab.
The dataset is the inbuilt MNIST dataset that Tensorflow/Keras provide.
Hey,
Just circling back to see if there are any updates on this?
Regards,
IM
I’m focused on the interface library right now. Since the simple fix didn’t work this has to become a project for me now.
Question, did you try binarizing the image (using actual thresholds - not just calling to_bitmap()) and feed that to the net? Previously, on our old CNN system you had to binarize the image and then invert the image to get things to work.
I.e. the network needs the background as white and the characters black. So, this requires and inverted binarize operation.
Ahh alright, then it might take a while I assume?
And yes I did try the image with White Background & Black characters (& vice versa).
I feel maybe it’s not just about the input images or the dataset trained on because as soon as the video begins the same result [a particular number(class) ] starts getting displayed and the result never changes no matter what the input is.
I hope you all are safe
Wishing you good health,
IM
Hi, I’ll be able to start working on this next week. I have to finish the latest firmware release documentation and get that out to everyone.
We have a goal of integration with Edge Impulse soon for automatic deep learning training so everything with TensorFlow needs to get working for the release coming next month after the one we are about to do.
Hi, I’ve updated the firmware to support int8/uint8/float32.
The model still needs to be int8/uint8 quantized internally, but, float32 is supported as the input/output layer.
It was tested on Mobilenet (uint8), the person detector (uint8), another person detector (int8), and then a flower model (float32 with int8 internally).
This file is for the H7 Plus. Let me make one for the H7.
firmware.zip (1.18 MB)
Regarding your model. Please try a dataset where the input data is a series of color images and verify that your model works okay before trying mnist.
We are getting things ready for Edge Impulse support. So, we expect TensorFlow support to be working smoothly moving forwards.
Hi,
I updated the H7 firmware that kwagyeman released on Fri May 15, 2020.
I follow the steps to prepare the tensorflow lite model for mnist.
partial of training code as below.
I modify some codes for channel dimention
mnist = keras.datasets.mnist
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()
# Normalize the input image so that each pixel value is between 0 to 1.
train_images = train_images / 255.0
test_images = test_images / 255.0
train_images = np.expand_dims(train_images, axis=3)
test_images = np.expand_dims(test_images, axis=3)
# Define the model architecture
model = tf.keras.Sequential([
tf.keras.layers.InputLayer(input_shape=(28, 28, 1)),
tf.keras.layers.Conv2D(filters=12, kernel_size=(3, 3), activation=tf.nn.relu),
tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
.......................................
Everything is fine on my PC.
I try these models on openmv h7.
What could i set image range? 0~255 or 0~1
import sensor, image, time, os, tf
sensor.reset() # Reset and initialize the sensor.
sensor.set_contrast(3)
sensor.set_pixformat(sensor.GRAYSCALE) # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA) # Set frame size to QVGA (320x240)
sensor.set_windowing((28, 28)) # Set 128x128 window.
sensor.skip_frames(time=100)
sensor.set_auto_gain(False)
sensor.set_auto_exposure(False)
#net = tf.load('/mnist_model_quant.tflite')
net = tf.load('/mnist_model_quant_io.tflite')
clock = time.clock()
while (True):
clock.tick()
img = sensor.snapshot()
out = net.classify(img.copy().binary([(0, 1)], invert=True))#?????????
print(out)
Thanks.
Should be 0-255. However, you need to match the color of the binary data with MNIST. So, if it’s black on white then 0 and 255. If it’s white on black then 255 and 0.
Thanks for your reply.
test code on my PC
img = np.ones((28,28,1),dtype=np.float)
white_img = img*255
test_image = np.expand_dims(white_img, axis=0).astype(np.float32)
tflite_model_quant_file = tflite_models_dir/"mnist_model_quant_io.tflite"
interpreter_quant = tf.lite.Interpreter(model_path=str(tflite_model_quant_file))
interpreter_quant.allocate_tensors()
input_index_quant = interpreter_quant.get_input_details()[0]["index"]
output_index_quant = interpreter_quant.get_output_details()[0]["index"]
interpreter_quant.set_tensor(input_index_quant, test_image)
interpreter_quant.invoke()
predictions = interpreter_quant.get_tensor(output_index_quant)
print('output:,',predictions[0])
output:, [0. 0. 0.0625 0.8515625 0. 0.078125 0.
0. 0. 0. ]
test code on H7
import sensor, image, time, os, tf
sensor.reset() # Reset and initialize the sensor.
sensor.set_contrast(3)
sensor.set_pixformat(sensor.GRAYSCALE) # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA) # Set frame size to QVGA (320x240)
sensor.set_windowing((28, 28)) # Set 128x128 window.
sensor.skip_frames(time=100)
sensor.set_auto_gain(False)
sensor.set_auto_exposure(False)
net = tf.load('/mnist_model_quant_io.tflite')
clock = time.clock()
while (True):
clock.tick()
myImage = image.Image("white.ppm", copy_to_fb = True)
graytmp = myImage.to_grayscale(True, rgb_channel=0)
out = net.classify(graytmp)
print(out)
output as below:
[{“x”:0, “y”:0, “w”:28, “h”:28, “output”:[0.5019608, 0.5019608, 0.5369792, 0.9202359, 0.5019608, 0.544761, 0.5019608, 0.5019608, 0.5019608, 0.5019608]}]
[{“x”:0, “y”:0, “w”:28, “h”:28, “output”:[0.5019608, 0.5019608, 0.5369792, 0.9202359, 0.5019608, 0.544761, 0.5019608, 0.5019608, 0.5019608, 0.5019608]}]
[{“x”:0, “y”:0, “w”:28, “h”:28, “output”:[0.5019608, 0.5019608, 0.5369792, 0.9202359, 0.5019608, 0.544761, 0.5019608, 0.5019608, 0.5019608, 0.5019608]}]
[{“x”:0, “y”:0, “w”:28, “h”:28, “output”:[0.5019608, 0.5019608, 0.5369792, 0.9202359, 0.5019608, 0.544761, 0.5019608, 0.5019608, 0.5019608, 0.5019608]}]
[{“x”:0, “y”:0, “w”:28, “h”:28, “output”:[0.5019608, 0.5019608, 0.5369792, 0.9202359, 0.5019608, 0.544761, 0.5019608, 0.5019608, 0.5019608, 0.5019608]}]
[{“x”:0, “y”:0, “w”:28, “h”:28, “output”:[0.5019608, 0.5019608, 0.5369792, 0.9202359, 0.5019608, 0.544761, 0.5019608, 0.5019608, 0.5019608, 0.5019608]}]
[{“x”:0, “y”:0, “w”:28, “h”:28, “output”:[0.5019608, 0.5019608, 0.5369792, 0.9202359, 0.5019608, 0.544761, 0.5019608, 0.5019608, 0.5019608, 0.5019608]}]
[{“x”:0, “y”:0, “w”:28, “h”:28, “output”:[0.5019608, 0.5019608, 0.5369792, 0.9202359, 0.5019608, 0.544761, 0.5019608, 0.5019608, 0.5019608, 0.5019608]}]
why they are so different? 0.5019608 means?
The sum should be 1 or not?
white.7z (207 Bytes)