cnn accuracy on m7

Discussion related to "under the hood" OpenMV topics.
Eric
Posts: 5
Joined: Wed Mar 06, 2019 7:12 am

cnn accuracy on m7

Postby Eric » Wed Mar 06, 2019 7:14 am

Hi,

I'm using your M7 to run CNNs. Unfortunately I noticed that the performance of the models is not the same as during the test phase. For example, I trained a person recognition (binary classification (person in picture or not)) on the COCO database. The accuracy on the evaluation set is ~80%. As input I use color images of 128x128x3. As network I use a CNN with 4 layers. After quantization I loaded the model to the M7 and executed it. In very few cases the model recognizes people in the camera image. I.e. the recognition rate is bad.

I did the following experiments to find the error: First I loaded 100 images of people from the COCO database onto the microcontroller and processed them one after the other through the model. Of these 100 persons only 11 were recognized.

As a further test I used the quantized caffe model and wrote a small demo application. The pictures taken by my webcam serve as input. The recognition rate of this demo application is subjectively perceived much better. So there seems to be an error during the transformation from the quantized caffe model to the CMSIS code. Is there anything known? Any ideas where the error could be?

Thanks
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Wed Mar 06, 2019 11:30 am

Hi, the code that runs the CNNs is fine. However, when transforming the image to a low res one that's done using nearest neighbor if the resolution is set higher than the CNN model. Are you running the camera in 128x128 pixel mode? Also, I'd be concerned about the lighting of the camera image. We lower the exposure on the camera and overclock it slight to achieve higher FPS. However, you can force the exposure to be much longer use the set_auto_exposure() method which will drastically improve the image quality which can have adverse effects on the CNN.

One note, running the camera in 128x128 mode causes a lot of blur. You may just want to crop 128x128 off of a larger resolution picture.
Nyamekye,
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Wed Mar 06, 2019 11:34 am

Here's how the image gets brought in.

https://github.com/openmv/openmv/blob/m ... /nn.c#L263

If you have ideas to improve this code we'd love to get more feedback on it. ARM honestly hasn't been tool helpful after releasing CMSIS-NN. I don't know exactly if they are going to provide more support. I'll know more in a few weeks. I'm going to this Tiny-ML conference in 2 weeks where I hear there's going to be some big news.
Nyamekye,
Eric
Posts: 5
Joined: Wed Mar 06, 2019 7:12 am

Re: cnn accuracy on m7

Postby Eric » Thu Mar 07, 2019 8:39 am

Thanks for the response. As input sensor format I use VGA. From this picture I cut a window of 256 x 256. In addition, I set the exposure time manually. But that doesn't seem to be the problem. As already mentioned, the model performance on images I read from the SD card is also bad. Here is the example of how I tested the recognition rate on data that was also present in the training material:

Code: Select all

import sensor, image, time, pyb, time, os, nn

# load model
net = nn.load('/persondetection_fast_s_128.network')
labels = ['no_person', 'person']

# Still need to init sensor
sensor.reset()
# Set sensor settings
sensor.set_contrast(1)
sensor.set_gainceiling(16)

# Set sensor pixel format
sensor.set_framesize(sensor.QQVGA)
sensor.set_pixformat(sensor.RGB565)


cnt = 0
for i in range(100):
    # Load image
    image_path = "/ppm_images/" + str(i) + "_1.ppm"
    img = image.Image(image_path, copy_to_fb=True)

    # Flush FB
    sensor.flush()

    # Add a small delay to allow the IDE to read the flushed image.
    time.sleep(100)

    #img = sensor.snapshot()         # Take a picture and return the image.
    out = net.forward(img)
    max_idx = out.index(max(out))
    score = int(out[max_idx]*100)

    score_str = "%s:%d%% "%(labels[max_idx], score)
    print(max_idx)

    if max_idx == 1:
        cnt+=1

print(cnt)
Furthermore, I have noticed that when a person is detected, the confidence is not very high (~51%). Also, I don't understand the sense behind the parameter softmax. If I set it to True, the output is wrong. The values do not add up to one. I don't think the error is in the CMSIS library. I had already used them for other projects without any problems. When I find the time I also look at the code you sent me.
User avatar
iabdalkader
Posts: 834
Joined: Sun May 24, 2015 3:53 pm

Re: cnn accuracy on m7

Postby iabdalkader » Thu Mar 07, 2019 10:58 am

Eric wrote:
Wed Mar 06, 2019 7:14 am
Of these 100 persons only 11 were recognized.
Are you sure the labels are not inverted ? 0 for person 1 for something else ? Note some accuracy is lost when the model is quantized, but shouldn't be this bad.
Eric
Posts: 5
Joined: Wed Mar 06, 2019 7:12 am

Re: cnn accuracy on m7

Postby Eric » Mon Mar 11, 2019 10:48 am

Of course not...
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Mon Mar 11, 2019 10:55 am

Okay, can you provide everything we need to reproduce it? Ibrahim can test it out.
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: cnn accuracy on m7

Postby ngattuso » Sat Mar 16, 2019 6:39 pm

Hi, I'd like to add that I've been seeing the same thing. When running the CNN on the M7, I see poor results
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Sat Mar 16, 2019 6:48 pm

Can you provide someway for us to reproduce the issue?

Can you provide the training and test dataset along with the labels and everything we need to quickly run it through caffe?
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: cnn accuracy on m7

Postby ngattuso » Sat Mar 16, 2019 7:10 pm

I just shared a Google Drive to you at your email: openmv@openmv.io

Download:
- "dataset.zip" (you might need to delete the MACOSX folder after unzipping.
- createTrainTest.py
- test.sh
- train.sh
- the "examples" folder
- trashscan_solver.prototxt
- trashscan_train_test.prototxt
You should already have the quantize and convert scripts. If not, they are on the drive too.

Note: the updated solver is on my other computer. All that is different is test_interval = 500, stepsize = 2500, max_iter = 10000

On my computer, I created a folder called "trashscan" and stored all these files inside. When running, I'm inside the "trashscan" directory. You shouldn't get any path issues if you do that too. The rest of the steps are identical to your guide. However, if needed, I can provide them (there's a word document in the drive with the steps but it isn't updated with the paths and info...though, it explains which scripts to call and the order; only quantize and convert need arguments passed)

Additionally, I attached the code for my program on the M7
Attachments
trashscan.py
(7.53 KiB) Downloaded 199 times
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Sat Mar 16, 2019 8:55 pm

Hi, can you re-send that? I accidentally deleted it because I wasn't expecting it. Google just sent an email saying it shared a drive with me but there was no identifying info so it looked like a phishing attack.
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: cnn accuracy on m7

Postby ngattuso » Sat Mar 16, 2019 9:23 pm

You should have the link for it now.

I'm going to try different image qualities right now. I have the following setup:

Code: Select all

sensor.reset()                      # Reset and initialize the sensor.
#sensor.set_contrast(3)
sensor.set_pixformat(sensor.RGB565) # Set pixel format to RGB565 (or GRAYSCALE)
#sensor.set_framesize(sensor.CIF)   # Set frame size to CIF (352x288)
#sensor.set_windowing((280, 288))    # Set window.
sensor.set_framesize(sensor.QVGA)   # Set frame size to QVGA (320x240)
sensor.set_windowing((160, 210))    # Set window.
sensor.skip_frames(time = 2000)     # Wait for settings take effect.
#sensor.set_auto_whitebal(False)     # Turn off white balance.
#sensor.set_auto_gain(False)
#sensor.set_auto_exposure(False)
Do you have any suggestions on things I can try? Note: I have two extra fb so I needed to run at QVGA.
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Sat Mar 16, 2019 10:00 pm

Not sure what you mean by your question.

Um, I will try to process this tomorrow since it should be a working feature. I'll try to get to the step of verifing you were doing everything right and performance just fell on the M7. That will then give something for Ibrahim to fix.
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: cnn accuracy on m7

Postby ngattuso » Sun Mar 17, 2019 5:13 pm

My question was just to see if there was an ideal sensor setup for the CNN or other values I should try instead of what I did. For example, if I use the exposure, what should be the milliseconds I use?

Okay, thank you for looking into my process for the CNN. Please let me know if there's anything else I need to provide you and please keep me informed with your results.

Thank you,
Nicholas
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Sun Mar 17, 2019 5:43 pm

Not really, just expect that the CNN needs to see images that look like what you trained on. Deviation from this will make the net not work at all.
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: cnn accuracy on m7

Postby ngattuso » Sun Mar 17, 2019 8:16 pm

Okay perfect! I took photos with the M7 and test in the same environment with the same objects. This is how I could tell that there was a problem
Eric
Posts: 5
Joined: Wed Mar 06, 2019 7:12 am

Re: cnn accuracy on m7

Postby Eric » Tue Mar 19, 2019 5:42 am

Sorry for the late reply and thanks for the offer. I have sent you all relevant scripts by e-mail. First you have to download the COCO database:

Train: http://images.cocodataset.org/zips/train2014.zip
Validation: http://images.cocodataset.org/zips/val2014.zip
Labels for Train and Validation: http://images.cocodataset.org/zips/val2014.zip

The next step is to unpack the database and place it in a convenient location to memory. Next the script prepare_database_for_caffe.py must be executed. In lines 98 and 99, change the corresponding path to the COCO database and the path to the location for the processed data. The script scales the images to 128 x 128, labels the data to categories 0 for no_person and 1 for peron. In addition, images in which people make up very little space of the image are removed. Finally, the data set will be balanced.

The next step is training. All relevant scripts can be found at cmsisnn/persondetection_fast/. The structure is similar to your example scripts. In the commands file you can find all relevant commands for training. Furthermore change the database path in persondetection_fast_train_test.prototxt. Last but not least I left the solverstate and the quantized model in the folder in case you want to start without a database. I also attached the normalization file.

If you want to test the quantized Caffe Model on the data of your webcam you can use the scripts execute_caffe_model.py, network_camera_client.py and network_camera_servery.py. To do this please change the path to the trained model in the script execute_caffe_model.py (lines 16, 17, 18)

In main.py you can find the code for the M7.

Many thanks in advance.
Eric
Posts: 5
Joined: Wed Mar 06, 2019 7:12 am

Re: cnn accuracy on m7

Postby Eric » Tue Mar 19, 2019 6:01 am

I couldn't send you an email with the rar. Therefore I added the rar here. I sent the password by mail.
Attachments
CodePersonDetection.rar
(514.41 KiB) Downloaded 209 times
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Wed Mar 20, 2019 1:25 am

Hi, I will get to looking at this. I've put the email on my desktop. Please note that I'm under a lot of work right now. I have to start Kickstarter shipping, release v3.3.0, and do taxes. While I'd like to give this a higher priority I have to focus on the other things first.
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: cnn accuracy on m7

Postby ngattuso » Wed Mar 20, 2019 7:23 pm

Eric, do you mind sending me your Matlab program to test the CNN? It would be great to test my CNN that way. If so, can you email me at dj_nick_gattuso@hotmail.com.

Thanks!
Nicholas
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Thu Mar 21, 2019 1:30 am

Hi, we verified there's a problem with the latest firmware with CNN stuff. We will do another release with a fix for this and a fix for another issue with set windowing a customer found.

Regarding CNN issues. We talked to Pete Warden recently and tensor flow light may be coming soon. If this happens we will move away from this custom stuff and have tensor flow support.
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: cnn accuracy on m7

Postby ngattuso » Thu Mar 21, 2019 6:07 pm

Hi, thank you for the verification! Do you know how long until the release? I'm asking because I have a demo on Wednesday and would like to keep an eye open to have my openMV updated earlier.

The tensor flow support would be amazing!

Thank you,
Nicholas
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Thu Mar 21, 2019 8:24 pm

Um, well Ibrahim fixed the issue with the file not being able to be opened. As for the accuracy I can't say how long its going to fix that. Getting you the binary for the latest firmware is pretty simple.
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: cnn accuracy on m7

Postby ngattuso » Fri Mar 22, 2019 3:26 pm

Okay no problem, if you can please let me know when you have the bin file ready!

Alternatively, how is the performance on the H7? Did that fall too? If not, I can use my H7 temporary until the M7 firmware is ready.

Thanks,
Nicholas
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Fri Mar 22, 2019 6:26 pm

Um, well, I've only tested the H7 with my cifar10 net and it works great as long as the image is binarized.

I'm sorry about this not being the best right now. However, with tensor lite now available we should be able to get this to some level of spec where these issues go away. The plan was to start main streaming this more with the H7.
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: cnn accuracy on m7

Postby ngattuso » Sat Mar 23, 2019 4:39 am

Okay great! Thanks for the information!

Definitely looking forward in the updates and new features!
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Sat Mar 23, 2019 10:56 am

Ibrahim will release v3.3.1 today which fixes the file loading issue.
Nyamekye,
ngattuso
Posts: 42
Joined: Thu Dec 27, 2018 8:29 pm

Re: cnn accuracy on m7

Postby ngattuso » Tue Mar 26, 2019 7:25 pm

Great! I just installed it!

Please keep me informed about the fix for the CNN processing

Thank you,
Nicholas
User avatar
kwagyeman
Posts: 3045
Joined: Sun May 24, 2015 2:10 pm

Re: cnn accuracy on m7

Postby kwagyeman » Wed Mar 27, 2019 12:27 pm

Ibrahim is on it. This is the only thing on his plate right now.
Nyamekye,
User avatar
iabdalkader
Posts: 834
Joined: Sun May 24, 2015 3:53 pm

Re: cnn accuracy on m7

Postby iabdalkader » Thu Mar 28, 2019 7:09 pm

@Eric I've tested your network using the first 1000 images in val_data on the camera, and the accuracy is very low indeed. Then I checked the first 100 images in val_data and I don't think the network will learn anything useful from these images (some images are mislabeled, others contain multiple persons, heads, hands, hidden people etc...). This may explain the low accuracy, I think if you filter the dataset you will get much better results.

Return to “Technical Discussion”

Who is online

Users browsing this forum: No registered users and 8 guests