Tensorflow Updates?

OpenMV related project discussion.
Project_Sat
Posts: 7
Joined: Sat Sep 14, 2019 9:02 pm

Tensorflow Updates?

Postby Project_Sat » Sat Sep 14, 2019 9:14 pm

Hey guys,

Thanks for all the work with the OpenMV. I am actually doing this really cool project that I plan on to deploy OpenMV with a custom built tensorflow model on a CubeSat (Cubesat? https://www.nanosats.eu/cubesat) in low earth orbit. We have already built and launched 3 such satellites and are all functioning (https://birds3.birds-project.com/). Now we want the satellite's camera to be a bit smarter in our future missions.

I have trained my model on tensorflow with Keras API. The model was then converted to tensorflow lite model. I need to now deploy that model on the OpenMV. I read here:
viewtopic.php?t=1287

that the next firmware update will have tensorflow support. However, when I checked the blog tab, it does look like the latest firmware has not been released. My question would be

1) When would it be released? Would there be a guide to load custom model on the MCU?
The reason why I ask is that we need to complete the prototype by end of September, would be really useful if I could have a proof of concept by then.

What I can offer in return is this:
A) Space qualify OpenMV camera (we have all the facilities here, thermal vacuum chamber, vibration and shock), also can do total ionizing dose radiation testing
B) Write paper, mention OpenMV and provide acknowledgment. I want the space community to use OpenMV in the future much like how arduinos and pi are being tested and used in space.

2) Or is there any other way to convert the tensorflow model to caffe, quantize and then load it up on the OpenMV?

Let me know. Thanks again, big fan here.
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: Tensorflow Updates?

Postby kwagyeman » Tue Sep 17, 2019 11:09 am

Hi, I will merge my PR for this. Um, let me get back to you with a more detailed response. I long post requires a long response from me.
Nyamekye,
Project_Sat
Posts: 7
Joined: Sat Sep 14, 2019 9:02 pm

Re: Tensorflow Updates?

Postby Project_Sat » Thu Sep 19, 2019 9:28 pm

Thanks, I will be waiting
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: Tensorflow Updates?

Postby kwagyeman » Thu Sep 19, 2019 11:42 pm

Hi Project_Sat,

The work to allow tensor flow support on the OpenMV Cam was completed back in April. We have been working on a DRAM camera that will be really good for this. Using the internal SRAM is also possible.

Anyway, this is what I can do... I can merge my PR for this soon. Once I do that the support for the feature will be in the firmware. Then, you can execute a tensor flow lite model that has been converted to a flat buffer file. There are a number of command line steps to build a usable model file but they are simple.

I wasn't able to test my code without SDRAM support. So, now that we have that I can.

As for other things you were offering, sounds good. Thanks.
Nyamekye,
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: Tensorflow Updates?

Postby kwagyeman » Fri Sep 20, 2019 1:43 pm

Hi, I'll be working on getting TensorFlow into the firmware release soon. Right now, the branch to merge is here:

https://github.com/kwagyeman/openmv/tree/kwabena/add_tf

If you rebase it ontop of the main branch you should be good to go. Note that I was building mobilenet into the main firmware which may not be the best idea. You might have to take that out.
Nyamekye,
Project_Sat
Posts: 7
Joined: Sat Sep 14, 2019 9:02 pm

Re: Tensorflow Updates?

Postby Project_Sat » Mon Sep 23, 2019 9:02 pm

Noted, thanks
Project_Sat
Posts: 7
Joined: Sat Sep 14, 2019 9:02 pm

Re: Tensorflow Updates?

Postby Project_Sat » Wed Oct 09, 2019 4:57 am

Hi again,

I seem to be bit stuck with what to do next. So far I was able to:

1) Train model in TensorFlow
2) Change the model to TensorFlow to TensorflowLite
3) use xxd -i model.tfile > model.c
4) Manually fix the variable inside from unsigned int to const unsigned int looking at the models trained here:
https://github.com/openmv/tensorflow-li ... obilenet.c

I looked at what you said earlier and I realized I didn't seem to grasp everything. "To rebase it on top of the branch" in github.

I was wondering what my next step would be to use model.c with STM32H7 on OpenMV IDE? Would this only be available after firmware is released? Is there a rough code example of implementation that I can base my code on? I apologize if I am repeating my previous question.
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: Tensorflow Updates?

Postby kwagyeman » Wed Oct 09, 2019 11:30 am

Hi, the branch I pointed to has the full implementation of tensorflow completed. You just rebase it ontop of main and then the tensorflow code will be available in the python API. You put the network on the SD card and then load it with the tensflow code and it should run.

That said, I was never able to test the code because the mobilenet model supplied by Google doesn't fit in our RAM.

...

I'm starting to work through getting this back on the OpenMV Cam in the main firmware now. I've completed all other tasks that are blocking working on this. We have a 32-bit sdram camera that's more or less working now.
Nyamekye,
Project_Sat
Posts: 7
Joined: Sat Sep 14, 2019 9:02 pm

Re: Tensorflow Updates?

Postby Project_Sat » Wed Oct 09, 2019 5:39 pm

Thanks, will get back with the results.
Project_Sat
Posts: 7
Joined: Sat Sep 14, 2019 9:02 pm

Re: Tensorflow Updates?

Postby Project_Sat » Sun Oct 13, 2019 11:44 pm

Hi again,

Some updates on my progress, do let me know if I am missing out or doing something wrong. Where I need help is to figure out what my next step would be.

Step1: Rebasing https://github.com/kwagyeman/openmv/tree/kwabena/add_tf on top of forked repository of main which is https://github.com/openmv/openmv

Code: Select all

git clone --recursive https://github.com/openmv/openmv
cd openmv
git remote add tf https://github.com/kwagyeman/openmv/tree/kwabena/add_tf
git fetch tf
git branch -a --v
Which gave me these results

Code: Select all

 remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/origin/openmv-1.9.4
  remotes/origin/update_examples
  remotes/tf/cmsis_improvements
  remotes/tf/kwabena/add_tf
  remotes/tf/kwabena/fb_alloc_caching
  remotes/tf/kwabena/fb_alloc_caching_2
  remotes/tf/kwabena/fix_bank_enables
  remotes/tf/kwabena/fix_bmp_24_bit
  remotes/tf/kwabena/fix_bmp_reading
  remotes/tf/kwabena/ov5640
  remotes/tf/kwabena/sdram_support
  remotes/tf/master
  remotes/tf/more_nn_nets
I then did this:

Code: Select all

git checkout tf/kwabena/add_tf
git rebase master
Had one conflict and after resolving that

Code: Select all

git add . 
git rebase --continue
Was then able to complete the rebase.

I am still new at this so I am still trying to figure out what my next step would be? How can I link this file with OpenIDE? Do i need to rebuild from source?

In another note, how about this. I can share my CNN trained on Tensorflow and then converted to Tensorflow lite along with the flat buffer file. The tensorflow lite code is only about 104kB and I am thinking should run on SRAM. That way I can still make sure my NN runs on the OpenMV cam and also perhaps get a basic example code from you that could help me run the CNN on the module. All I need to show for my research is that the CNN runs on an MCU and does it's job. That's it, then I can proceed with my writing.

Let me know your thoughts. If you want me to share the CNN, I can post it here as soon as I get the reply.

Thanks again for the time. I know you guys are busy.
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: Tensorflow Updates?

Postby kwagyeman » Mon Oct 14, 2019 12:07 am

Hi, just wait for me to finish the porting. I've already started on it and managed to get past the first sets of issues with updating TensorFlow. I should be done getting the software ready to integrate into our firmware on Monday and then be done getting it merged by the end of the week. Because TensorFlowLite for Microcontrollers has so much code in it I have a separate repo where I compile it into a library file for integration into our firmware.

...

As for what you have done so far. Yes, you have to compile the firmware and then flash the camera. There's no documentation for the tf module however so you have to read the c code and fix any bugs that are there until I am finished.
Nyamekye,
Project_Sat
Posts: 7
Joined: Sat Sep 14, 2019 9:02 pm

Re: Tensorflow Updates?

Postby Project_Sat » Mon Oct 14, 2019 2:41 am

Sure, will wait then.

Thanks again

Return to “Project Discussion”

Who is online

Users browsing this forum: No registered users and 1 guest