RGB channel extraction

Discussion related to "under the hood" OpenMV topics.
User avatar
kwiley
Posts: 107
Joined: Fri Dec 21, 2018 5:22 pm

RGB channel extraction

Postby kwiley » Sat Jan 19, 2019 12:22 pm

There's a grayscale conversion function, but I don't see a channel extraction function. At the current time, would get_pixel() be the only approach? I realize Lab does something like this, but I was just thinking about raw channel extraction...actually, now that I just wrote the word raw, I'm wondering if I can do this from the Bayer data, but still, it would be iterative and Python unless there is a compiled function for it. What are your thoughts on color channel to grayscale image extraction, be it RGB, HSV, or even Lab (the 3-to-1 channel extraction problem still exist after the Lab transform)?

Is grayscale conversion (presumably channel averaging) the only channel-flattening option currently supported?
User avatar
kwiley
Posts: 107
Joined: Fri Dec 21, 2018 5:22 pm

Re: RGB channel extraction

Postby kwiley » Sat Jan 19, 2019 12:28 pm

Oh, and it looks like the RGB/Lab conversion is on a single pixel, so it would still involve iterating over get_pixel(). I don't think I can do a Lab transformation on the entire image like image.to_grayscale(), to_rgb565, and to_rainbow().
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: RGB channel extraction

Postby kwagyeman » Sat Jan 19, 2019 12:41 pm

Um, I'm not sure what this would be needed for. The A and B or U and V channels in LAB and YUV aren't interesting to look at. The image is basically flat with no contrast.

...

:)

Would you like to take a crack at implementing this? It would be pretty easy for you to add an extra argument to to_grayscale() to pull out the channel you want instead of just the grayscale average.

If I could convert you into a source contributer I've love it.

Doing this code change is really easy.
Nyamekye,
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: RGB channel extraction

Postby kwagyeman » Sat Jan 19, 2019 12:43 pm

Um, you can also set the pix format to BAYER and then get_pixel() will return the pixels of the Bayer image. In the Bayer image format none of our algorithms work however except get pixel and joeg compression for viewing.

When you set the image size to VGA RGB565 we automatically switch to Bayer to fit in RAM.
Nyamekye,
User avatar
kwiley
Posts: 107
Joined: Fri Dec 21, 2018 5:22 pm

Re: RGB channel extraction

Postby kwiley » Sat Jan 19, 2019 2:21 pm

There are lots of image processing algorithms that operate on the RGB channels separately, although in my case I'm less interested in per-channel image manipulations (levels or something like that) and more interested in ML applications on a per channel basis. For example, I might want to train and classify just the red channel as opposed to the grayscale image on the expectation that mixing green and blue into the signal simply muddies (averages away via grayscale conversion) otherwise useful discriminatory information that the red channel otherwise provides. An 8-bit grayscale image only preserves 8/3 bits or perhaps 2.5/3/2.5 bits of RGB information, whereas the red channel extracted whole-parcel provides 8 bits (or I suppose perhaps only 5 true bits if it originated from a 565 image) of red information.

At any rate, that's my thinking on the matter. It might not pan out, but that's the idea. I was just wondering about the prospect of channel extraction.

As you point out, if I need it, I can try to implement it myself. I mostly post these questions because I'm very new to OpenMV (despite it having been around for a few years; I'm very late to the party; I even missed the H7 launch) so I'm not sure if some of these things are already provided in ways I am unaware of. For example, you explained that I had overlooked max-pooling in another thread. That's exactly what I'm asking, not to request new future work, but to clarify what is currently supported.

As to your final point, I may very well become a code contributor. I admit I'm very new here and I'm weary to jump into the code when I don't really know the lay of the land yet. I need to look over whatever tutorials or guidance are available to get into the firmware or library levels of the project.

Cheers!
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: RGB channel extraction

Postby kwagyeman » Sat Jan 19, 2019 8:45 pm

The wiki on how to edit the firmware is here: https://github.com/openmv/openmv/wiki

It's actually not that hard. It's a pretty straight forward process to edit the code and add new stuff since there are so many examples in the code on how to do so.
Nyamekye,
User avatar
kwiley
Posts: 107
Joined: Fri Dec 21, 2018 5:22 pm

Re: RGB channel extraction

Postby kwiley » Sun Jan 20, 2019 1:38 am

Ugh, not Mac native, huh? I'll see how far I can get, but to be honest, I'm disinclined to fully install Linux on my Mac just for this purpose when I don't otherwise use Linux very much. I installed Linux on a Mac once (admittedly it was in 1998; perhaps it's easier now) and the experience left serious emotional scares. I still hug myself at night thinking about it.

I'll give it a shot when I have some time to cozy up with the Linux install directions, but I'm pretty busy. I just don't know if that's a project I want to take on right now.
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: RGB channel extraction

Postby kwagyeman » Sun Jan 20, 2019 3:44 am

Use VMWare Fusion. Anyway, I'll be doing code...
Nyamekye,
User avatar
kwiley
Posts: 107
Joined: Fri Dec 21, 2018 5:22 pm

Re: RGB channel extraction

Postby kwiley » Sun Jan 20, 2019 10:41 am

:) The thing I find most intriguing is the possibility of writing my own compiled libraries and accessing them from a Python driver. The platform as a whole is very appealing. I'll just have to find the time.
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: RGB channel extraction

Postby kwagyeman » Sun Jan 27, 2019 7:54 pm

I've added this feature now. Just pass rgb_channel=0/1/2 to to_bitmap(), to_grayscale(), to_rgb565(), and to_rainbow() and the methods will use the r/g/b channels respectively for whatever they are doing. For to_rgb565() this means that I zero the other channels (i.e. I only copy the selected one). For the rest of the methods this means I use the selected channel as the grayscale input for the conversion versus the Y value computed from the R/G/B color channels. Passing any other value to rgb_channel= will cause the default behavior.
Attachments
firmware.zip
(918.6 KiB) Downloaded 243 times
Nyamekye,
User avatar
kwiley
Posts: 107
Joined: Fri Dec 21, 2018 5:22 pm

Re: RGB channel extraction

Postby kwiley » Sun Jan 27, 2019 11:15 pm

May I presume this also includes the recent changes with regard to image rotation?

Now I just have to learn how to flash the firmware. I see where the docs say "Use the IDE built-in dfu tool to update the firmware." but I haven't figured out what that means yet. I noticed that if I click the firmware version at the bottom of the screen it says I'm up to date and then asks if I want to update to the release version. But if that is only pulling release versions, presumably it won't work with a firmware file I download directly, such as above.

Can the IDE only update to the release version? To update from a file do I have to use the manual directions?

I'll dig around the forum and docs a bit. I haven't looked too hard for firmware directions yet.

Thanks for the update!
User avatar
kwiley
Posts: 107
Joined: Fri Dec 21, 2018 5:22 pm

Re: RGB channel extraction

Postby kwiley » Sun Jan 27, 2019 11:36 pm

Oh, the "Run Bootloader" menu option seems to do the trick.
User avatar
kwiley
Posts: 107
Joined: Fri Dec 21, 2018 5:22 pm

Re: RGB channel extraction

Postby kwiley » Sun Jan 27, 2019 11:41 pm

Ummm, funny question, but did you do anything to the fonts in the recent firmware? Drawn text looks quite different all of a sudden (and much worse, very hard to read now, at least at the same font settings).

I only see this new artifact at scale 2. Scales 1 and 3 look good.
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: RGB channel extraction

Postby kwagyeman » Mon Jan 28, 2019 12:00 am

I made draw text support non-integer scaling. Hmmm, will fix. I'm going to work on drawing updates next.

I'll fix this, fix an issue with ellipse drawing at extreme angles, and add the rotated text stuff next.
Nyamekye,
User avatar
kwiley
Posts: 107
Joined: Fri Dec 21, 2018 5:22 pm

Re: RGB channel extraction

Postby kwiley » Mon Jan 28, 2019 12:56 am

Oh hey, look at that. You did. That's an awesome addition. It might be worth looking at the text quality a bit, at least to replicate the previous integer values (make 2 like the old 2) even if the floating values have some issues for the time being.

It's not too important.

Cheers!
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: RGB channel extraction

Postby kwagyeman » Sat Feb 16, 2019 9:53 pm

Hi, I fixed the draw text stuff and added the rotation features you wanted.

Given our current amount of feature expansion has grown quite large we will be cutting a release soon.

Code: Select all

# Text Drawing
#
# This example shows off drawing text on the OpenMV Cam.

import sensor, image, time, pyb

sensor.reset()
sensor.set_pixformat(sensor.RGB565) # or GRAYSCALE...
sensor.set_framesize(sensor.QVGA) # or QQVGA...
sensor.skip_frames(time = 2000)
clock = time.clock()

while(True):
    clock.tick()

    img = sensor.snapshot()

    for i in range(10):
        x = (pyb.rng() % (2*img.width())) - (img.width()//2)
        y = (pyb.rng() % (2*img.height())) - (img.height()//2)
        r = (pyb.rng() % 127) + 128
        g = (pyb.rng() % 127) + 128
        b = (pyb.rng() % 127) + 128

        # If the first argument is a scaler then this method expects
        # to see x, y, and text. Otherwise, it expects a (x,y,text) tuple.

        # Character and string rotation can be done at 0, 90, 180, 270, and etc. degrees.
        img.draw_string(x, y, "Hello World!", color = (r, g, b), scale = 2, mono_space = False,
                        char_rotation = 0, char_hmirror = False, char_vflip = False,
                        string_rotation = 0, string_hmirror = False, string_vflip = False)

    print(clock.fps())
Attachments
firmware.zip
(930.18 KiB) Downloaded 234 times
Nyamekye,
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: RGB channel extraction

Postby kwagyeman » Sat Feb 16, 2019 9:57 pm

I will not be adding rotations other than 90 degrees to draw text. If you need more than that you can now create a blank image, draw text on it, rotate the blank image, and then draw that image on the main image. Yes, not as easy but that's basically what I would have to do int he C code to add arbitrary rotations and I was trying not to re-write draw text.
Nyamekye,
daniel
Posts: 2
Joined: Fri Jul 26, 2019 7:56 am

Re: RGB channel extraction

Postby daniel » Fri Jul 26, 2019 8:07 am

I don't get it, you upload a firmware for divide the channels, i need to have only the red channel (infrared channel ), I did what you said in the forum and what you sent in your link (ubuntu-qt-launch to open mv and the run bootloader) in the run bootloader is able to upload a new file firmware.bin (the one in the forum), i try to upload the bin file the launched from openmv in the qt but always appeare unable to connect (less 5 seconds) in the second time i try to reinstall the firmware then the cam erase the files and the process with the installation when the camara finish the installation the green led dissapears and then there is non white-red-blue color led saying me that the test was failed i have to reinstall the old firmware but i still need to have a firmware only with red channel so i am not sure if a should modify that firmware or i still not sure if it is just a problem with the firmware file you uploaded. thanks for you attention :), The first time i try it says unable to connect to your openmv Cams Normal bootloader.
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: RGB channel extraction

Postby kwagyeman » Fri Jul 26, 2019 2:41 pm

Hi, the latest firmware that has been release has the above features. You do not need to upload new firmware to your camera manually.
Nyamekye,
daniel
Posts: 2
Joined: Fri Jul 26, 2019 7:56 am

Re: RGB channel extraction

Postby daniel » Mon Jul 29, 2019 2:49 am

Can you send me an example code of how you use the method in the OpenMV for change to red channel please?
User avatar
kwagyeman
Posts: 3335
Joined: Sun May 24, 2015 2:10 pm

Re: RGB channel extraction

Postby kwagyeman » Mon Jul 29, 2019 12:21 pm

Do:

img.to_rgb565(rgb_channel=0)

This will extract the red channel only. If you want the image to go to grayscale do:

img.to_grayscale(rgb_channel=0)

Then the image will be the red channel but in grayscale.
Nyamekye,

Return to “Technical Discussion”

Who is online

Users browsing this forum: No registered users and 5 guests