Place a snapshot into a new white created image with different resolution and display it in the Frame Buffer preview

Kosu · May 16, 2023, 4:33pm

Hi. I tried to take a snapshot with a resolution of 128x64px and then copy it to the centre of a created image of 128x128px, but nothing, I can’t do it.
Here is an image of what I want to do:
New Project(1)

kwagyeman · May 16, 2023, 9:29pm

What’s your code?

You need to allocate a second image buffer and then use the draw image method.

Kosu · May 17, 2023, 6:51am

I can’t see any difference in the Frame Buffer preview (in the ide).

import sensor, image, time

sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.B128X64)
sensor.skip_frames(time = 2000)

clock = time.clock()

while(True):
    clock.tick()
    img = sensor.snapshot()
    img2 = image.Image(128, 128, sensor.RGB565, True)
    img2 = img2.draw_image(img, 0, 32, 1, 1)
    print(clock.fps())

Kosu · May 17, 2023, 9:03am

By saving the image I can see that it worked, but I want it to be seen in the Frame Buffer preview for debugging.

Code:

import sensor, image, time, os

sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.B128X64)
sensor.skip_frames(time = 2000)

clock = time.clock()

while True:

    clock.tick()
    img = sensor.snapshot()
    img2 = image.Image(128, 128, sensor.RGB565)
    img2.draw_image(img, 0 ,32)
    img2.save(os.getcwd() + "image.jpg")
    print(clock.fps())

I have seen that there is the Frame Buffer class, do I have to use it for the second image buffer?

Kosu · May 17, 2023, 10:27am

Okay maybe I made something correct here, but I still can’t see the image I want in the frame buffer preview (I want the 128x128px image in the frame buffer preview).

import sensor, image, time, os

sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.B128X64)
sensor.skip_frames(time = 2000)
img2 = sensor.alloc_extra_fb(128, 128, sensor.RGB565)

clock = time.clock()

while True:
    clock.tick()
    img = sensor.snapshot()
    img2.draw_rectangle(0,0, 128, 128, (255,255,255), fill=True)
    img2.draw_image(img, 0 ,32)
    img2.save(os.getcwd() + "image.png")
    print(clock.fps())

oramafanis · May 17, 2023, 1:41pm

I don’t know if you can but what I do is
print(img.compressed_for_ide())
And then connect to the ide terminal to see the desire image …
It works for me

Kosu · May 17, 2023, 2:17pm

Hey, thanks for the reply. It works and it doesn’t work. It shows both the images (the one from the snapshot() and the one I call the function on) one after the other, and the frame rate goes down.

Here is what I get from executing the code:

video_forum_compressed_for_ide

Code:

import sensor, image, time, os

sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.B128X64)
sensor.skip_frames(time = 2000)
img2 = sensor.alloc_extra_fb(128, 128, sensor.RGB565)

clock = time.clock()

while True:
    clock.tick()
    img = sensor.snapshot()
    img2.draw_rectangle(0,0, 128, 128, (255,255,255), fill=True)
    img2.draw_image(img, 0 ,32)
    print(img2.compressed_for_ide())
    print(clock.fps())

What I am trying to get is the preview of the 128x128px image only.

oramafanis · May 17, 2023, 6:13pm

Please use the terminal to see the image you want. Not the ide preview.

kwagyeman · May 17, 2023, 6:40pm

Hi, the way our frame buffer architecture is setup the IDE is always trying to grab the first image.

Basically, when snapshot is called it jpeg compresses the current image in the primary frame buffer and sends that to the PC. Then it puts the next image from the camera in the frame buffer.

So, if you want to see your image you need to load it into the primary frame buffer.

import sensor, image, time, os

sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.B128X64)
sensor.skip_frames(time = 2000)
img2 = sensor.alloc_extra_fb(128, 128, sensor.RGB565)

clock = time.clock()

while True:
    clock.tick()
    img = sensor.snapshot()
    img2.draw_rectangle(0,0, 128, 128, (255,255,255), fill=True)
    img2.draw_image(img, 0 ,32)
    img.set(img2) # overwrite the old image and replace with the new one
    print(clock.fps())

kwagyeman · May 17, 2023, 6:41pm

If you were just trying to crop/scale your image you can do that in place. Wanting to add black borders is an odd thing to do.

What’s the reason for wanting to do this? While it may look good it creates pixels that have no purpose in the image. This is why we don’t have a library function for it.

Kosu · May 17, 2023, 6:54pm

I have a wide angle lens and need to use OpenMV for object detection. I need to use square images for Edge Impulse training, but when I set the resolution a square resolution, like 128x128px, the edges of the image are cut off. So to avoid losing parts of the image, I thought about adding white pixels in the shortest axis. Maybe it’s not a brilliant idea, but it was the first thing that came to mind.

Kosu · May 17, 2023, 6:57pm

Thanks it works now. The reason I wanted it to be displayed in the ide preview was because of the dataset editor

Kosu · May 17, 2023, 7:02pm

The code does not work. I think the cause is that the pixel format is not the same.
Error: “OSError: Images not equal!”

kwagyeman · May 17, 2023, 7:32pm

So, just to note, the CNN will learn the black border and only respond to images that have that.

You 100% do not want to feed the CNN with a black border in the image. This will break the model.

You have to crop/scale your image such that the object of interest is in the 128x128 filed of view. Also, you need to avoid warping the image. So, the scale needs to maintain the aspect ratio.

Kosu · May 17, 2023, 9:46pm

Let me explain the problem from the beginning.

Goal

I have the Ultra Wide Angle Lens mounted on the OpenMV. I have to make it detect different letters. I’m using Edge Impulse for the AI. On Edge Impulse, in the create impulse section, I’m using the “Object Detection (Images)” as learning block. This force me to use images with a square resolution as input data both for training and usage of the AI model after the deployment.
In short what I need to do is detecting different letters in walls. Using Edge Impulse have me to use only images with square resolution.

Resolution problem

Here is the problem! If i set the a square resolution such as:

sensor.set_framesize(sensor.320x320px)
# or
sensor.set_framesize(sensor.128x128px)
# and so on...

I lose a significant left and right portion of image.
This is the image with a square resolution (320x320px):

image_with_square_resolution

Here is the image with a non square resolution (I used sensor.set_framesize(sensor.QVGA) to take this snapshot, which means 320x240px):

image_with_correct_resolution

As you can see in the first photo, the one taken with the square resolution, it has been cut a very important portion of the image.

Our idea

Talking to my teacher, who is studying at university for a degree in AI, it turned out that cropping or squashing the image is not a good solution in this case. Therefore, we thought of adding white pixels in the shortest axis to obtain a square image and keep the original image intact.
Here is how the image with that “manipulation” should look like:

image_manipulated
(I have used gray in this case just because the website background is white).

Given that for the training of the AI I have to select the area where the letters are for each image and for each letter, it does not matter if there is a a “big” area of only white pixels, as they are not in the letter’s square.
This is what I mean:

image_with_red_rectangles

The white (gray in this case) area is never included in the red rectangles, so it will never be taken into account, it is like they are not there.

Reasons

So that’s why I’m trying to get the 320x240px snapshot drawn in the 320x320px white image. The reason of why I need it to be displayed in the frame buffer preview is to use the dataset editor to take the dataset for the Object Detection to detect the letters.

I hope i explained the problem in a more detailed way. If there is something you don’t understand or I badly explained, I will re-explain with pleasure.

kwagyeman · May 17, 2023, 10:36pm

Yes, I understand.

Question, are you doing FOMO or a classifier? Or both? A classifier would take a cropped image of each letter. FOMO would just find all the letters location.

Also, given the lens warp I suggest you use: img.lens_corr(1.8) to dewarp the lens. The CNN will have an easier time without the lens warping.

As for why set did not work… yeah, the images aren’t the same size. So, it has to do something more complex.

import sensor, image, time, os

sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.B128X64)
sensor.skip_frames(time = 2000)
img2 = sensor.alloc_extra_fb(128, 128, sensor.RGB565)

clock = time.clock()

while True:
    clock.tick()
    img = sensor.snapshot()
    img2.draw_rectangle(0,0, 128, 128, (255,255,255), fill=True)
    img2.draw_image(img, 0 ,32)
    img2.copy(copy_to_fb=img) # overwrite the old image and replace with the new one
    print(clock.fps())

Kosu · May 18, 2023, 6:50am

Yes I’m using FOMO, in particular the model “FOMO (Faster Objects, More Objects) MobileNetV2 0.35”.

kwagyeman · May 18, 2023, 5:54pm

I see. Okay, well, I guess the above is all you need. FOMO spits out the class.

Regarding the lens shading issue. I noticed this. I will offer an SDK function later this year when I have time to write it to deal with correcting the lens vignetting. The camera sensor can actually do this itself… but, it’s not documented well.

Kosu · May 18, 2023, 10:01pm

If I run the code, I get this error “OSError: The new image won’t fit in the target frame buffer!” when executing the copy() method.

kwagyeman · May 18, 2023, 10:29pm

Yeah, because it’s bigger than the frame buffer…

Try this:

import sensor, image, time, os

sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.B128X64)
sensor.skip_frames(time = 2000)
sensor.set_framebuffers(1)
img2 = sensor.alloc_extra_fb(128, 128, sensor.RGB565)

clock = time.clock()

while True:
    clock.tick()
    img = sensor.snapshot()
    img2.draw_rectangle(0,0, 128, 128, (255,255,255), fill=True)
    img2.draw_image(img, 0 ,32)
    img2.copy(copy_to_fb=img) # overwrite the old image and replace with the new one
    print(clock.fps())

You have to set the number of frame buffer to 1 because by default the system will allocate 3 buffers for triple buffering to give you the best FPS. When it does this it can’t grow the frame buffers anymore. If you just have 1 then it can automatically figure out that it can grow the frame buffer so it will allow you to do so.

Sorry this is complex. When you start using more than 1 buffer in user space how our memory allocation works can’t really be hidden. Here’s what’s happening BTW.

 ┌──────────────────────┐
 │                      │
 │                      │
 │   Frame buffer 1     │
 │                      │
 │                      │
 ├──────────────────────┤
 ├──────────────────────┤
 │                      │
 │   Frame buffer 2     │
 │                      │
 │                      │
 ├──────────────────────┤
 │                      │     
 │                      │
 │                      │
 │                      │
 │   Frame Buffer 3     │
 │                      │
 │                      │
 ├──────────┬───────────┤
 │          │           │
 │          │           │
 │          │           │
 │          │           │
 │          ▼           │
 │                      │
 │                      │
 │    Free Space        │
 │                      │
 │                      │
 │                      │
 │                      │
 │                      │
 │          ▲           │
 │          │           │
 │          │           │
 │          │           │
 │          │           │
 │          │           │
 ├──────────┴───────────┤
 │                      │
 │                      │
 │   Allocated FB       │
 │                      │
 └──────────────────────┘

So, in triple buffer mode your frame buffer can’t grow. If you just had one then it could. Your allocated frame buffer grows from the bottom.

When you execute a function it will then allocate RAM in that free space up until it hits the frame buffer area where images from the camera are stored. Note that our DMA driver in triple buffer mode is automatically updating frame buffers in the background to give you the fastest FPS while the processor works on 1 of the three.

We used this stack system instead of malloc because it’s MUCH faster to allocate and deallocate RAM in large amounts without wasting any space.

The MicroPython heap where MicroPython objects are stored is a heap though. Because of this you don’t want to allocate large objects in it since it can become fragmented and etc. We keep the frame buffer memory area separate for this reason.

Topic		Replies	Views
img.difference() question OpenMV Products	2	3861	July 18, 2017
Trying to create a reference picture on the fly OpenMV Products	1	434	May 25, 2022
How to implement new sensor to openMV? OpenMV Products	6	2560	August 23, 2018
Take a snap and save image in while loop,the ide report error OpenMV Products	8	198	October 20, 2023
How to only send self-allocated framebuffer to OpenMV IDE? OpenMV Products	1	2278	December 18, 2018

Place a snapshot into a new white created image with different resolution and display it in the Frame Buffer preview

Goal

Resolution problem

Our idea

Reasons

Related Topics