Image not rendering correctly on Arduino Giga Display

Arduino Boards
1

I am trying to display a 800x480 JPEG image loaded from a file on an Arduino Giga Display shield using the code below.

test
test800×480 66.7 KB

import image
import display

lcd = display.DSIDisplay(
    framesize=display.FWVGA, portrait=True, refresh=60, controller=display.ST7701()
)

img = image.Image('test.jpg', copy_to_fb=True)
lcd.write(img)

while True:
    pass

It displays a cropped image in the portrait orientation.

test_p_t
test_p_t1920×1473 159 KB

If I set the portrait=False in the display.DSIDisplay initialization it fills the display with 4 pieces of the image.

test_p_f
test_p_f1920×1547 198 KB

I can use the following code to transpose the image but it slows down the fps terribly so I wanted to use previous code with correct rendering/orientation.

img.to_rgb565()
img.replace(transpose=True)
2

This is a flag for the display driver to rotate the display settings/config, you should not change it (it’s a config for the display). This display can’t be rotated it’s 480x800, the only way to display that image is with img.replace(transpose=True)

3

Hi, you should reformat the jpeg to the right orientation.

(Unfortunately, since the display is rotated from what you’d want normally the MCU has to do a ton of extra work to display things)

4

Hi @kwagyeman @iabdalkader

Now I am using the code below.


while True:
    buf = get_jpeg_frame(); # some method to fetch streaming jpeg data frame
    img = image.Image(800, 480, image.JPEG, buffer=buf, copy_to_fb=True)
    img.to_rgb565()
    img.replace(transpose=True)
    lcd.write(img)

It works but it is slow. I assume to_rgb565 method uses software jpeg decoder.
Now I have enabled the hardware jpeg decoder and built the firmware from source.
The code below works flawlessly and it is considerably fast! I believe this time lcd.write method must be using the hardware jpeg decoder.

while True:
    buf = get_jpeg_frame(); # some method to fetch streaming jpeg data frame
    img = image.Image(800, 480, image.JPEG, buffer=buf, copy_to_fb=True)
    lcd.write(img)

But when I include the code for transposing the image (the same code I used with software jpeg decoder), it stalls after a few frames. I checked and found it always stalls at the 4th frame. I tried to skip first few frames (to verify if any frame may have some issue) but it still stalls at the 4th frame wherever I start. How is the data being handled by the img.to_rgb565() and lcd.write(img) methods?

5

To RGB565 and LCD write do the same thing internally. They decode the jpeg to RGB565. Both use the hardware decoder. All jpeg logic passes through the hardware decoder when enabled.

while True:
    buf = get_jpeg_frame(); # some method to fetch streaming jpeg data frame
    img = image.Image(800, 480, image.JPEG, buffer=buf, copy_to_fb=True)
    lcd.write(img)

So, the above works with the hardware decoder? What code stalls?

There might be bugs in that PR to enable the hardware decoder since it hasn’t been merged.

6

Yes, the code below works with hardware decoder enabled.

while True:
    buf = get_jpeg_frame(); # some method to fetch streaming jpeg data frame
    img = image.Image(800, 480, image.JPEG, buffer=buf, copy_to_fb=True)
    lcd.write(img)

But the code below does not work and stalls at img.to_rgb565() after a few frames.

while True:
    buf = get_jpeg_frame(); # some method to fetch streaming jpeg data frame
    img = image.Image(800, 480, image.JPEG, buffer=buf, copy_to_fb=True)
    img.to_rgb565()
    img.replace(transpose=True)
    lcd.write(img)
7

What does:

while True:
    buf = get_jpeg_frame(); # some method to fetch streaming jpeg data frame
    img = image.Image(800, 480, image.JPEG, buffer=buf, copy_to_fb=True)
    img.to_rgb565()
    lcd.write(img)

Do - I expect it to stall? Both to_rgb565 and write will use the jpeg decoder to decode the image to RGB565.

8

Yes, the code below also stalls at img.to_rgb565().

while True:
    buf = get_jpeg_frame(); # some method to fetch streaming jpeg data frame
    img = image.Image(800, 480, image.JPEG, buffer=buf, copy_to_fb=True)
    img.to_rgb565()
    lcd.write(img)

And, this also stalls:

while True:
    buf = get_jpeg_frame(); # some method to fetch streaming jpeg data frame
    img = image.Image(800, 480, image.JPEG, buffer=buf, copy_to_fb=True)
    img.to_rgb565()
9

Okay, will try to look at this and fix the issue. Shouldn’t take long. There’s a lot of issues with the PR I need to address though to actually merge it into main which I can’t tackle right now.

10

Hi, can you provide the image file you are using? Also, how did you get the hardware decoder working. Did you branch off my PR or enable the hardware decoder in jpegd.c that’s already in the firmware?

I remember having to fight a stalling issue in my PR that kept causing the thing to crash. I believe I fixed it in my PR though.

11

Hi @kwagyeman,

I forked the master branch and enabled the hardware decoder in the jpegd.c file.
I built it using the git Actions for the Arduino Giga R1 only. You can use the firmware from the latest Development Release:

12

Yeah, okay, I know what the issue you are running is. The jpeg decoder hardware is just locking up. I ran into this myself. In my PR I was able to fix this.

Can you share the image you are using? I need to reproduce.

13

Here is the image file: https://github.com/metanav/openmv/blob/master/test.jpg

You can reproduce it using the code below:

import image
import display

lcd = display.DSIDisplay(
    framesize=display.FWVGA, portrait=True, refresh=60, controller=display.ST7701()
)

count = 0
while True:
    img = image.Image('test.jpg', copy_to_fb=True)
    img.to_rgb565()
    img.replace(transpose=True)
    lcd.write(img)
    count += 1
    print(count)
14

Thanks!

15

Hi @kwagyeman

Did you get a chance to merge the PR?

Thanks!

16

It’s going to take a while. I might have the changes done this week.

1 Like
17

Hi, I fixed my PR and it’s working. The attached binary should unblock you:

import image
import display
import time

lcd = display.RGBDisplay()

count = 0
clock = time.clock()  # Create a clock object to track the FPS.
while True:
    clock.tick()  # Update the FPS clock.
    img = image.Image('test.jpg', copy_to_fb=True)
    #img.to_rgb565()
    #img.replace(transpose=True)
    lcd.write(img)
    count += 1
    print(count)
    print(clock.fps())  # Note: OpenMV Cam runs about half as fast when connected

This is my code for the OpenMV Cam Pure Thermal which has a screen like the Portenta. I get 46 FPS with the above code, and then 16 FPS if I enable to_rgb565() and transpose(). to_rgb565() without transpose is 36 fps. Anyway, the system does not get stuck.

firmware.zip (1.1 MB)

This firmware is for the Arduino GIGA. It is rebased off main with my PR as of 10/25/2023.

…

I need to refactor quite a few things with this PR before it will be officially merged.

1 Like
18

Thanks a lot, @kwagyeman! I am going to test it soon.

19

Hi @kwagyeman,

I’ve tested the firmware you shared and it works really well. Thank you! The problem with the Arduino Giga R1 Display is we need to transpose the image that is a bottleneck now. Can you suggest how to improve the performance of the transpose? Can we use CMSIS DSP matrix transpose methods?

20

It’s already been speed up. It used to be 5 FPS.

The solution is that you need to not transpose. Rotate the images before you send them to the system. Transpose completely butchers the memory architecture. It’s literally just the processor copying bytes. But, it’s a row access per column. So, it’s extremely inefficient memory wise.

An alternative is for write() to support transpose. This is a todo. It would speed things up by making it able to do this for you. However, I cannot promise a fast ETA on that.