The DCT code can’t be pulled out of the JPEG code easily. However, the FFT code can easily spit out a 2D FFT of the image in 1 line of C code.
Um, basically, I can make a method that would take an ROI. It would then compute the FFT of that ROI. Note that the function won’t be able to operate on a large image so you’ll need to use the pool() methods to reduce the resolution (or just pick a small res).
Anyway, after the FFT runs… yes, I can grab the value of the highest bin… or, I can grab the value of the highest 3x3 bins. It’s unclear what value you want exactly out of the FFT.
Um, think about the FFT as a 2D mountain map. I can find the largest value… that would tell you the most dominate frequency. But that’s not what you want. You want the power of the high frequencies… so… it kinda sounds like I should just return a histogram of the FFT. There’s already a lot of code for histogram stuff built-in like finding the max, mode, median, etc. Then you can do anything you please.
So, anyway, I’ll take the FFT of the image, compute the magnitude, and then create a histogram of the FFT. You will be able to select the number of bins in the histogram. As for inspecting the histogram you’ll be able to re-use all of the already built-in histogram stuff. For example, if you’re goal is to get the max frequency we have a method called get_percentile() which you pass a percentile to… like 0.99 and it returns the value that’s at the 99% of the histogram distribution. Basically the max value that’s not an outlier. A larger value for this means the image is sharper… a lower value means the image is not sharp.
Thanks-- pity about the DCT methods not being exposed. So, FFT to the rescue!
FFT of a region-of-interest sounds perfect. Histogram sounds hugely useful and very flexible. I am actually rather excited about your comments. Great suggestions, thank you. For sure, I owe you a coffee, and I look forward to shining a spotlight on this camera and the really impressive libraries and IDE.
One thing to keep an eye out for: There is a possibility that a differentiation or Laplacian might be needed ahead of the FFT, but let’s try just the straight FFT first and see if its high-order bin(s) detects focus (etc) by itself. I expect it will do the job.
Hey, getting your feature in is on the queue of things to do. I should be able to completely tackle it by the weekend. Maybe sooner. It’s not hard but I’m working on phase correlation stuff right now.
One of the answers talks about taking circular rings from the FFT space and putting that in a histogram. This is likely what I will do for the FFT. The question is will this work for you?
Given the diagonal cut image you were talking about you’ll see a sinc() like function in the frequency domain histogram. I.e some bin will have a high peak falling off to other bins around it. You’ll then be able to measure using the mode() to determine the image quality. The mode will fall as the image is blurry.
For a natural image the FFT will look like white noise though with the DC value being the biggest.
I’m a little concerned about frame rate, but this looks like a broadly useful approach.
My actual application will be crunching the results into a scalar figure of merit, higher = sharper/better image. I want to do this quickly, to facilitate optimization of the image (focus, mostly). The slant-edge is a classical target for such things since it serves as a step-input to the imaging system, meaning it contains all frequencies, and we’re basically evaluating the rolloff due to the optics. More high-frequency content = less rolloff. Optimize the HF content and you’ve optimized the image, at least within the region-of-interest. The approach you found in that link strikes me as useful for any image… which would seem to be a good thing!
Thank you again. I am hoping you are finding this interesting too!
Note, you will be able to high pass the image with morph() using a high pass filter kernel if you want to do the FFT on the delta image versus the normal image.
Hi, I’ve been working on your request… and I’m not sure I’m getting what I need from the FFT for doing anything useful. Attached is a binary that does the FFT. Please let me know if you get anything useful from it.
Whats happening here…
2D FFT
Mag
FFTShfit (puts zero freq components in the center of the 2d image versus the edges)
Linpolar (note that I weight the stuff on the top of the image (low frequencies) less than the bottom parts, if I don’t do this you just get the DC components since they are so much bigger).
Draw the FFT on screen.
# Hello World Example
#
# Welcome to the OpenMV IDE! Click on the green run arrow button below to run the script!
import sensor, image, time
sensor.reset() # Reset and initialize the sensor.
sensor.set_pixformat(sensor.RGB565) # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.B128X128) # Set frame size to QVGA (320x240)
sensor.skip_frames(time = 2000) # Wait for settings take effect.
clock = time.clock() # Create a clock object to track the FPS.
while(True):
clock.tick() # Update the FPS clock.
img = sensor.snapshot() # Take a picture and return the image.
print(img.get_fft_histogram().get_statistics(), clock.fps())
The get_fft_histogram() method returns a histogram of the fft of the image. Each bin is equal to the sum across the image from the left to right. There’s a bin by default per row in the image.
Pass “phase=True” to get the phase from the FFT instead of the magnitude.
…
Anyway, if you can get back to me quickly I’ll be able to merge this feature into the next release. Otherwise it will have to wait. It will be useful to me if you could do “img.save(“file.ppm”)” on a few test images and send me the ppm files. Please don’t save images directly from OpenMV IDE because those are of lower quality due to jpeg compression. Also, I’m daring the FFT on screen right now for debug. The final image won’t do that.
From what I’ve done so far, radial sums of the frequency bands of the FFT don’t seem to yield useful results. But, I don’t have test patterns to compare with.
Not sure what to do here. I thought the magnitude of radial bands would fall off as you get away from the top of the image but that doesn’t appear to be the case. I could switch things around and sum bins by changing the x/y axes. This would make each bin represent the frequencies from 0 to inf in each rotational direction. This seems like the only information the FFT encodes.
…
Note, I’ve tried looking at the log() of the magnitude. That just gives you an FFT image where everything is relevant and you can’t really see any details. The power away from the center does not fall off smoothly. It’s basically like one giant mount from the DC part and then little peaks on all the other frequencies. Taking the log just makes the big DC peak as small as the smaller frequency peaks. It does “whiten” the FFT space… but, not much else.
…
Finally, note that solving your problem will help me solve my phase correlation issues too with being able to create a system capable of undoing rotation/scale issues. openmv.zip (2.1 MB)
Do, “img.linpolar(reverse=True)” to get back the FFT unshaped.
Mmm, so, one thing I noticed… the strength of like… two arms from the center increase the sharper an edge. And the rotation of those two arms changes based on the camera direction.
Anyway, it seems like the 2D FFT kinda forces you to have to pick a direction to sum up magnitude on. You can’t just look at all directions it seems. It’s quite clear that a sharp edge on the image moves all power into two radial directions from frequencies 0 to inf. Note that half of the FFT is the same as the previous half.
Here are some first fruits. I’m finding the mean statistic to be a decent measure of focus, at least of a small portion of the image dominated by a black/white transition element. Perhaps there is a better statistic to use, but this is a start! I’m not doing any differentiation, high-pass filtering or anything of the sort yet.
I tried the rotation of the histogram bins like I was mentioning and that didn’t work at all. I guess what I posted previously is the best we’ll get for this. To be honest… it’s kinda useless except for getting how sharp the image is.
By no means useless. It is the beginnings of all sorts of analyses of image quality, and yes sharpness is really a key quantity. The circles may not have panned out (yet) but there’s all sorts of stuff that we haven’t tried yet, such as differentiation. MTF is, from what I’ve seen, documented as commencing with a differentiation and there’s probably a good reason for that.
Now the foundation is there! Thank you!
I’ll upload the images as (and how) you requested in a bit.
I’m finding I can’t save to .ppm if the image is GRAYSCALE. Python complains that the images aren’t .ppm if they’re grayscale. So, unlike what I used for the FFTs in my earlier post, I’m using RGB565 in all the attached files. These save as .ppm without issue.
==> From my playing so far, the mean statistic seems to be much less useful as a measure of sharpness in RGB565. It does not vary as much with defocus as it does for grayscale, for which it peaks very satisfyingly when focus is achieved. In the short term, grayscale seems the way to go for my purposes, at least using this statistic. Perhaps there is another statistic (or prior filtration) that will work better for RGB? In any case this may illuminate why you’ve been unimpressed with the utility of this in your own testing.
(Images are in the .zip. The forum won’t let me upload .ppms.) focusing images.zip (118 KB)
Super, note you have to save as a .pnm for grayscale. See this for more info: Netpbm - Wikipedia (Yes, I have plans to allow you to save any format to any format. Wrote the code to but it’s not been committed yet).
