OpenMV as Autoguider for astrophotography?

OpenMV related project discussion.
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Tue Jul 28, 2020 3:21 am

You guys seem to be the perfect hardware for a headless astrophotography autoguider

Background: the Earth spins, so the stars in the sky are always moving. If you want to take a photo of something really dark and far away in the night sky, you need to do long-exposure photos but since the stars are moving, you need to move the camera in the opposite direction of the Earth. You can buy a star tracker for this but some star trackers allow you to add a optional secondary camera, an autoguider camera, to move more accurately, you get a sharper image.

Problem being they are $150 for just the camera, plus another $50 for the guide scope, plus you need a laptop to run the software.

I'm playing around with the idea of making all of this cheaper, or at least get rid of the laptop.

Anyways, the algorithm should be very simple. Find a dot, follow the dot by giving the star tracker (or EQ mount) one of four basic signals: left, right, up, down. (it's controlled via a ST4 cable, just 4 signals, active low, very simple)

Usually during the initial setup, it'll do a calibration. It locks onto a dot, tells the tracker/mount to go left for a few milliseconds, and see how much it moved and in what direction. This should be easy too.

My options are:
* smartphone with a telephoto lens and a Bluetooth nRF51 doing the ST4 signalling (it sounds cheap but I'd rather still have my phone in my hand all night)
* raspberry pi and HQ camera and a CS telephoto lens (price getting expensive)
* ESP32 cam (where do I get lenses for these? The camera's quality is usually terrible, dead pixels and such. Plus, I'm not excited about writing image processing in C++)

OpenMV hits a good price, has the super telephoto lens, runs image processing, has GPIOs. Optionally I can add the LCD shield or the Bluetooth shield.

Now I have no idea what the OpenMV camera would actually see when it's pointed at a star. This is why I'm here on this forum.

My next bit of research revealed that OpenMV is limited to 0.5s of exposure. This should be OK, I think... my 350mm camera lens would see star trails at 2.5s. 0.5s should be fine if I point it at a super bright star like Vega.

The max gain is 32. No idea what that looks like in terms of brightness and noise. I'm honestly not sure how to convert gain to ISO, I know "base ISO" is supposed to be gain of 1, right?

I'm guessing the super telephoto lens is fixed focus at infinity? As long as the star looks round, my idea should work, I just need a point or circle detector algorithm.

Can somebody with the super telephoto lens take a couple of photos of the night sky to share with us? Extra points if you can run the circle detector and provide a plot of XY position frame-to-frame so we can get an idea of the motion noise. (don't worry, we can filter this noise out, the earth doesn't change speed)

Has anybody experimented with adding a heat spreader to the bottom of the OpenMV camera sensor PCB?

Thanks, hope you all had a chance to see Comet NEOWISE
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Tue Jul 28, 2020 10:18 am

Hi, you're going to need to buy the camera to get the answer to most of these questions.

Anyway, you can make the max exposure longer than 0.5 seconds. If you turn the PLL on the camera off and do other ticks you can make it very long since we give you direct hardware control of the system. E.g. slow the clock way down, etc.

As for the camera to use, the global shutter module will give you the most precise picture. However, it's expensive. The default camera is cheap and has large pixels for good low light response. So, it will probably do the job too.

Tracking the stars can be done with find_blobs().
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Tue Jul 28, 2020 4:57 pm

Hi thanks for the info so far. I ordered the plus version just now.

Calculations tells me that it'll be a pretty crappy guider, 11.6 arc sec per pixel. I might end up turning this into a polar alignment camera instead. Gonna need to write a fake plate solver.

Testing is going to be a pain lol a few nights ago a firetruck showed up to kick away all the photographers and astronomers from a spot on Skyline.
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Wed Jul 29, 2020 5:55 am

hey I just read about
The OpenMV Cam has two memory areas for images. The classical stack/heap area used for normal MicroPython processing can store small images within it’s heap. However, the MicroPython heap is only about ~100 KB which is not enough to store larger images. So, your OpenMV Cam has a secondary frame buffer memory area that stores images taken by sensor.snapshot(). Images are stored on the bottom of this memory area. Any memory that’s left over is then available for use by the frame buffer stack which your OpenMV Cam’s firmware uses to hold large temporary data structures for image processing algorithms.

If you need room to hold multiple frames you may “steal” frame buffer space by calling sensor.alloc_extra_fb().

If sensor.set_auto_rotation() is enabled this method will return a new already rotated image object.

Note: sensor.snapshot() may apply cropping parameters to fit the snapshot in the available frame buffer space given the pixformat and framesize. The cropping parameters will be applied to maintain the aspect ratio and will stay until sensor.set_framesize() or sensor.set_windowing() are called.
The H7 Plus has enough RAM to perform find_blobs() on the full 5MP resolution image, right? I don't need high frame rate.
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Wed Jul 29, 2020 10:30 am

Yep! It will be super slow. But, no problem.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Wed Jul 29, 2020 3:35 pm

I guess I'll find out how slow when it gets here

My plan is to sort the blobs by brightness. It looks like I can get a list of blobs easily. My next questions:

Is there a limit on the number of blobs detected? There could be... ahem... a lot of stars.
If a blob’s bounding box area is less than area_threshold it is filtered out.

If a blob’s pixel count is less than pixel_threshold it is filtered out.
I need the exact opposite of that. I need to keep small blobs but remove big blobs. I am guessing that I have to use threshold_cb? Or do I feed it a negative number or something? (this isn't very important, a big blob would mean I need to retry the capture anyways)

I see I can get a circle representing each blob. What's the best way to determine the "brightness of a star"? I think I see a b_and() function that accepts a circular mask, and I can probably call get_histogram() to get the statistics, but that sounds like it'd operate on an entire 5MP image, whereas I only need to sum up like maybe 10 pixels. Would you suggest I just iterate over the rectangular bounding box with get_pixel() instead?

Is there a more lightweight call for get_histogram if I only need min and max? Would it be faster if I iterated over the entire image myself? Honestly I just want a function that tells me if I have at least one true white pixel and one true black pixel.
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Wed Jul 29, 2020 5:39 pm

Hi, just let find_blobs() run with no area/pixel thresholds and it will return all the blobs.

The callback method is useful however to stop before a blob is added to the output list and execute functions on that blob. You can target the ROI of get_histogram to lie on the blob and then filter the blob out by it's brightness.

E.g. get_histogram(roi=blob.rect())
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Thu Jul 30, 2020 5:05 am

Do you guys have any specs on the optical errors of the lens and the soldering planarity of the sensor? I need he view to be perfectly parallel with the scope holder, or find a way to calibrate away that error. So I want to know how straight the lens is and how level the soldering is on the sensor.

Pretty sure polar scopes are usually machined aluminum. Commercially available polar alignment cameras and autoguide cameras all look like machined aluminum.

Guide scopes don't really care about the parallelism since there must be a calibration routine anyways. I'm ignoring this use case for now.

I don't own a lathe. I have my own FDM 3D printer and at work we have a Form 2. My current plan is to just do as good of a job as possible and then run a timelapse capture session overnight, calibrate based on the centroid of the circles captured.

Plan B is to just swap with an already aligned polar scope, and see what the difference is. But just touching the whole setup after alignment is risky. This will need to be done during the day with a stationary object on Earth, not a star, so maybe I can just bolt the scope holder to something that's sturdier than a 3 legged tripod.
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Thu Jul 30, 2020 10:24 am

Haha, no.

There's a reason the camera is cheap... Because it's not calibrated. At all :). We could add two 0s. To the price for all of that if you want.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Mon Aug 03, 2020 4:31 am

I envisioned that the WiFi shield provide a preview of the image to a smartphone. It's gotta be full resolution for the exposure check. The actual plate solving bit can be just SVGs.

I am wondering if there are performance penalties. I am considering using a ESP8266 as a co-processor just to be the HTTP server if the performance penalities for the ATWINC1500 is too high.

Very simply... is the ATWINC1500 driver completely triggered by interrupts? Or is there a periodic SPI poll?

I'm sure you optimized the crap out of everything for your race cars but just wanna make sure.
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Mon Aug 03, 2020 11:39 am

Hi, the ATWINC1500 used to be bad. But, when doing the interface library updates for it I fixed a lot of issues with it. It works using 40 MHz SPI. Transmit is very fast and doesn't block. RX works but is not as performant. The WINC1500 cannot actually handle a large packet stream input from another device over WiFi without falling over itself. Generally, what happens is that for UDP messages are dropped. TCP works better for RX since the WINC1500 can slow the external device down. Transmit can do 15 Mb/s if you were just moving a large pre-allocated array. However, you should get about 7.5 Mb/s in practice on average.

Anyway, use TCP to do transmit. You should get above 5 Mb/s easily.
Nyamekye,
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Mon Aug 03, 2020 11:43 am

I'm sure you optimized the crap out of everything for your race cars but just wanna make sure.
Working on it. At the start of this year a lot of things weren't. Right now... SDRAM is as fast as it can be. And we've got the camera DCMI driver working as fast as it can. We will be adding double buffering soon to finish maximizing the performance out on that. Regarding I/O. When I made the interface library we fixed all issues with SPI/I2C/CAN/UART to get the best speeds out of these interfaces and verified that the bandwidth is maxed out with a logic probe.

SD Bandwidth needs to be worked on still (will do after double buffering is enabled).
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Wed Aug 05, 2020 9:45 pm

Just got it in the mail

The M12 thread is really loose unless the lens is screwed all the way in, but if it's all the way in, it's definitely not focused right. This is a big problem but I might just solve it with teflon tape or thread locker. Or maybe I could stick an o-ring between the housing and the lens?

Really wishing the WiFi shield did not come with the pins soldered

Sensor was dirty on arrival. Sensor cleaner for my camera works wayyyy better than rubbing alcohol. But I had to clean it while the view was live.

Wish I could do more today but its cloudy.

Could you please send me the code that enables longer exposure times than 0.5sec? Thanks.
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Wed Aug 05, 2020 10:56 pm

Hi,
The M12 thread is really loose unless the lens is screwed all the way in, but if it's all the way in, it's definitely not focused right. This is a big problem but I might just solve it with teflon tape or thread locker. Or maybe I could stick an o-ring between the housing and the lens?
You have to focus the lens and then tighten the screw. I think we will switch to a screw on M12 thread lock ring.
Really wishing the WiFi shield did not come with the pins soldered
Lots of folks wish the opposite! :)
Sensor was dirty on arrival. Sensor cleaner for my camera works wayyyy better than rubbing alcohol. But I had to clean it while the view was live.
Hi, we get the camera sensor cleaned by the factory now days. Where did you buy this unit? Our official modules all are cleaned now.
Could you please send me the code that enables longer exposure times than 0.5sec? Thanks.
What model are you using? The OV7725? Just use sensor.__write_reg() to change settings:

https://cdn.sparkfun.com/datasheets/Sen ... OV7725.pdf

sensor.__write_reg(0x0D, 0x00) # Turn off PLL
sensor.__write_reg(0x11, 0x80 + 3) # Divide input clock by (3+1) * 2 # can go up to 63

This will really slow down the clock... note that you you make a single image take longer than 3 seconds our driver will say the camera timedout. You can make that longer if you need it.

After which you can use sensor.set_auto_exposure(false, exposure_us=1000000)

To get a 1 second exposure.

Here's how exposure works: https://github.com/openmv/openmv/blob/m ... 725.c#L403
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Wed Aug 05, 2020 11:38 pm

This is the H7 Plus, so the sensor is a OV5640, I don't see any documents about the registers at a glance. Is it the same code though?

The lens I'm using is the telephoto lens. What screw? There's a small hole on the top of the housing but it's not a screw. I know the default lens has that ring that sets the focus but that doesn't exist on the telephoto lens.

The camera is from Elmwood, SN appears to be 10888
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Thu Aug 06, 2020 12:04 am

Ah, yeah, the H7 plus has a screw lock ring and not the regular set screw. For our first production run with it the sensor might not have been totally clean.

You can take the screw lock off the default lens and put it on the Telephoto screw.

As for the H7 plus. Give me a second.
Nyamekye,
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Thu Aug 06, 2020 12:12 am

Here's how exposure works:

https://github.com/openmv/openmv/blob/m ... 640.c#L791

The OV5640 is actually a very complex camera SoC. Much more so than the OV7725. As such, it has a very complex clock tree. If you work out the math for the clock we are feeding it... it's actually running at 800 MHz internally.

https://cdn.sparkfun.com/datasheets/Sen ... asheet.pdf

You're going to want to edit this stuff:

0x3036 SC PLL CONTRL2 -> Bit[7:0]: PLL multiplier (4~252) Can be any integer for 4~127 and only even integer for 128~252
0x3037 SC PLL CONTRL3 -> Bit[3:0]: PLL pre-divider 1, 2, 3, 4, 6, 8

The default firmware pushes the camera literally as fast as it can go before the image quality starts to really get bad. For an easy slow down that our firmware understands just lower the PLL mul value. It's by default set to 0x64. So, just lower that to like 0x8 and that will slow all clocks down by 8x. The auto exposure code then will calculate the requested exposure time based off that being lowered.

sensor.__write_reg(0x3036 , 0x08)
pyb.delay(500)
sensor.set_auto_exposure(false, exposure_us=1000000)
pyb.delay(500)
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Thu Aug 06, 2020 12:58 am

You can take the screw lock off the default lens and put it on the Telephoto screw.
No I can't, the correct focal point is like 1mm away from the bottom, there's no thread in that region, and the ring is too thick anyways. I think some teflon tape would work. I would permanently glue it down once I make sure that thermal expansion/contraction due to night weather doesn't affect the focus too much.

The code you've posted is a bit wrong, the address 0x3036 should be 0x3037, then it worked. I can push it past 0.5 sec but I can't seem to get anything slower than 1 sec now, even setting it to 2 sec would reply back with about 1.06 FPS in the serial terminal. I thought the timeout was 3 sec?

I turned off my room lights and it's picking up some street lamps... damn clouds but hey we need some rain lol
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Thu Aug 06, 2020 2:21 pm

Yeah, the timeout in snapshot is 3 seconds.

That datasheet I liked to is how we developed our firmware. Took a lot of effort because it's not really enough documentation. Pretty much for any of these modernish camera SoCs now days big companies just use FAEs to develop customer products and there really isn't documentation about how things work. It's all kept internally to the company.

Anyway, just try out different PLL values to slow the camera down.

If you want precision control get the global shutter camera. It has clear register setting documentation so you can control it precisely.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Fri Aug 07, 2020 4:53 am

Ok weird stuff happened, I swear I pushed the PLL multiplier down to 66 at my desk and the whole thing worked fine and I almost got to as long as 3 seconds worth of exposure

Then... night time... I couldn't push it past 1.4 sec because the PLL wouldn't let me change multiplier without doing the initialization error

But I got this shot of some power lines, oh, and Jupiter
6_30_1400000.jpg
Plate solving with Astrometry http://nova.astrometry.net/user_images/ ... #annotated which looks correct, confirmed it with Stellarium

(camera is upside down due to 3D printed tripod mount)

Still plenty of more work to do, including that WiFi AP problem I posted to viewtopic.php?f=3&t=1986

Thanks for all the help so far

COOOOOOOL you can see Callisto and Ganymede with this thing
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Fri Aug 07, 2020 1:28 pm

Awesome!

Also, besides lowering the PLL you can increase the divider output. The PLL has to be within certain freqs to lock. So, you can't set it to anything. But, you can increase the output divider as much as you want.

You can also bypass the PLL.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Sat Aug 08, 2020 4:06 am

Tonight I actually took images of Polaris. I couldn't even see it with my eyes due to the light pollution, but the camera can definitely picked it up. I scripted something that iterated through all potential exposures and collected a bunch of images to run image processing on later.

It turns out, the image.Image(path) constructor is REALLY picky. It would not open the .jpg files that it saved, claiming "unsupported format". I used Affinity to save them as 8 bit RGB JPG with sRGB ICC profile and it still says "unsupported format". So I used MS Paint to save the files as 24-bit bitmap BMPs. These are some fat 15 megabyte files, 2592 x 1944.

Well... now OpenMV loaded the images, I run img.to_grayscale(copy = False) on it to save memory and to speed up blob finding. I run .find_blobs and get this exception: "Out of normal MicroPython Heap Memory! Please reduce the resolution of the image you are running this algorithm on to bypass this issue!"

That's weird... I can run the following code and not get the exception

Code: Select all

import sensor, image, time

sensor.reset()                      # Reset and initialize the sensor.
sensor.set_pixformat(sensor.GRAYSCALE)
sensor.set_framesize(sensor.WQXGA2)
sensor.skip_frames(time = 2000)     # Wait for settings take effect.
clock = time.clock()                # Create a clock object to track the FPS.

while(True):
    clock.tick()                    # Update the FPS clock.
    img = sensor.snapshot()         # Take a picture and return the image.
    hist = img.get_histogram()
    stats = hist.get_statistics()
    blobs = img.find_blobs([(stats.mean() * 3, 255)], merge = True)
    print("fps = %f    ,    blobs = %u" % (clock.fps(), len(blobs)))
(above code was not run on an image that actually contained stars, but I did manage to get like 30 blobs at one point, randomly waving the camera around my apartment)

So... either I'm loading the image into frame buffer wrong, or my demo code is about to explode the RAM and I just don't know it yet?

Anyways... coding in a park on a picnic table at midnight isn't that romantic, and the sprinklers came on while I was there... please help me run my code on the images while I am at home lol thanks

EDIT: oh yea I did try throwing gc.collect() everywhere, didn't help
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Sat Aug 08, 2020 11:12 am

We don't support opening jpg files. Just saving them. There's been a GitHub enhancement issue to support this for a while.

You are running out of ram because you tried to copy the image to the MicroPython heap. Please note that you are using a Microcontroller. Not a PC. We are able to do a lot but through tricks.

When you did the copy you need to assign it to the frame buffer which is 32 MB.

extra_fb = sensor.alloc_extra_fb(2592, 1944, sensor.GRAYSCALE)

img.to_grayscale(copy=extra_fb)
Nyamekye,
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Sat Aug 08, 2020 11:14 am

Read this:

https://docs.openmv.io/openmvcam/tutori ... chitecture

You tried to load the 10MB image to the 256KB MicroPython heap.

Be very careful when dealing with large images. You have to be very explicit about using frame buffers which are stored in the 32MB SDRAM.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Sun Aug 09, 2020 9:04 pm

to_grayscale(copy = extra_fb) doesn't work, the error says that extra_fb is an image object and the parameter "copy" is expecting an integer. I'm guessing I'm supposed to pass the pointer to the allocation, not the whole object?

anyways... instead of using to_grayscale, I did simply used get_pixel and set_pixel to do the image conversion. It takes like 5 minutes... but now, when I do find_blobs, it's saying "MemoryError", no other text.

Do I need to stream the bitmap into the sensor's frame buffer?

I'm seriously considering pointing the camera at a paper picture of the sky now... But even that's difficult with the telephoto lens.
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Sun Aug 09, 2020 9:10 pm

Hmmm,

to_grayscale() doesn't support frame buffer relocation with copy. I apologize.

do this:

extra_fb = sensor.alloc_extra_fb(2592, 1944, sensor.GRAYSCALE).replace(img).to_grayscale()

replace() copys an image from one frame buffer to another. So, it will transfer the main image to that frame buffer. Then to_grayscale() will reduce the frame buffer from RGB565 to GRAYSCALE.

It would be more efficient per say if copy supported targeting frame buffers. That's a to do.

https://docs.openmv.io/library/omv.imag ... ge.replace

Replace supports loading files too.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Mon Aug 10, 2020 2:06 pm

I think that worked, it says it found 14 stars pretty fast, I think the blob finding is fast enough for this.

Weird thing happened, I ran some code that looks like

Code: Select all

        for i in stars:
            cir = i.enclosing_circle()
            img.draw_circle(cir[0], cir[1], cir[2], 255, 1, False)
notice that the color is 255, and the img object is supposed to be grayscale. I save the image as another jpg, and the circles, they are... drumroll... YELLOW lol, it looks nice but I'm super confused why it's yellow. It's like the image became CMYK or something?

Interestingly, I found that sensor noise isn't a problem with this sensor. Light pollution is the major enemy. In soccer field nearby I really wouldn't push the gain past 50, histogram mean would be something like 55. On the top of the mountains, I can push it to 128 and still get perfectly black skies.

Honestly, at the soccer field, I couldn't even see the big dipper with my eyes, let alone the small dipper, soooo it's kind of a useless test case since nobody in their right mind would attempt astrophotography in that location.
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Mon Aug 10, 2020 2:57 pm

Hi,

Please note the difference between a grayscale image and an RGB565 image that is just grayscale. When you take 255 and treat that as an RGB565 byte reversed value you get yellow.

Use tuple notation (255, 255, 255) to avoid confusion. If you pass a number that number is treated as a RAW pixel.

You called that method in the original image you captured which is RGB565, not the grayscale copy in another buffer.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Mon Aug 10, 2020 3:10 pm

Hey is the cx and cy of a blob weighted center or just bounding box center?

The lens seems to be giving the stars a slight asymmetrical glow since the focus isn't perfect, the cx and cy isn't exactly centered on what should be considered the center. So I wrote some code to iterate only through the region of interest to find the weighted center instead of using cxf() and cyf(). Not sure if I'm wasting my time doing something that's done faster in the C backend?

edit: I'm using brightness as the weight, so I'm guessing you didn't do this in C already since it makes no sense for something that uses thresholding
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Mon Aug 10, 2020 3:47 pm

CX/Cy are the centroid (weighted)

They are based on the binary mask however. Not the brightness. If you want brightness just loop over the pixel region in Python using get_pixel(). The objects should be small so performance should be fine.

Note, there's a special parameter that gives you the x and y histogram of the blob. If you output these and use them you can handle larger blobs without a slow down. The centroid can be directly determined through the use of the x and y histogram of the thresholded blob pixels (i.e. only pixels that pass the threshold are added to the histogram)
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Wed Aug 12, 2020 1:24 am

I have an idea about calibration. The goal is to find the "center of rotation" when the camera is rotated.

For a polarscope camera, calibration is usually done by comparing two images, if in both images, I can identify Polaris, it means I have more than one star to work with. This is easy. See attached image, where X is the calculated center of rotation.

But... this is just a stretch goal. What if I could do this during the day?

Looking through the documentation. I can use find_displacement ? It says it can do translation but not rotation, or do rotation but not translation. I have no idea if I can... do it twice? Will the algorithm handle that? If I had two points and an angle, I can compute a triangle, with the third point being the center of rotation.

The other method would be keypoint matching, but it seems like the match object returns a list of points only for the second image, not the first? Or can I run the matches twice, the second time with the parameters swapped? This would only work if the returned two lists of match coordinates are sorted and can be correlated! I'm running the demo and I'm 99% sure this won't work...

Just a stretch goal, it might not even be more accurate than doing it at night. The lens would be fixed to focus on stars, any near by scenery would be blurry and feature extraction might not work well, or be inaccurate.
Attachments
rotation center.png
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Wed Aug 12, 2020 11:18 am

find_displacement() is phase correlation on the whole image. It can find rotation displacement. However, it's not designed for an FFT of more than 1024x1024 pixels.

I had tried to design the algorithm to handle both rotation and translation but was unable to get that to work. The translation part works well. The rotation and scale doesn't really.

...

I think you should just put the centroid of the starts on matrix and use the npy like library on board to work some linear algebra.

...

I don't quite know how you'd see the stars during the day.

...

All methods require strong features. If you don't have anything with corners to look at then you're out of luck. The sky will not cut it.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Thu Aug 13, 2020 3:19 am

that idea was meant for day time pointed at like a far away building for feature extraction

forget about that, bad idea

I'm actually 99% implementing the Python code. My HTTP server is fully working as well and I'm minimizing the amount of work that the Python has to do. Anything that can be done via Javascript on the smartphone will be done by Javascript instead. A lot of data is being sent out as JSON using ujson, and it makes it super easy for me to debug via just my computer.

BUT!!!!

find_blobs is throwing MemoryErrors! Only sometimes! "Out of normal MicroPython Heap Memory!"

I have added gc.collect() to many places

I have added micropython.opt_level(2) to every file

I am using micropython.const(expr) where I can

I have commented out most useless functions

I can print out the memory data using micropython.mem_info(True)

I might have the skills to dig into the firmware C code, maybe there are options in the makefile I can simply disable?

Can I simply find the malloc() that's failing and giving it a static buffer instead? Or is there a dynamic buffer that's failing?

Should I try cross-compiling py files into mpy?

What can I do that doesn't involve reducing the image resolution?

I have a lot of

Code: Select all

if self.debug: print("xxxxxxx")
, do those add up significantly on the heap?

Help me out here! Sooooo close!!!! After this I just need to do HTML and JS, no more Python!

Code: Select all

ERROR[681153]: <class 'MemoryError'>
Traceback (most recent call last): >> File "<stdin>", line 441, in main >> File "<stdin>", line 435, in task >> File "<stdin>", line 369, in solve >> File "star_finder.py", line 37, in find_stars
MemoryError: Out of normal MicroPython Heap Memory! Please reduce the resolution of the image you are running this algorithm on to bypass this issue!

stack: 1444 out of 64512
GC: total: 246080, used: 68848, free: 177232
 No. of 1-blocks: 1020, 2-blocks: 100, max blk sz: 1188, max free sz: 10123
GC memory layout; from 30003c90:

a whole bunch of stuff here

if you are curious, the latest code is up on https://github.com/frank26080115/OpemMV ... mv_filesys

EDIT: WOOOOOOT I compiled my own firmware that removed many struct members of the linked list node "find_blobs_list_lnk_data", and reduced "FIND_BLOBS_CORNERS_RESOLUTION" to 12, and also I'm not tracking 3 lists of stars any more, just 1 list, so the gc can collect older lists, I am now very watchful of what can be gc'ed

I also reduced the USB CDC buffer sizes from 512 to 128

The error seems to have stopped but I haven't stress tested it outdoors yet, but the report says 147 stars at one point (I'm pointing it at crap around my apartment, with some weird thresholds), which should work fine later on

Fingers crossed!

Also, it seems like burning a new firmware heats it up so much that the PLL freaks out

Oh and if you wanna see my firmware changes, https://github.com/frank26080115/openmv ... ightweight
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Thu Aug 13, 2020 12:07 pm

I don’t think you need to reduce the USB CDC buffers. But, okay. Weird that you ran out of heap. Must be a lot of stars. That’s quite a lot of allocations.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Fri Aug 14, 2020 12:43 am

The only reason why it saw that many stars is that I've disabled the checks against the histogram just for the sake of stress testing. Testing against my realistic samples, it's doing fine. I've also added a warning mechanism for "too many stars".

Serving a HTML page with a single threaded HTTP server is stupid annoying. I can't just use script tags or style tags, I actually have to use JS to load one file at a time, jquery and jquery-ui are both huge even when minified. I can't even have more than one on-going AJAX requests going at once so one AJAX completion has to trigger the next one.

I spent the first part of the day just researching ways to signal to a server that it can't have more than one socket open at a time, I never found a way. Even if the WINC supported multiple pipes, without multiple threads, it wouldn't even help. And even if the H7 is dual core, I need like 5 threads lol

I'm going to have to render my detected stars onto a SVG canvas just to save the bandwidth on the image transfer.
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Fri Aug 14, 2020 1:29 am

You can have more than 1 socket open at a time. The WINC supports 6 TCP sockets at once.

http://ww1.microchip.com/downloads/en/D ... 02796B.pdf

6 TCP
4 UDP

...

Maybe have the server do less work? Just provide the data to a client and have them do all the work?
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Fri Aug 14, 2020 4:29 am

Maybe have the server do less work? Just provide the data to a client and have them do all the work?
That is what I am doing actually. The problem is that if I have a single HTML page but two javascript files and one CSS file, when the HTML is loaded, 3 + 1 (for the favicon) simultaneous HTTP requests would go out at once.
6 TCP
In practice this isn't helpful if I am serving one request at a time. On each loop it would've served one request, it's never accepted another socket when I use socket.accept() on the second request. It's the most reliable when I stagger my requests to one at a time.

By the way, my method of loading JS one at a time is working great on desktop but failing on mobile so it's no-go. I'm going to use the Python to pack the JS and CSS into the main body of the HTML instead and serve it all as one big file.

I got as far as rendering the stars detected onto a SVG. All the data I would ever need has been packed neatly into JSON so that was actually easy.

Sometimes the AJAX would fail, something would go wrong with the WINC and it'd never accept another socket connection until a reboot. I added my own "watchdog timer" for it and reboots the WINC but this only works well in station mode, haven't tested AP mode but I think it'd lose the WiFi client so I'm pessimistic. Plus, my own phone would just go back to default WiFi if I lose the fake WiFi and everything would just not work after that.
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Fri Aug 14, 2020 1:09 pm

Mmm, I'm not a web programmer. I think making the camera serve complex webpages is more than I think it should be doing. The MCU makes more sense as just a device which is sending data via a 1 socket connection to a web server online which can supply that rich data. I wouldn't make it the web server.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Fri Aug 14, 2020 6:15 pm

I think you misunderstood. To keep the MicroPython as lightweight as possible. I need to make the HTML pages as complex as possible. We are in agreement on this point.

But the problem is that if the HTML has many JS files, the browser sends a bunch of HTTP requests all at once. Once the JS has loaded, the browser runs the JS, not the microcontroller.

The problem is only handling those HTTP requests. This only happens once on page load.

I just solved this problem by concatenating the JS files with the original HTML page. Concatenation is happening in Python, this way I can still keep separated JS files for organization. It's saving me soooo much headaches.

The MCU is running a loop running snapshots and find_blobs as fast as it can. It is sending a list of stars, image statistics, and some other info to the browser when the browser asks for it at regular intervals. It is accepting commands to change exposure and stuff. The image is a SVG generated by the browser using the list of stars and image statistics, the JPG is never actually transferred.

The Polaris pattern matching code is still Python but it's not heavy at all, only 20 possible stars to match. The bigger patter matching code will have to be JS and optional, 300+ star database, with each star having a "signature" made with about 4 other nearby stars. It wouldn't even load the database into Python.

The only problem left is that I still need to make sure that the JS only send one AJAX request at a time. So basically, I can't just say "update the exposure when the slider is moved", I have to make it do "when the next star list arrives, instead of requesting the next star list, check if the slider has moved and update if so, otherwise, request the next star list". I'm now doing this by using the UI events to push into a queue, and having the status update pop from that queue.

The hardest part is how to handle stuff when the WINC simply stops working.
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Fri Aug 14, 2020 7:13 pm

Yeah, I'm saying just send the raw data to a server in the cloud and have the camera just be a data generator.

Mmm, I guess however, this is remote in the wildnerness.

Okay, I see why it needs to be onboard.

Um, yeah, cool, so, if you see things that need to be fixed in the firmware let us know.

I will say that the WINC1500 routinely drops TCP sockets. If you see the RTSP code I wrote you need to pretty much assume the socket will break on you randomly and re-create it. I wish I knew how to fix this... when I was debugging the thing I noticed that after sending it data for a while it would just close the socket for not necessary a good reason.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Sat Aug 15, 2020 3:12 am

Gah, different night different problems.

You feel that heat today? San Mateo has got rolling blackouts. I drove up to Davis to a friend's place, who has a AC'ed house, carrying all my gear. The sky is great here.

Bad news 1 is that the memory is still a problem but now it seems to be more related to my way of setting the threshold. I can deal with this. I need to make that much more configurable and think about better ways of automating exposure and calculating or adjusting thresholds.

My old threshold is the image mean multiplied by 3. Which worked great on my light polluted simulations... but now it's obviously making the threshold so low that noise is becoming stars.

A potential technique might be to do an initial check of a smaller region just to see if it's getting an unexpected number of stars without blowing up the RAM... hmm...

The bigger bad news... The sensor's got hot pixels! Permanent white pixels, not just single pixels, blobs! And I'm also seeing amp glow

This is probably why the more expensive cameras are all aluminum... heat sink...

I can probably write some code to register the hot-pixels and filter them out. I might buy a few more OpenMVs just to see if the damage is permanent. My work has a temperature controlled chamber I use for testing battery safety and I can toss the camera in to see how it handles different temperatures.

I let it cool down and removed the WiFi shield and the pixels disappeared... I'm gonna buy a can of super-cold later lol

I researched small peltiers and they are all too thick and cost like $30...

Dude can I ask you to make me a for the camera? The WiFi shield is making it impossible to engineer any way of heat sinking the camera. A cable would let me mount the camera somewhere with more room and I can stick cooling solutions on it. I'm sure a lot of other OpenMV users would appreciate an extension cable too.
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Sat Aug 15, 2020 1:36 pm

Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Mon Aug 17, 2020 8:48 pm

Thanks, I guess I didn't see it on elmwood or sparkfun and thought it didn't exist.

I managed to be able to serve scaled and compressed JPG over HTTP, but a weird thing is happening. Sometimes the socket times out (OSError -6), file was being sent but not fast enough. But, it seems like this even would cause the camera to not detect a new frame has completed.

Of course this is using my own modified firmware with the non-blocking snapshooting so you might not be able to, or want to, help with this problem. Do you have any hints? I have no idea how exceptions (I have no idea how exceptions work in C) and interrupts (I think it might be a DMA interrupt being missed?) interact with each other.

The workaround is a simple timer that resets the camera sensor, it's not a fatal bug but very very annoying.

God damn this heat, sooo many hot pixels and the PLL won't let me do some values, stay cool man

edit: I did allocate extra frame buffer before scaling, I told the scale function to copy to the extra frame buffer, so it's not some buffer writing collision, I think...
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Mon Aug 17, 2020 10:40 pm

Socket error 6 has nothing to do with the camera. It’s thrown by the wind 1500 module randomly wherever it feels like and you have to close the socket. It’s just a general socket error. There’s no real recovery from it other than closing and reopening the socket.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Tue Aug 18, 2020 7:04 pm

I wrote a "closeall" function into the WINC and py_winc part of the firmware. Now, when my watchdog times-out, it doesn't reboot the WINC, it closes all the sockets regardless of whether or not they are open. It's kind of a dirty hack.

It's actually working great! This is much faster at recovering from a bad request, and doesn't drop the actual WiFi connection.

Haven't tested it in AP mode yet or on mobile.

I found a funny situation that can be encountered with the non-blocking snapshot function, I did a in-place scale of the image before a snapshot finished, and the next snapshot finished with an image at the scaled size, not the full size expected. The problem was solved with the extra framebuffer allocation. Definitely need to be more careful about that buffer.

I also changed the firmware to accept negative numbers for area_threshold and pixels_threshold, if the numbers are negative, then it'll filter out large blobs but keep small blobs. I also added height_threshold and width_threshold. This is an additional effort to prevent using up heap memory. Not sure if you want these changes officially? I can't do a pull-request right now with it, I stacked that change on top of other changes that you probably don't want.

Added a filtering for hot-pixels, just a list of X,Y coordinates, very easy.

Added a database of 2740 stars with 500 of them identifiable, had to fit it within 200kb of data and added about a second more worth of HTTP load time. The JS is doing the pattern matching.

Kinda feature complete, need to do more testing in AP mode and mobile, and more night time tests when the clouds are gone.

Thanks again
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Wed Aug 19, 2020 3:04 pm

Probably should have an explicit area max versus min. So, make a feature request and I can implement it better.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Sat Aug 22, 2020 4:39 am

Hey have you thought about selling something like https://www.ccdcmoslens.com/product/cs- ... s-adapter/ ? Or how about https://www.securitycamera2000.com/prod ... -Base.html ?

I am code feature complete. I'm doing only UI tweaks right now waiting for the smoke to clear. A CS-mount 25mm lens would be a f/1.2 like https://www.ebay.com/itm/Yohii-1-3-F1-2 ... 4269266267 , that'd give me almost twice the potential frame rate, or let me lower the gain by half.

Would all of this work out? Or am I missing something? Would it be impossible to focus? If the focal plane is too far forward I can 3D print a spacer, but if it's too far back then I'm screwed.

I ordered an entire second H7 Plus yesterday. I need the new one since mounting a CS mount or a M12-to-1.25inch adapter requires me to put the WiFi on the bottom instead of the top.

Oh and can you please tell me the exact screw sizes of all three screws that comes with the lens mount? The two black ones for mounting and the silver one for the lens threads? I'd ideally purchase a few bags from McMaster-Carr
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Sat Aug 22, 2020 10:36 am

Measure the screw sizes... with the camera you have. We are just buying cheap sub $1 M12 lens mounts. It’s not really engineered by us. The screws are just self tapping screws. You need not get exactly the right size for them.

Regarding C mount. I haven’t explored it. Given C mount lenses are more expensive it wouldn’t make a lot of sense for our market.
Nyamekye,
frank26080115
Posts: 32
Joined: Tue Jul 28, 2020 2:32 am

Re: OpenMV as Autoguider for astrophotography?

Postby frank26080115 » Tue Sep 01, 2020 6:09 pm

still living in a bbq, can't test outside still

I made a PCB design, https://github.com/frank26080115/OpenMV ... der-shield , it's a fork of your WiFi shield, but I shifted the pins up so I can have more room for the CCTV camera lens, plus I added a I2C port expander plus optoisolator for ST-4 control. I'm not going to sell it or anything but let me know if you have a problem with how I preserved your copyright text (I didn't change it at all)

I also figured out how to put compiled versions of my micropython code directly into the firmware flash as frozen mpy. But it took so much room that I had to remove a bunch of stuff, so I removed everything that had to do with neural networks, the FIR, and TV. It is my understanding that this will significantly reduce RAM being used as the execution will happen from flash, so I will be able to detect more stars before running into the heap memory exception.

I am doing documentation, the README is up on my repo along with some other educational pages https://github.com/frank26080115/OpenMV ... raphy-Gear , let me know if you don't want me to use that one photo or logo from your website
Last edited by frank26080115 on Thu Sep 03, 2020 6:50 pm, edited 1 time in total.
User avatar
kwagyeman
Posts: 4675
Joined: Sun May 24, 2015 2:10 pm

Re: OpenMV as Autoguider for astrophotography?

Postby kwagyeman » Tue Sep 01, 2020 7:04 pm

Hey, really great write up. I'm going to tweet it.

Awesome work!

Also, you are free to use our logos and etc. However, if you plan on selling the product you cannot use the logos in such a way that makes people think that your product is an official OpenMV product.

...

Yes, moving the scripts to flash frees up RAM since the bytecode is in flash now. Note on the latest firmware we moved the stack to the ITCM in the cortex m7 which makes the stack 64KB and frees up over 10KB for the heap. So, you should have a lot more RAM.

...

Finally, you are free to remix the board design too if you want. That said, it's not super easy to do the H7 Plus board routing however given the SDRAM.
Nyamekye,

Return to “Project Discussion”

Who is online

Users browsing this forum: No registered users and 0 guests