Endless Flashing White Light When Solar Powered (Memory Fault?)

wantafanta123 · March 25, 2021, 7:02pm

Hey there, I am still using the custom board I made as well as the custom firmware I made in this post, and I was able to create the custom firmware.bin files and get past my slow development problem. However; I have ran into a new issue that is relatively peculiar.

In my application, I am having the camera capture an image once per minute, then upload that image to a server over a modem. This board is connected to a Li-Po battery that is being charged by a solar panel and charger circuit. The board works fine for a few hours but eventually just starts flashing a white light as though the memory was corrupted. I am then forced to flash the board with the bootloader.dfu file as well as the firmware.bin file, I have not found another way to get the board un-bricked.

Important Findings:

When disconnecting the power and rebooting - it keeps flashing the white light
When the board powers on with or without a good power source, the white light stays on for a few seconds (this seems to be a sign that the flash memory is being reformatted)
When connecting the bricked board to a reliable power source - it keeps flashing the white light
When connecting to a reliable power source from the start - it will work correctly for at least 2 days and I have not seen any issues. (this leads me to believe that when the battery is low and the board is powering on and off over and over again, the memory is being corrupted?)

Thank you so much for all of your help, you solved my other problem and hopefully this will be my last one haha

Here is a link to what is happening: OpenMV Camera flashing - YouTube

kwagyeman · March 25, 2021, 7:28pm

How am I supposed to debug a custom board and custom firmware?

Ask a specific question please.

kwagyeman · March 26, 2021, 4:24pm

Hi, I understand if I’m being less than helpful here… but, please understand… you are saying to me that you took our hardware design/software and built your own custom system on it.

That’s cool. But, then you say it’s modified from our design and that you modified the firmware.

That’s cool too. Finally, you say that things are working and you need help…

The problem is that you’re own you’re own though at this point. An example, forgetting to put a RC delay on the STM32H7’s reset pin can cause it not to boot. A little detail, but, if you were building your product in contract with us we’d tell you about issues like this.

To note, you’re also asking for help support for a product that you aren’t paying for help support per-say… while I try to help everyone, you should expect that we can’t really help you effectively if you didn’t engage us before the product design.

iabdalkader · March 27, 2021, 12:03am

Why don’t you try debugging the board in that “bricked” state and see what’s going on with a debugging image (DEBUG=1) ? BTW if it’s flashing an LED then some code is still running to flash that LED, so it’s Not exactly bricked, you need to figure out where it’s failing. Note that white light, which actually looks green to me but anyway, if it’s the same as our RGB LED blinking white that’s the error handler which gets called with a text error message. If you break there that’s a very good clue to start with. It also tries to log an error to the storage before flashing the LED, if by bricked you mean it doesn’t mount the storage you could dump that flash sector and/or QSPI and mount it as fat image with loop device and read the plain text message.

kwagyeman · March 27, 2021, 12:21am

If you built a SWD debugger into your board you should be able to trace execution there too.

Note that we are going to start producing OpenMV cams soon with standard ARM jtag headers to make issues for folks.

wantafanta123 · April 7, 2021, 1:47am

I think you are on to something here. Just to be sure, the DEBUG=1 parameter shoud be passed during the ‘make TARGET=board_target’ command?

for example: ‘make TARGET=board_target DEBUG=1’

Also when the light is blinking, the board cannot mount to a serial port or as a flash device. It looks like the board is attempting to reboot multiple times in very fast succession.

*** The flashing stops when I put the board in DFU mode and erase the flash memory sector so the flash memory may be getting corrupted when it boots. Is there any way to keep the board from reformatting the flash memory on boot if the files are corrupted?

kwagyeman · April 7, 2021, 2:16am

Yes, edit the code in main.c. There’s a reformat call if it can’t mount the disk. However, the files aren’t going to be readable by firmware.

iabdalkader · April 7, 2021, 3:01am

I think the filesystem gets corrupted in a way that it still gets detected valid initially, then it hardfaults when it actually tries to mount the filesystem… With the cam in that state could you try to insert an SD card ? This forces the code into the other path where it mounts the SD instead. This is possibly related:

github.com/micropython/micropython

stm32/storage: Prevent attempts to read/write invalid block addresses.

micropython:master ← andrewleech:storage_oob

opened 11:20PM - 04 Mar 21 UTC

andrewleech

+12 -4

This addresses the root cause of the QSPI deadlock issue seen in https://github.…com/micropython/micropython/pull/6707 The problem there was a corrupted (?) LFS1 partition on the qspi flash is causing it to attempt to read an invalid address. In this situation the QSPI driver does not have any checks for this and will get stuck waiting for data that never arrives. Ideally all the spi/qspi functions would also be extended to check the transfer error flag and return an error if this happens. An initial POC for this is available in https://github.com/micropython/micropython/pull/6992 but I expect a fair bit more work is needed here to make this suitable for inclusion. This PR attempts to make sure these bad-address deadlock conditions get caught early and never make it to the qspi driver in the first place. As such it's a much smaller change that should be safe by itself, though I don't know if there are other conditions that might also cause uncaught qspi issues.

And this:

github.com/micropython/micropython

WIP: stm32/qspi: Detect transfer errors and pass back up to spiflash driver.

micropython:master ← andrewleech:qspi_errors

opened 11:17PM - 04 Mar 21 UTC

andrewleech

+145 -71

This is an initial attempt to catch and pass up errors in QSPI Flash transfers. … It only has support for stm32 qspi driver. The regular SPI driver should also likely be extended to pass up errors (if these are detectable). I haven't looked at what's needed to make other ports compatible with this. Also, `qspi_read_cmd()` was already directly returning the read value as a `uint32_t`. Returning an error code instead of this is not a good way to handle this, likely the return value should be the error code like other functions here now have and the read value passed back in a pointer.

wantafanta123 · April 19, 2021, 9:54pm

Could you point me to the lines where the re-format happens? I went through the main.c file in src/omv/ and cannot find it. I am not the best with C so maybe I missed it.

kwagyeman · April 20, 2021, 5:20pm

github.com

openmv/openmv/blob/master/src/omv/ports/stm32/main.c#L605


      
                      }
                  } else {
                      if (pyexec_friendly_repl() != 0) {
                          break;
                      }
                  }
          
          
        nlr_pop();
              }
          }
          
          
if (usbdbg_script_ready()) {
              nlr_buf_t nlr;
              if (nlr_push(&nlr) == 0) {
                  // Enable IDE interrupt
                  usbdbg_set_irq_enabled(true);
                  #if OMV_ENABLE_WIFIDBG && MICROPY_PY_WINC1500
                  wifidbg_set_irq_enabled(openmv_config.wifidbg);
                  #endif
          
          
        // Execute the script.

wantafanta123 · April 26, 2021, 10:55pm

Thank you guys so much for the fast responses. This is a weird request, especially considering I asked how to avoid this, but it could be very helpful to me if I could somehow trigger the reformatting of the flash memory from the python scripts (like how it happens during the line you sent me).

This would be a way to completely reset the board if something goes drastically wrong

kwagyeman · April 27, 2021, 2:31pm

Use the low level disk methods in the pyb module and write a 0 block of data to the first sector of the flash. This will corrupt it causing a reformat to happen by our firmware.

Also, you can use the stm module to directly write to device registers and you can read the data sheet for how to erase a flash sector.

Topic		Replies	Views
OpenMV Problem OpenMV Boards	2	6772	May 26, 2019
Flashing Green, then white once, then dead... bricked? OpenMV Boards	6	2718	January 12, 2020
H7 camera flashes red when plugged in, then starts blinking white OpenMV Boards	5	1892	December 22, 2019
H7 external power (white plug) OpenMV Boards	2	283	December 5, 2022
Custom board issues OpenMV Boards	12	429	July 15, 2021

Endless Flashing White Light When Solar Powered (Memory Fault?)

Related topics