[Solved / Archived] Firmware changes to enhance reliability

I guess you saw the "high reliability mode" from this topic. I would like to reiterate that we never used that name. The special mode we implemented in firmware after that discussion is called "guaranteed wake mode", and it only does exactly that. Giving the extra chance to access the device remotely dosn't mean the system is more reliable. We do not use that name because we don't want to cause such kind of misunderstanding. There is no such a mode that can magically make the system more reliable, otherwise we would have used it as the default mode already.

The "holding TXD high to prevent power cut" issue only exists in firmware revision 1 and 2. Since firmware revision 3, even if the TXD stays high after the shutdown, or the system crashs during the shutdown, the power will still be cut after the timer is due. You can confirm this by doing some simple tests.

If your Witty Pi 4 are with firmware revision 1 or 2, then you need to upgrade their firmware.

You can also change how the firmware works by modifying the firmware, and make sure it suits your needs.

Okay; can you help us better troubleshoot this issue then? 

Like I said these systems seem to shutdown correctly in the logs (see below) but sometimes every 10 - 30 days or so will just not come back online. When we physically visit the device the pi LED is red (so it has power) but has shut down, the Witty has not cut the power and the only way to get it out of this situation is to physically disconnect and reconnect the power.

The device then comes online fine (obviously showing a reconnection of power in the logs) and works fine again across multiple daily shutdown/startup cycles before seemingly for no reason doing exactly the same.

The system is running v7 firmware and also had the guaranteed wake mode enabled (with 12 hours set) so it should have also had it not correctly ran the schedule turned on after 12 hours. It's as if it gets stuck in a state where as it doesn't cut the power, it then never boots the devices properly and never see's pin 17 to show boot.

[2024-09-01 07:44:59] Shutting down system because scheduled shutdown is due.
[2024-09-01 07:44:59] Halting all processes and then shutdown Raspberry Pi...
[xxxx-xx-xx xx:xx:xx] Witty Pi daemon (v4.21) is started.
[xxxx-xx-xx xx:xx:xx] System: Raspbian GNU/Linux 10 (buster), Kernel: Linux 5.10.17-v7+, Architecture: armhf
[xxxx-xx-xx xx:xx:xx] Running on Raspberry Pi 3 Model B Plus Rev 1.3
[xxxx-xx-xx xx:xx:xx] RTC offset register has value 0x00
[xxxx-xx-xx xx:xx:xx] Seems RTC has good time, write RTC time into system
[xxxx-xx-xx xx:xx:xx] Writing RTC time to system...
[2024-09-05 12:00:43] Done 🙂
[2024-09-05 12:00:43] Firmware ID: 0x26
[2024-09-05 12:00:43] Firmware Revison: 0x07
[2024-09-05 12:00:43] Current Vin=12.51V, Vout=5.05V, Iout=0.64A
[2024-09-05 12:00:43] System starts up because power supply is newly connected.
[2024-09-05 12:00:48] Send out the SYS_UP signal via GPIO-17 pin.
[2024-09-05 12:00:49] Pending for incoming shutdown command...
[2024-09-05 12:00:49] Schedule next shutdown at: 2024-09-06 07:45:00
[2024-09-05 12:00:49] Schedule next startup at: 2024-09-06 08:00:00

Could the guaranteed wake mode be amended so that before a 'gauranteed wake' event it calls CutPower and then powers back up again?
Thanks
Jay

The "guarenteed wake mode" can not help you on this. The logic of this mode is implemented in the sleep() function in firmware, while your Pi was still powered and its firmware was not in sleep mode. A Pi in that status can not trigger the guarented wake, because it is not in sleep yet.

We have no longer seen similar issue after using firmware reversion 3 or above.

It will be difficult for you to troubleshoot it, because it only happens once every 10 days more.

You may consider modifying the firmware, remove the logic that you do not need, and add some code to make sure the power will be cut.

A proper power cut should contains these steps:

systemIsUp = false;
cutPower();
sleep();

The scheduled shutdown is implemented in the processAlarmIfNeeded() function.

It is still recommended to have some delay before cutting the power, but you don't have to rely on the timer1.

I've looked through that function, would this modification essentially be best placed after this line, and replace the emulatebuttonpress? That way at least we can keep the register tracking of reason for shutdown etc.

else if (canTrigger && !alarm2HasTriggered && overdue_alarm2 >= 0 && overdue_alarm2 < 2) { // Alarm 2: shutdown

Can you think of any other reason this might be happening, that we could troubleshoot without doing physical firmware mods?

We had another one today we've visited to cut the power and it came straight back online with these logs.

So I understand the logic correctly, assuming the firmware is newer than v3 (these all are) - on shutdown, it no longer waits for TXD to go low, it just waits for the shutdown wait timer and then cuts power? Is there anywhere else it could be getting stuck?

If we put the shutdown timer to 0 seconds, would that help - or could cause other issues?

[2024-09-10 20:00:00] Shutting down system because scheduled shutdown is due.
[2024-09-10 20:00:00] Halting all processes and then shutdown Raspberry Pi...
[xxxx-xx-xx xx:xx:xx] Witty Pi daemon (v4.21) is started.
[xxxx-xx-xx xx:xx:xx] System: Raspbian GNU/Linux 11 (bullseye), Kernel: Linux 6.1.21-v7+, Architecture: armhf
[xxxx-xx-xx xx:xx:xx] Running on Raspberry Pi 3 Model B Plus Rev 1.4
[xxxx-xx-xx xx:xx:xx] RTC offset register has value 0x00
[xxxx-xx-xx xx:xx:xx] Seems RTC has good time, write RTC time into system
[xxxx-xx-xx xx:xx:xx] Writing RTC time to system...
[2024-09-16 11:24:04] Done 🙂
[2024-09-16 11:24:04] Firmware ID: 0x26
[2024-09-16 11:24:04] Firmware Revison: 0x07
[2024-09-16 11:24:04] Current Vin=13.56V, Vout=5V, Iout=0.79A
[2024-09-16 11:24:04] System starts up because power supply is newly connected.

@insideoutgrp I don't have a clue in mind. If we can already imagine how it could get stuck, we would have reprouced the issue in lab and also fixed it.

As I mentioned, the scheduled shutdown is exactly defined in this code block:
https://github.com/uugear/Witty-Pi-4/blob/main/Firmware/WittyPi4/WittyPi4.ino#L890-L898

I can't tell you what is the better way to modify it -- we think we are using the best way already. We implemented this firmware with many functionalities, if you don't need all of them, it is indeed possible to simplify the code and make the logic you need more straightforward. Also if you want to add some code to debug the firmware, it is almost impossible without (temporarily) removing some functionalities, because the current firmware is already using up the 8kB program space in MCU.

If you remove the emulateButtonClick() function call, the software (daemon.sh) will not know the firmware is trying to shutdown the Pi, and no "sudo shutdown -h now" command will be executed. In such case, no matter when you call the cutPower() function, it will not be a graceful shutdown. That equals to directly unplug the power supply. It is not healthy to your Pi, and it could even damage the SD card in use.

@insideoutgrp I want to add that, if for some reason the SYS_UP signal does not reach the MCU, the firmware will not be in correct state to handle the incoming shutdown event, and similar issue might happen.

The SYS_UP signal is sent from the Raspberry Pi, via GPIO-17 to the MCU, and it is sent automatically by the daemon.sh during the boot.

If you look into the daemon.sh, on line 182~192, you can see how it is done:

# indicates system is up
log "Send out the SYS_UP signal via GPIO-$SYSUP_PIN pin."
gpio -g mode $SYSUP_PIN out
gpio -g write $SYSUP_PIN 1
sleep 0.1
gpio -g write $SYSUP_PIN 0
sleep 0.1
gpio -g write $SYSUP_PIN 1
sleep 0.1
gpio -g write $SYSUP_PIN 0
sleep 0.1
gpio -g mode $SYSUP_PIN in

In order version of software, the GPIO-17 was toggled for once to send out one SYS_UP, now it does it twice and two SYS_UP signal are sent in a row. We made such modification because we realized the importance of SYS_UP signal and we want to make sure it will be received by the MCU (firmware).

Maybe you would like to make the SYS_UP signal even easier to be received, by increasing the sleep time (e.g. from 0.1 to 0.3) between setting GPIO-17 high and low. Or even sendign one more SYS_UP signal to be sure.

When the SYS_UP signal is received by the firmware, the white LED blinks quickly. With the current software you will see it quickly blinks twice during the boot.

You may also want to double check the GPIO-17 connection.

If the sys-up isn't received, would there be a way to tell in the logs etc - as the Witty seems to be all communicating okay.

Would one way of proving it be to remove the shutdown -h now command from the utilities.sh that way unless the MCU cuts the power the system would remain on?

I don't think that could prove it. 

Removing the shutdown command from the software only makes graceful shutdown impossible.

It may be possible to keep a bit in the register to indicate the SYS_UP has been received. However I am not sure if we still have the program space for this logic.