Take an inventory of how your system tells you something is wrong. Push notifications through a companion app. E-mail. A chat bot. Maybe a siren. Every one of them lives inside the patient, an arrangement any hospital would query at the planning stage.
The trouble with a house that reports its own health is that it does so with the same mouth it uses for everything else. When that mouth stops, the silence sounds exactly like good news. For a long time we mistook it for good news. This is the story of teaching a building to have someone outside worry about it.
Now notice what they have in common: they are all processes inside the thing that might fail. You have built a smoke detector that is powered by the fire. The push integration, the SMTP client and the chat bot all run in the same process as the automation engine. When it stops, they stop, and nothing sends a final message, because sending is exactly the capability that just died.
Invert the logic
Stop trying to send a message when things break. Instead, send a message continuously while things are fine, and let the absence of that message be the alarm. This is an old idea from industrial control, a dead-man's switch, and it is a better fit for home automation than almost anything else you can do in an afternoon.
every 5 minutes: GET https://<external-monitor>/<id>
external service: if no ping for (period + grace) → alert me
The critical property: at failure time, nothing inside your house has to work. No template renders, no integration loads, no network stack in your box does anything at all. Silence is the signal.
Make the alert carry a diagnosis
One heartbeat tells you something is wrong. Several tell you where. Put one on the automation controller, one on the router, and one at any remote site. Then the pattern of which went quiet is the diagnosis:
| Controller | Router | Remote site | What it means |
|---|---|---|---|
| silent | ok | ok | The controller itself, process or machine |
| silent | silent | ok | Router, uplink or power at the main site |
| ok | ok | silent | Power or connectivity at the remote site |
| all silent | The monitoring service itself, or something very wide | ||
Two failure modes in one channel
Most monitoring services accept a "fail" variant of the ping URL. That gives you a second state for free: alive but unwell. Route a short list of internal health signals to it, but keep the list short. A false alarm at 3 a.m. destroys trust in the channel faster than a missed one.
The most valuable entry on that list is your own monitoring layer. If the thing that watches your sensors freezes, that fact should escalate out of the box, rather than dying quietly alongside everything else.
Details that decide whether it works
- The alert channel must not route back through your system. If the monitoring service notifies you via your own chat bot, you have rebuilt the circular dependency you were removing.
- Grace longer than a normal restart. Otherwise every update pages you, and you will mute it. Measure how long your restarts actually take, then double it.
- The repair must beat the page. If something self-heals within ten minutes, do not alert at eight. Alert at thirty, and only about faults that self-healing could not fix.
- The ping URL is a credential. Anyone holding it can forge a heartbeat and silence the alarm. Keep it out of version control.
- Test it by deliberately failing. An untested alarm path is indistinguishable from a broken one, and both look exactly like a quiet week. Batch that testing into one window, scattered tests look exactly like a flapping system.
On the remote end
Cheap consumer routers often can't persist a cron entry across a reboot. If the device has no boot hook, have your controller re-assert the schedule periodically, the same way you would re-assert any other remote state it forgets. Then accept the residual: if the remote device reboots while your controller is down, nothing re-arms it. That fails toward a false alarm rather than a missed one, which is the right direction.
The paperwork has to live off-site too
A dead-man's switch tells you the house is down. What happens next depends on paperwork: the restore runbook, and the one piece of configuration no backup contains, the host's own network setup, without which a freshly restored machine boots onto the wrong network and cannot reach its backups. For a while, the runbook describing how to rebuild this machine lived on this machine. It was well written, kept current, and stored in the one place guaranteed to be unavailable in the scenario it was written for. Filed, so to speak, inside the patient.
Wherever your heartbeat monitor lives, put a copy of the recovery notes next to it. The alarm and the instructions belong on the same side of the disaster.
There is a school of thought that says a system this self-aware is a sign of anxiety. We prefer to think of it as the building finally getting a next of kin.