Nobody is watching the watchman

Take an inventory of how your system tells you something is wrong. Push notifications through a companion app. E-mail. A chat bot. Maybe a siren. Every one of them lives inside the patient, an arrangement any hospital would query at the planning stage.

The trouble with a house that reports its own health is that it does so with the same mouth it uses for everything else. When that mouth stops, the silence sounds exactly like good news. For a long time we mistook it for good news. This is the story of teaching a building to have someone outside worry about it.

Now notice what they have in common: they are all processes inside the thing that might fail. You have built a smoke detector that is powered by the fire. The push integration, the SMTP client and the chat bot all run in the same process as the automation engine. When it stops, they stop, and nothing sends a final message, because sending is exactly the capability that just died.

Invert the logic

Stop trying to send a message when things break. Instead, send a message continuously while things are fine, and let the absence of that message be the alarm. This is an old idea from industrial control, a dead-man's switch, and it is a better fit for home automation than almost anything else you can do in an afternoon.

every 5 minutes:  GET https://<external-monitor>/<id>

external service: if no ping for (period + grace) → alert me

The critical property: at failure time, nothing inside your house has to work. No template renders, no integration loads, no network stack in your box does anything at all. Silence is the signal.

Make the alert carry a diagnosis

One heartbeat tells you something is wrong. Several tell you where. Put one on the automation controller, one on the router, and one at any remote site. Then the pattern of which went quiet is the diagnosis:

Two failure modes in one channel

Most monitoring services accept a "fail" variant of the ping URL. That gives you a second state for free: alive but unwell. Route a short list of internal health signals to it, but keep the list short. A false alarm at 3 a.m. destroys trust in the channel faster than a missed one.

The most valuable entry on that list is your own monitoring layer. If the thing that watches your sensors freezes, that fact should escalate out of the box, rather than dying quietly alongside everything else.

Details that decide whether it works

• The alert channel must not route back through your system. If the monitoring service notifies you via your own chat bot, you have rebuilt the circular dependency you were removing.
• Grace longer than a normal restart. Otherwise every update pages you, and you will mute it. Measure how long your restarts actually take, then double it.
• The repair must beat the page. If something self-heals within ten minutes, do not alert at eight. Alert at thirty, and only about faults that self-healing could not fix.
• The ping URL is a credential. Anyone holding it can forge a heartbeat and silence the alarm. Keep it out of version control.
• Test it by deliberately failing. An untested alarm path is indistinguishable from a broken one, and both look exactly like a quiet week. Batch that testing into one window, scattered tests look exactly like a flapping system.

On the remote end

Cheap consumer routers often can't persist a cron entry across a reboot. If the device has no boot hook, have your controller re-assert the schedule periodically, the same way you would re-assert any other remote state it forgets. Then accept the residual: if the remote device reboots while your controller is down, nothing re-arms it. That fails toward a false alarm rather than a missed one, which is the right direction.

The paperwork has to live off-site too

A dead-man's switch tells you the house is down. What happens next depends on paperwork: the restore runbook, and the one piece of configuration no backup contains, the host's own network setup, without which a freshly restored machine boots onto the wrong network and cannot reach its backups. For a while, the runbook describing how to rebuild this machine lived on this machine. It was well written, kept current, and stored in the one place guaranteed to be unavailable in the scenario it was written for. Filed, so to speak, inside the patient.

Wherever your heartbeat monitor lives, put a copy of the recovery notes next to it. The alarm and the instructions belong on the same side of the disaster.

There is a school of thought that says a system this self-aware is a sign of anxiety. We prefer to think of it as the building finally getting a next of kin.

The watchman cried wolf

In mid-September, five weeks after the heartbeat went in, the language model that reviews the house overnight filed a worrying finding: ten timeouts in one day against the external monitoring service, and a suggestion that the safeguard might be quietly failing. That is precisely the sentence everything above was written to make impossible, so we took it seriously and went to look.

The log was indeed full of cries. Twenty-three timeouts across four days, ten of them on the day that prompted the finding, every one an ERROR, and all but one stamped twenty-five or twenty-six seconds after a five-minute boundary — the exception, a minute past the boundary, was a second attempt. That detail is the whole story. The heartbeat makes up to three attempts, ten seconds apart, with a twenty-five-second limit on each, so a timeout twenty-five seconds into a cycle is attempt one giving up. What the log never said, because nothing was writing it down, was what happened next. We counted. Only once in four days did two attempts fail in the same cycle, and not one cycle lost all three. The retry had rescued every check-in in those four days, and the safeguard had been working the entire time.

Measured live, the path was healthy: name lookup instant, handshake a tenth of a second, eight probes out of eight returned. About two percent of first attempts stall, the same occasional hiccup our e-mail, chat bot and parcel tracker all see on the way out of the flat. That is exactly why the ten-reader review put a retry here in the first place; before it existed, slow evenings dropped pings outright, and since it went in the service on the other end has noticed nothing at all. So we left the limit where it was. It had already gone from ten seconds to twenty-five in August, and anything that takes longer than that to fetch a few bytes is a stuck connection rather than a slow one; a longer wait would only delay the retry that fixes it.

Log the cycle, not the attempt

The fault was not in the ping. It was that a whole failed cycle, the one event that would actually matter, had nowhere to show up. Each attempt reported its own small failure and the cycle as a whole reported nothing, which is like a hospital charting every cough and never the diagnosis. The rule that auditors need the same discipline as alarms turns out to have a mirror image: an auditor will also cry wolf at noise nobody ever meant it to count.

So the heartbeat now stamps a timestamp helper on every successful check-in, and a sensor counts the minutes since: under five on an ordinary day, past fifteen and the monitoring service is already in its grace period with nothing to show for us. A cycle that loses all three attempts writes one greppable word to the log — the marker reads DEADMAN-CIKEL-PADEL, "dead-man cycle failed", because the night shift greps in the house's own language. That word, or the sensor past fifteen, is now the only thing it may report. The bullet above says the repair must beat the page; this is the same rule pointed inwards, where the repair must also beat the log entry.