Every system that runs long enough acquires a small collection of stories it would rather not tell. These are ours: every one happened on one installation, every one was invisible for a while, which is the only reason they are worth writing down. A system that breaks loudly teaches you nothing you would not have learned anyway. They are grouped by the kind of lie involved and told with the affection one develops for old injuries.
Part one: the system attacking itself
The logger that logged its own logging
An automation triggered by error-level log events, whose job was to collect them for a nightly review. It had a noise filter so it would ignore its own chatter. The filter lived in the conditions block.
Conditions are evaluated synchronously, inside event dispatch. So when an error arrived while some other template was halfway through rendering, this filter rendered nested inside that render, and the framework raised an exception about re-entrant rendering. That exception was logged. At error level. Which re-triggered the automation.
Roughly 215,000 events per hour. 1.65 million database rows and 6.4 GB overnight, on a disk with 28 GB total. Disk full by morning, core down for four hours.
mode: queued with a
maximum was already set. The crash happens before the run is queued, so the limit never
sees it.The first fix moved the identical filter into the action body, on the reasoning that it runs in its own task there and cannot render re-entrantly. That reasoning held for three days. Then the same loop returned, through the filter in its new home, and the fix that has actually held is in the field guide: no template in the decision path at all.
An explicit loop-breaker string in the ignore list and a recorder exclusion for that event type stay in as well, so no future storm can reach the disk. Not because three layers are elegant, but because each of the first two had, in its day, also looked sufficient.
The command that blinds every alarm you own
There is a convenient service that reloads everything at once. On this installation the notification group is a legacy YAML construct, and reloading everything tears down the notify platform and re-registers it, over several minutes, during which every notification fails silently. Measured: 177 failed notification calls over 39 minutes.
Every watchdog in the house notifies through that group. So the command you run to deploy a monitoring improvement is also the command that switches off all your monitoring, for longer than most incidents last. Reload the specific domains instead; they are individually safe.
The leading cause of memory pressure was the diagnostician
Symptom: the web interface crawls, pages time out, the system feels dead. Meanwhile the local API answers in about ten milliseconds. That combination, slow over the network, fast locally, is rarely an application problem. Here it was swap. Serving assets and doing TLS gets stuck behind disk I/O while everything already in memory stays instant.
The largest discretionary memory consumer turned out to be the long-running AI assistant session being used to investigate it, at over a gigabyte. The fastest remedy was to restart the terminal add-on it was living in. There is a certain dignity in diagnosing a problem and discovering the answer is to leave the room.
Part two: when the monitoring is the broken thing
Sensors that freeze at “healthy”
Nine polled sensors stopped updating and held their last value for between two and a half and three and a half hours. No error, no unavailable state, no log line. Two of them were watchdogs:
- an integration-health sensor, frozen at
0, meaning “nothing is broken”; - a radio-silence sensor, frozen at
0, meaning “we just heard from the network”.
Both were, for those hours, structurally incapable of firing. Not misconfigured. Not thresholded wrong. Reporting good news with total confidence, because good news was the last thing they had managed to say.
They froze in groups (several stopping within the same second, at a few distinct moments) and it did not correlate with poll interval: a 45-second sensor died while a 180-second one stayed healthy. The mechanism is still unidentified. That is the honest status, and it is why the mitigation is generic rather than clever: every polled sensor now has a staleness check (value older than three times its own interval), driven by a clock trigger, because clock triggers were verified to keep firing while the entire polled tier was dead.
The zombie container
An add-on reporting started. The container runtime reporting running, restart count zero, not killed for memory. The service inside it: dead. The integration that talks to it sitting in retry.
The logs were completely empty, and that emptiness was the only signal available. These add-ons run a supervision tree; if the actual services die but the supervisor process that is PID 1 keeps running, the container never exits, so the runtime, the supervisor and its own watchdog all agree that everything is fine. Nothing above process level can see it.
Detection now needs three things it should not need: listing processes inside the container and looking for a supervisor entry with no child beneath it, probing the port from another container, and reading the integration's runtime state, because the stored config file records only whether an entry is disabled, never whether it is working.
The liveness check that was measuring the wrong country
A router at a second property, checked every few minutes to prove that site was reachable. It went red at times that correlated suspiciously with problems at the first site.
Cause: that router's default IPv6 route pointed into the VPN tunnel. Anything it fetched without explicitly forcing IPv4 left through the other country. The check was dutifully proving that the path it was supposed to be independent of was working.
The tidy-up that would have silenced everything
An entity auditor that reports things which have gone missing was producing noise from a few known-absent entities. It offers an option to ignore entities in the unavailable state.
Which is to say: an option to ignore precisely the thing the tool exists to report. It was added and removed again before it took effect, in favour of narrow per-entity patterns. It is a footnote rather than a section only because of the order those two edits happened in.
Part three: devices that lie
The air monitor that only answers its own neighbours
Reachable by ping. Its port demonstrably open, and the packets demonstrably arriving, because the device returned port unreachable for closed ports, which means its network stack was alive and receiving. The application-level hello got no reply at all. A packet-counting rule on the return path counted zero: nothing was eating a reply, there was no reply.
The same request from a machine on the device's own LAN: instant answer. The application checks the source subnet and ignores anything else, a check invisible from every layer below it.
The fix cannot be local: it has to be source NAT on the far router, so the request arrives wearing a local address. Scoped to that one device, because a subnet-wide rule would quietly reroute other things. And re-applied every ten minutes, because the stock firmware has no persistent script hook and silently discards the rule whenever any setting is applied.
The television that requires you to switch the amplifier off
A 2010 set with early audio-return-over-control-bus. The audio link establishes only when the television wakes the amplifier from standby. Wake the amplifier yourself (from an automation, on the correct input) and you get a correctly selected input and no sound.
So the entire body of the automation is: turn the amplifier off. The television notices and wakes it, and audio appears about six seconds later. Every attempt to improve this by adding a power-on, an input selection or a retry loop made it worse, reliably.
The state that distinguishes working from broken is one word: playing means audio is flowing, on means the input is selected in silence. If your automation treats those as the same thing, it will report success into an empty room.
The ghost that stole an identifier
A new automation was given an ID that was unused in the configuration file. It was still held in the entity registry by an automation deleted long before. The new one therefore loaded under the old one's entity name.
Its own cooldown logic referred to itself by the name it believed it had. That name resolved to nothing, which evaluated as “never triggered”, which meant the cooldown was permanently satisfied. It worked. It just had no rate limit, and nothing anywhere said so.
The timestamp that invented an event
The history API returns UTC. A slice of it was read as local time during an analysis. One real event, two hours displaced, appeared as two events; a follow-up check was then run against the empty window; and a confident, wrong conclusion was written down and had to be retracted later.
The data was never wrong. Not one reading was missing or bad. The entire failure was a presentation layer two hours out, and it produced a finding that felt exactly as solid as a real one.
Part four: small print, large teeth
- A failing command in a VPN interface's post-up hook tears the tunnel down. A kernel setting that cannot be applied inside an unprivileged container exits non-zero; the wrapper treats any non-zero post-up as fatal and removes the interface. The tunnel was completely down until it was reverted, from a line that was supposed to be an optimisation.
- One netmask, applied to everything. A VPN add-on took the prefix length from the IPv4 address and applied it to the IPv6 address as well, producing a nonsense route. Always write the prefix length explicitly, even where it seems obvious.
- Shell commands that are not run by a shell. The command string is split into arguments directly, so braces and quotes in an inline JSON payload break the invocation. The automation reports success; the script never runs. Nothing appears in any log.
- Malformed JSON on a webhook does not produce an error, it prevents the trigger from firing at all. No 400, no log entry, no trace. The endpoint accepts the request and nothing downstream happens. Form encoding, for anything you cannot personally validate.
- Database compaction that requires free space equal to the database. Which means it fails exactly when the disk is full, which is the only time you want it. The variant that writes to a new file only copies live data and works.
- 2.7 GB of package-manager cache on a 28 GB disk shared with backups, contributing nothing, for months.
- The restore runbook lived on the machine it existed to rebuild. It documented the host network configuration that no backup contains, the configuration without which a fresh machine cannot reach the network to fetch the backup. Written down, kept current, and located in the one place guaranteed to be unavailable in the scenario it was written for.
The thread running through all of it
Almost none of these were caused by a wrong value. They were caused by a correct-looking answer arriving from a component that was no longer entitled to give one, a frozen sensor, a container reporting on a process it can no longer see, a health check measuring a path it was meant to be independent of, a timestamp in the wrong frame.
Which suggests the question worth asking about any piece of monitoring you own: if this were dead, what exactly would it say? If the answer is “the same thing it says now”, it is not monitoring anything. It is decoration with a good uptime record.
None of these happened because anyone was careless. All of them happened because something was clever. That is the lesson, and it is not a comfortable one.