The first collection covered the system attacking itself and the monitoring lying about its own health. This one is mostly about the layer underneath: the tools you reach for while fixing something, the dependencies you did not know you had, and the moments where two independent sources agreed with each other and were both wrong. If the first collection was the system attacking itself, this one is mostly friendly fire.
The first collection of horror stories was about the house. This one is about the tools we used to fix the house, which turned out to have opinions, dependencies and, on one memorable day, a shared delusion. If part one taught humility, part two teaches paranoia, which is humility with a checklist.
The tools you fix it with
The one-liner that empties the file it is reading
A configuration file needed one line changed. The edit was written as a single expression: open the file for writing, and pass it a list built from reading that same file.
open(path, "w").writelines([ ... open(path).read() ... ]) # do notOpening for writing truncates immediately, before the argument list is evaluated. By the time the read happens there is nothing left to read. The expression is not wrong in a subtle way; it destroys its own input as its first act, and then dutifully writes the empty result.
It blanked a reverse-proxy configuration and took the web interface down until it was restored from a backup that existed by luck rather than policy. Read into a variable first. The extra line is not verbosity, it is the difference between an edit and a deletion.
The guard that matched its own comment
A patch script had to add an import to several files, and needed to be idempotent, so it checked whether the import was already present before adding it:
if "text_sanitize" not in source: # add the importThe same patch also inserted an explanatory comment mentioning
text_sanitize, above the line it was guarding. On the run that mattered, the
comment landed first. The guard then found its own documentation, concluded the work was already
done, and skipped the import in three files.
The result was not a crash on the next line. It was a crash later, in a rarely-taken branch, in scripts that had reported success. It surfaced only because a test asked "and is anything actually stored?" and the answer was no.
The command that runs somewhere other than where you think
Shell-command sensors on this platform execute inside the application container, which has no container runtime tooling of its own. A health check written to inspect containers, pasted from a working shell session, therefore did nothing at all, correctly configured, correctly scheduled, executing in a place where the command does not exist.
Dependencies you did not know you had
The local setup that quietly depends on the cloud
A smart plug controlled entirely locally, its cloud integration disabled on purpose. Local control needs a per-device key. That key changes every time the device is re-paired (after a factory reset, or a Wi-Fi change) and the only way to retrieve the new one is the vendor's cloud API.
So the cloud project has to be kept alive indefinitely, unused, purely so that local-only operation remains recoverable. Delete the account you no longer use and the device becomes uncontrollable the first time it is reset, locally as well. The independence was real, but it had a renewal clause.
The cleanup whose blast radius included the evidence
An automated mailbox purge, entirely reasonable in isolation, had a scope that included the electronic invoices another importer depends on as its only source. The bills would have kept arriving and being deleted, and the importer would have kept reporting nothing new, truthfully.
The statistics that survive every cleanup
Long-term statistics are not purged by the retention setting that governs everything else. Two consequences, both surprising in the moment: shrinking the database does not shrink them, and a bad value written into them outlives every purge you throw at it. They have to be repaired specifically, by name.
The timestamp helper that is born lying
A fresh date-time helper does not initialise to empty. It initialises to today at midnight, a claim one reviewer disputed and the platform's source settles: the no-restore fallback is today's date with a zeroed clock. Every "when did we last see this?" latch therefore starts life asserting that the event happened earlier today, which is exactly the answer that suppresses the alert you built the latch for.
Two sources, both wrong at once
The appliance that reported Off while it was running
A washing machine, watched two ways on purpose: the manufacturer's cloud service, and a smart plug measuring actual power. Redundancy, correctly designed, different vendors, different transports, different failure modes.
Mid-cycle, the cloud reported the appliance as Off (it disconnects and the last known state is served as current). And in the same window the plug's power meter froze, it reports on change, and it had stopped reporting, holding 3.6 W for 156 minutes.
Two independent sources, simultaneously confident, simultaneously wrong, in opposite ways. The lesson is not "add a third source". It is that every source needs a freshness check of its own, because the failure that matters is not disagreement, disagreement is visible and therefore useful. It is two sources that have both stopped and are still answering.
The long-press that was a factory reset
An air-quality sensor has a status LED that is slightly too bright at night. The integration exposes no way to turn it off. The device has one button. Holding the button turns out to be the factory reset, and the device left the network, taking its pairing and history references with it.
A small story, but the shape is common: the absence of a software control is information. If the vendor exposes five read-only channels and no LED setting, the LED is not configurable, and the button is not a hidden menu.
Two rules that came out of all this
The common thread, again, is confidence. Every one of these went wrong while something was certain. We are working on being less certain, and it is going slowly, because we are fairly sure we are right about that too.