A worked example: verifying the wrong thing

Everything else on this site is a lesson stated in the abstract, which is a comfortable way to write about your own mistakes. This page is two incidents with timestamps, chosen because both are clean examples of a check that passed while verifying something irrelevant, and because both times the warning against exactly this mistake was already published here.

This is the story of an automation that was tested, verified, and wrong. All three are true. The lesson is not that testing is useless; it is that we tested the thing we had built rather than the thing we were afraid of, and the difference cost four hours of electricity and a small amount of dignity.

The automation that was supposed to help

A review found a plausible gap: nothing guaranteed the air conditioner was off when the flat was empty. The fix looked like a one-liner: when the alarm arms to away, send the unit an off. The unit is infrared, so the system cannot ask it anything; it only knows what it sent. We knew that, which is why we stopped to check something first.

The check that passed

The right question got asked: is that off a discrete command or a power toggle? Because a toggle sent to a unit that is already off does not keep it off. It starts it.

We opened the device's code table, confirmed that off appeared once, under its own name, distinct from every other row, and wrote "discrete, safe, verified" into the automation's description in capital letters, with the date.

Uniqueness in a config file is not a property of the protocol. That a command has its own row says nothing about what the hardware does with it. We had checked the label on the box, not what was in it.

What actually happened

The alarm armed, the command went out, and three minutes later the living room started cooling and kept cooling for eighty minutes, while a bedroom sensor in the same flat drifted down by two tenths of a degree.

Time	Living room (has the AC)	Control room
t − 90 min	28.0 °C	27.2 °C
t = 0	away mode → “off” sent
t + 3 min	27.0	27.2
t + 19 min	26.0	27.2
t + 75 min	25.0	27.0

The tell is not the drop; evenings cool. It is the comparison between rooms. That room is reliably the warmest in the flat, half a degree above the others. During the incident it was 1.2 degrees below them. Weather does not flip the sign on one room. A unit ran for four hours, cooling nobody, and the safety automation written to prevent exactly that reported success while it happened.

Why the check was worthless

The real test takes one line: compare the off frame with the known power-toggle frame. They were identical. The handset has a single power button; the captured "off" was always the toggle, filed under a name that implied otherwise. And three pieces of evidence were already in the repository:

A comment two lines above the command, reading in full: "sends the power-toggle frame (the captured 'off' code). On→off, off→on."
An assumed-power-state flag for this device plus two manual "resync" scripts. Nobody builds an assumed-state mirror for a device that takes discrete commands. The architecture was saying this is a toggle as loudly as architecture can say anything.
A warning on this very site, written days earlier, telling the reader to check for exactly this before writing an automation that fires while nobody is home.

We did not lack the information. We had written it down twice and published it once. What we did was stop looking the moment we found an answer we liked, which is a different failure and a much harder one to notice, because it feels exactly like diligence. The check you run is the one that is easy from where you are standing, not the one that could come back negative.

The fix: act only on positive evidence

The naive repair, "only send the command if we believe the unit is on", closes this bug and reopens the original: the belief is an assumption, and after someone uses the handset the assumption is wrong all week. So the automation sends the toggle only when the assumption says "on" and a measurement does not contradict it (that room cooler than the mean of three others, the signature of a running unit). If the sensors are unavailable, the assumption alone decides. If only the measurement says "on", nothing is sent, because a single failed-low sensor or a cold draught at arming time would otherwise send a toggle to an off unit: the same bug in a third hat. Guessing has a direction here, and it is the wrong one.

Separately, a level-triggered check runs every fifteen minutes while the flat is empty and measures the same signature. It only notifies. Auto-correcting a toggle from a heuristic is the identical bug wearing a better hat.

"Verified" now means something specific in this house: somebody measured the outcome, not the configuration. It is a higher bar. It is also the only one that has ever caught anything.

Note the asymmetry. Sending a command is edge-triggered and optimistic: it happens once and assumes. Checking the outcome is level-triggered and sceptical: it keeps asking for as long as the condition could be true. A one-way channel needs both, and the second is the one people skip.

Confirming the repair the way it should have been done

Having been taught this lesson at some expense, we did not declare the unit off and move on. We sent the toggle and watched the gap between that room and the rest.

t + 0 min   living 25.0   others 26.3   diff -1.27
t + 8 min   living 25.0   others 26.3   diff -1.30
t + 11 min  living 26.0   others 26.3   diff -0.30   <-- turned around
t + 16 min  living 26.0   others 26.4   diff -0.36

Eleven minutes after the command the gap collapsed from −1.3 to −0.3, and the other rooms did not fall to meet it. The cold room warmed. That is a confirmed off. The previous "confirmation" was a file inspection and a confident paragraph. (Since then the louver got a tilt sensor, and the unit can finally answer.)

The sequel, in which the rule catches its own tooling

The second incident is smaller and more embarrassing, because the thing misconfigured was the assistant session that helps maintain this site. A dashboard dropdown chooses which model it runs on. We would set it, and within a day it would be answering as an older model again. Each investigation produced a culprit that was verified and then wrong: an environment variable (found, disabled, and regenerated from a template at the next restart), a settings file (already correct), and a watchdog that turned out never to have started, so that nothing was watching the watcher and the post looked fine.

What settled it was the thermometer move: stop reasoning about configuration and read a measurement. Every answer in the transcript records which model produced it. Lined up by timestamp, the switches happened at moments when no command of any kind was issued on the machine, which acquits every suspect on it. The lever was in a pocket: the phone app attaches its own model choice to every message, and that quietly outranks the variable, the file and the dropdown put together.

The fix was not a fourth layer of configuration. It is a sensor next to the dropdown showing which model actually answered last. The dropdown states the wish; the sensor states the truth; disagreement is now visible in seconds instead of reconstructed across a day of confident diagnoses.

The rule this leaves you with

Before writing that something is verified, ask: what measurement would I see if this were false, and am I looking at it?

Things that feel like verification and are not: the config entry exists, the job is scheduled, the value is unique in the file, the service returned 200, the trace shows it ran. Every one confirms that you did a thing. None confirms the world changed. This matters most on one-way channels, where no acknowledgement arrives and the temptation to accept configuration as proof is strongest. There, the outcome has to be confirmed by an independent sensor: a thermometer, a timestamp, a door that actually opened.

And if you have already written the warning down somewhere, consider reading it.