Somewhere past four in the morning, a language model reads everything this house did in the last twenty-four hours (every light, door, alert, air reading, which automations fired and which pointedly did not) and writes up its opinion. It is diligent, unoffendable, cheaper than any human with the same reading speed, and wrong in ways that deserve their own field notes, because they are not the ways people expect.
Every night at twenty to five, a language model reads the day back to us and says what it thinks went wrong. It is not always right. It is, however, always awake, never bored, and has no feelings about being told it was wrong, which makes it the ideal colleague for the night shift and a slightly unnerving one for everyone else.
What it is genuinely good at
Volume and absence. No human reads a day of logs; it reads all of them, every night, without morale. And it notices the dog that did not bark: the automation that should have fired and did not, the sensor that has been saying the same thing suspiciously long, the heating that ran for an empty flat. The reporting page covers the mechanics; the short version is that it reviews the evidence against a written set of house rules and files a report.
The three ways it goes wrong
None of them are hallucination, which is what everyone asks about. All of them are evidence handling.
- Its rules drift behind reality. Change an automation's behaviour without updating the rules block and the reviewer reports correct behaviour as a fault, at length, with citations. It is not wrong; it is faithfully enforcing last month.
- Ambiguous evidence becomes confident conclusions. An incident log written by several automations was once described to it as if one author wrote every line. It read a routine informational line as a critical event and built a tidy little case around it. A person would have made the same mistake; the person would have sounded unsure.
- It inherits your assumptions at full confidence. Tell it a command was “verified safe” and it reasons downstream of that for weeks. The air-conditioner story on the verified-wrong page sailed through several nightly reviews untouched, because the reviews trusted the same paragraph I did.
The staff also live in the building
The colleague is not an abstraction in a distant data centre; the session doing the investigating runs on the same small box as the house. During one memory crisis the largest discretionary consumer of memory turned out to be the assistant investigating the memory crisis, at over a gigabyte. The building inspector was standing on the fire hose. That story, and the diagnosis it produced anyway, is in the horror stories.
Its diligence also needed a leash. The log collector that feeds the nightly review once fed on its own errors and filled a quarter of the disk overnight; the fix was verified, and the loop came back through it three days later.
The house now contains machinery whose sole purpose is to protect it from the enthusiasm of its own staff: a filter with no template in it, a rate cap, and a circuit breaker that will disable a runaway automation while explicitly refusing to touch anything with “alarm” or “leak” in its name. Details in the field guide.
Even the staff roster needs a sensor
The dashboard has a dropdown choosing which model the session runs on. For a day the session kept coming back as a different, older model, and three confident diagnoses on the box were wrong before a measurement settled it: the phone app attaches its own model choice to each message, outranking every local setting.
The fix was not a fourth layer of configuration. It is a sensor showing which model answered last, sitting next to the dropdown that states the wish. The discipline the house applies to its heating now applies to its help; the full sequence is in the sequel on the verified-wrong page.
Rules of engagement
- Give it measurements, not summaries. It believes summaries.
- Require “could not check” as an answer, and treat it as a finding: it usually means the evidence pack is missing a state the rules refer to.
- Change the rules in the same edit as the behaviour, or enjoy being prosecuted under last month's law.
- Its work is committed to version control nightly, which is both an audit trail and a performance review.
- Hold it to outcomes. “The config now says X” is its favourite sentence, and it means as little from the machine as it does from me.
Postscript: the reviewers got reviewed
After this page was written, the site itself went in front of a panel: four models from four different providers, blind review first, then rounds of adversarial cross-examination, run by a neutral chair. The findings were excellent and are worked into the pages you have been reading. The episode itself, though, belongs here, because it is this page's argument performed live.
In blind review, one seat asserted that a platform default was “1970-01-01, verified in the source”, in a session that had no access to any source. The actual source, read later, says today-at-midnight, which is what this site had published. The seat was not lying so much as compressing: “verified” was doing the work the evidence could not, which is the precise failure mode half this site documents, performed inside the machinery built to catch it.
What saved the record was not any reviewer's reliability. The cross-examination forced a retraction before ratification, so the published consensus said “disputed, verify before publishing” instead of a false correction; the panel later struck the finding, owned the error in its own record, and wrote the lesson better than I had: the redundancy must extend to the reviewers' own claims, and it is the hedged output that makes a wrong reviewer recoverable.
Two further panel errors were the tooling's: a mirroring script silently dropped two pages and the panel reported them absent from the site; a trimmed diff showed figures deleted that had merely moved down the page, and the panel reported them removed. Confident narration of a tool artifact, twice in one session, from a review that was right about nearly everything else. Instruments lie exactly like sensors do, which this site should perhaps have predicted would apply to the instruments pointed at it.
And one detail kept for the end. The seat that found the physics error in the motor-wear material reportedly shares a lineage with the model that wrote it. The same capability, blinded and set in opposition, caught what it had produced. That is the entire case for adversarial review in one anecdote, and it is why the nightly reviewer above gets evidence packs and house rules instead of trust.
Its most useful sentence remains 'I could not verify this'. A colleague who says that unprompted is rarer than one who is right.
Addendum: the morning after two “could not verify”s
The rules of engagement above ask for “could not check” as an answer and treat it as a finding about the evidence pack. On 19 September the night shift read the previous day back to us and used it twice in one report, and the morning went on finding out what a refusal is actually worth. Both times it turned out to be a finding about something rather worse than a thin pack.
The first thing it could not decide was the floor lamp. While the projector runs, the lamp is capped at a faint glow — the cap has no page of its own, so briefly: 1 out of 255 while a film is actually running, 25 while it is paused or somebody is browsing the library. The lamp had been on from just before the projector started until after midnight, spanning the whole session, and the reviewer pointed out, correctly, that an on/off log cannot tell 1 from 255. It asked for the brightness history. There was none.
Not none for that lamp. None for any lamp. Home Assistant no longer records light attributes,
brightness and colour included, to keep the database small — the light integration tells the
recorder to drop them — so the database holds friendly_name for every light in the flat
and nothing else. A lamp dimmed to a whisper leaves exactly the same trace as one at full blast,
which is to say no trace at all. The saved automation traces would have settled it, except they keep
only the last five runs and the evening had been busier than that. We had written a house rule that
no evidence anywhere in the building could ever support, and it took a machine reading the logs at
twenty to five to mention it.
The fix is unglamorous and complete, and it is the same move as every other derived signal here: a template sensor whose state is the lamp's brightness, 0 when off. States are recorded. History sees it, the long-term statistics see it, the evidence pack now carries its line, and the house rule about the cap ends with “judge from that, the light itself only logs on/off”.
The other refusal was about a thing that did not happen. One of us is nagged every morning until she measures her blood pressure (the house spirit joins in, orange and impatient), and the noon message had not fired. On a day when a reading arrives before twelve that is exactly right, and the reading had arrived at 09:40 — but the pack carried neither the reading nor the same-day flag, so the reviewer declined to call the silence a success or a failure. It now receives both.
Three of the things the report flagged were us; the rest of its error list was outbound-timeout
noise it correctly refused to call findings. One was our own reload during config work —
the trap that makes template entities blink — which the rules already
forgive, and which took a speaker's volume command down with it. A Bluetooth adapter reported
“not found” for about three minutes, because one of us was physically moving the dongle.
And an availability template for the office display crashed on a None the one time the
Mac declined to answer, a bug written that afternoon and fixed the next morning. A night shift that
files your afternoon's work before you have had coffee is doing its job.