Prepared at the Direction of Counsel
The hospital network's AI safety dashboard worked beautifully. Then a memo arrived explaining what a log is in a courtroom — and the watching got a curfew.
This story is from the AI Scenario Explorer, which pairs eight forces shaping AI, two at a time, into 28 maps — each map with four corners, each corner a different way that pairing could play out. A vignette takes one corner and makes it concrete: structured conjecture, not prediction. About the project →
The safety dashboard is the best thing the hospital network has built in years. This is the story of how it gets a curfew.
In February a radiology assistant goes live across the network's nine hospitals. It reads scans, drafts findings, flags what tired eyes miss at three in the morning. Beside it, a clinical engineer builds the watcher: every recommendation logged with the model version that produced it, every radiologist override tracked, a weekly report ranking the model's misses. For a season, the network knows more about its machine than the vendor does.
An honest log is an unflattering one. The model drifts on one scanner's images. Confidence runs hot for older patients. Items go on the fix list; some are fixed that quarter, others wait on the vendor, the budget, the next version. The list is long, because honest lists are long.
In August, a peer network three states away loses a lawsuit. Plaintiff's counsel had won three years of AI logs in discovery, and the deposition was mostly quotation: you knew in March; the fix shipped in January. The verdict matters less than the transcript, which travels through every general counsel's office in the sector within a month. The memo that follows, at our network, does not say stop measuring. It explains, patiently, what measurement is in a courtroom: knowledge — dated, attributable, permanent. Whether the network took reasonable care is arguable. The day it knew is not.
What happens next is renovation, not demolition. Nobody switches the watcher off; failing to test is its own kind of negligence, and the regulator now requires logs. Instead, the weekly report becomes a standing meeting — spoken, no slides. Retention drops to the mandated minimum. A vocabulary pass sweeps the templates: defect becomes observation, miss becomes variance. The deep evaluations are re-scoped as legal engagements — the red team reports to the law firm now — and every page they produce is born with a stamp at the top: Privileged & Confidential — Prepared at the Direction of Counsel.
The compliance audit still passes. It passes better than ever: everything mandated is retained and tamper-evident, every checkbox has its artifact. The raw logs still exist — the law requires them. What is gone is the layer that made them knowledge: the ranking, the fix list, the sentence that says this one matters. The machine is watched as closely as ever. What the institution writes down about the watching now has a curfew, and what it learns lives in the one medium no search term finds and no successor reliably inherits: memory.
The network has met this problem before, in humans. When clinicians would not report their own near-misses for fear of the courtroom, Congress built a protected channel — report the error, the report is privileged, the lesson travels. It works. So in the same building, on the same night: a resident's error goes into the protected system and comes out as training; the model's error goes into a meeting and comes out as a shorter retention schedule. The human near-miss becomes a lesson. The machine near-miss becomes a liability, and is handled accordingly.
This is a vignette from the AI Scenario Explorer — structured conjecture, not reportage: one corner of Institutional adaptation × Alignment/assurance given room to breathe. Signs it is arriving: AI evaluations and red-teaming contracted through counsel as standard practice; retention schedules for AI records shortening after high-profile discovery fights; mandated audit summaries published while substantive findings stay privileged. Signs it is not: a protected reporting channel for machine near-misses, the way medicine got one for human error; documentation rewarded as a defence strongly enough that thorough logs become the cheaper risk; courts treating engineered short memory as spoliation so consistently that the curfew stops paying.
What an institution allows itself to know about its own machine is decided by what a plaintiff's lawyer may someday read aloud. Until knowing is protected, the safest institutional memory is a short one — and the alarm keeps working, on a twenty-four-hour clock.