The Dark Basin

The AI that matters most may be the one you never see.

The AI systems easiest to discuss are the ones built to be seen. Consumer products have interfaces; frontier labs publish benchmarks, demonstrations and safety reports. Their failures circulate as screenshots. Visibility determines which systems become the public picture of AI.

Consequence does not require a public interface. A trading model used inside a fund, a discovery system used inside a pharmaceutical company, or a capability used by an intelligence service may influence markets, medicine or state power while remaining almost entirely unavailable for inspection. We may see the organisation, and sometimes its output, without seeing how much authority the system has or how it reaches its conclusions.

Astronomy is comfortable with this situation. Most known planets outside our solar system have never been seen: they are detected by their gravity — a wobble in the star they orbit, a dip in its light. Neptune was found on paper first, predicted from the perturbations of Uranus before any telescope pointed the right way. Consequential private AI is that kind of object — dark itself, detectable mainly in the pull it exerts on what we can see.

Imagine a map of AI systems drawn from three things: the power they exercise, how many people touch them directly, and the evidence they leave in public. In one region — much power, few hands, little evidence — the ground falls away. I call that region the Dark Basin: AI that exercises real power — over markets, over which drug candidates advance, over what a state's intelligence services can see and do — yet remains structurally invisible to the public debate about AI. Secrecy need not be conspiratorial here. The incentives that might pull these systems into view point the other way.

Every other piece in this collection grows from one map: two forces, four corners. The basin is not a corner. It kept appearing in map after map, whichever two forces were on the axes — which is what you would expect of a low place in the whole landscape rather than a feature of any one slice through it. Three axes are the fewest that show it as what it is.

How much is out there? Mostly, nobody can say — but one patch of the basin has been measured. In January 2025 the FDA proposed the first framework for assessing AI models used in drug development, and mentioned, almost in passing, that it had seen more than five hundred drug and biological product submissions with AI components since 2016.

A submission is not a system — one model may sit behind several, and a "component" may be small. But the count is a rare public aggregate: a glimpse of how extensively AI is entering one industry. The activity is documented; the systems are not. The announcement tells you almost nothing about which models, how much of each decision they carry, or how well they perform. That is the shape of the whole basin — knowable in aggregate, dark in the particulars. And drug development is the well-lit case, the sector with a regulator that had a reason to count. For trading models and classified capabilities, no regulator has published a count.

A system's prominence in the debate tracks how public-facing it is, not how much it matters. Consumer chatbots are visible because millions touch them. Frontier labs are visible because their business model needs talent, capital, licence and trust, so they publish and testify. But nothing about deciding a lot requires being seen — and several forces keep the deciders dark at once.

Trade secrecy makes silence the legal default. National-security classification makes disclosure a crime. With no consumer interface there is nothing to screenshot and no user to complain. The buyers — pension funds, sovereign wealth, intelligence agencies — have professional reasons not to talk. The domain regulators govern the output and have lacked the mandate or capacity to interrogate the model. And every opaque actor benefits from opacity being normal. Any one of these is defeatable; together, so far, they never have been — which is why "just mandate transparency" goes nowhere. It attacks one force while the others compensate.

Visible and invisible AI are also developing in opposite directions. The visible kind is broad but shallow: it can demonstrate a little of everything, and it is deployed cautiously where the stakes are high, because a public lab cannot afford to fail catastrophically in public.

The invisible kind is the mirror image — narrow and deep. No public benchmark, because the only audience is the operator. Run hard in its one domain, because the operator owns the consequences privately. The defensible version of the scary thought is narrow: not that these systems beat the public frontier at everything, but that in their one domain they may be deployed far past what anything public reveals — and public benchmarks cannot tell you how far.

The basin has a far side. Past some size, a system's effects give it away even when nothing else does — the wall rises, but only to the waterline: the world learns the thing exists, not what it is. That threshold is the companion piece, Too Big to Hide.

So the public, calibrating on the visible frontier, overestimates AI's breadth and underestimates its depth; we form our intuitions on a disproportionately visible layer. The governance aimed at that frontier — capability thresholds, compute disclosure, lab licensing — mostly misses the basin, whose systems need not run on frontier-scale compute and need not belong to anyone holding a licence. What does reach it, as the FDA has just shown, is the domain instrument, not the grand one. We have mostly been governing the part that was already in view.

I cannot prove this. That the systems inside quantitative funds genuinely carry the decisions; that internal pharma models run ahead of their published versions; that sovereign cyber runs on AI at all — I assume each because of how those sectors are built, not because I have seen inside one. Neither, in any way that could be checked, has the public. The claim would collapse if consequential internal AI turned out to be regularly and independently audited, or if its capabilities were routinely overstated. I can find no sign of the first. The second I cannot rule out from where I stand. The wobble method has its own cautionary tale: the astronomer who found Neptune later inferred a planet called Vulcan from a wobble in Mercury's orbit, and it was never there — the anomaly had a different explanation entirely. How the map was drawn, and how I tried to keep the drawing honest, is its own piece: When AI Agreement Isn’t Evidence.

So how do you govern what you cannot see? Sector by sector, where the systems already live: securities regulation that asks about algorithms, not just positions; drug regulation that reaches upstream into the discovery model, as the FDA's has begun to; intelligence oversight that can follow what classified AI is doing. Duller than one grand AI law — and aimed where the power actually is.

Either of the lists below is checkable. That is the point of writing them down.

Signs the basin is real	Signs it is smaller than it looks
Regulators in other sectors start to count — and find populations the size of the FDA's, or larger	Audit regimes actually reach internal models, sector by sector
A domain regulator requests model-level access and is refused, in public, on trade-secrecy grounds	Disclosed discovery models turn out no better than their published baselines
An enforcement action or a leak reveals an internal system materially past public assumptions	Internal capabilities prove no deeper than the public ones

I don't know which way it goes. But if the basin is where the incentives say it is, then some of the most consequential AI in the world is, right now, AI that nobody ever had to hide.

---

This is structured conjecture from the AI Scenario Explorer — one region of the map, drawn from the incentives, with no claim about probability.