The Coming Epistemic Crisis AI Safety Is Not Ready For

1. Research generation is being automated faster than research evaluation

Language models are becoming genuinely useful across the research pipeline: literature review, experiment scaffolding, analysis, drafting. One underrated upside is that this lowers the cost of moving between fields. When the topic-specific grind gets cheap, a researcher’s ability transfers more easily; expertise becomes more topic-agnostic. For AI safety, which recruits most of its people from other disciplines, that is good news.

But the same tools lower the cost of producing plausible-looking research much faster than they lower the cost of judging it. Evaluation still runs on senior human attention: reading, checking, placing work in context, deciding what matters and what gets funded. That attention does not scale with model capability. This asymmetry, generation getting cheap while evaluation stays expensive, is the inflection point, and everything in this essay cascades from it.

• Language models now handle much of the routine work of producing research, and they make it easier for people to move between fields, which helps a field that recruits from other disciplines.
• Judging research still depends on senior human attention, and that does not get cheaper as the models get better.
• The gap between cheap production and expensive judgment is where everything else in this essay begins.

2. The volume shock is already visible, and institutions are responding by raising the bar

Detector-based analysis of ICLR 2026 found several hundred submissions written wholly by LLMs, and roughly one in ten with majority-LLM content. arXiv stopped accepting position papers and review articles in computer science without prior peer review, citing the volume. LessWrong holds LLM-generated content to a higher bar than human writing. Meanwhile, the same analysis found one in five reviews fully LLM-generated, closing a loop nobody designed, in which machine-written papers meet machine-written reviews and human judgment thins at both ends.

These responses are mostly reactionary: they gate work by raising the cost of entry. That collides with the upside from (1): cheap field-switching is worth little if nobody can afford to read what the switcher produces. Field-building runs into the same wall: it keeps widening the funnel on the assumption that newcomers will be integrated into the field, and expensive evaluation of their work is exactly what slows that integration down.

• The volume shock is already measurable: at one conference, several hundred submissions were written entirely by models, and about one review in five was as well.
• Institutions are responding by making entry more expensive, not by making evaluation cheaper.
• Machine-written work is now being judged by machine-written reviews, which removes human judgment from both sides at once.
• These responses work against field-building, which needs newcomers’ work to be cheap to assess.

Nobody is making evaluation cheaper. Everyone is making entry more expensive.

3. Field-building trains more candidates than the field can absorb, and their output is straining evaluation capacity

Field-building programs are scaling admissions faster than the field scales absorption. Introductory programs plan to train 100,000 people by 2030; research fellowships like MATS accept applicants at rates in the low single digits, a hyper-selectivity that the field’s own needs assessment traces to scarce supervision capacity. The people who are trained but not placed do not vanish. Underemployed, motivated, and equipped with the same generation tools as everyone else, their output lands on the field’s open surfaces: grant applications, forum posts, conference submissions, cold emails to researchers.

The problem is not that weak work exists; weak work has always existed. The problem is that filtering consumes the scarcest resource the field has, the attention of the people qualified to filter: mentors, admissions committees, grant evaluators, forum moderators. Slop is better understood as the indiscriminate spending of evaluator attention than as pollution.

None of this requires the work to be bad. Suppose most of it is decent and some of it genuinely contributes: with no infrastructure to absorb the volume, the same systems get stressed, and a stressed filter does not select for the best work. It selects for the work that is cheapest to judge. Legible, agenda-adjacent work tends to clear the bar; work that would take an afternoon to understand often does not, however good it might be. That is not a neutral filter, and it is worth asking how it skews the field’s output. (4) takes up what follows.

• Training programs are growing much faster than the field’s capacity to supervise or hire the people they train.
• Those who are trained but not placed keep producing work, and it arrives on the field’s open surfaces: applications, forum posts, submissions, cold emails.
• The real cost is not the weak work itself but the evaluator attention spent filtering it, which is the scarcest resource the field has.
• None of that depends on the work being weak: decent work in volume strains the same filters, and a strained filter tends to select for what is cheapest to judge rather than what is best, which is not neutral in how it skews the field’s output.

4. Even though grantmakers have the most leverage, they ironically have the faintest signal

Consider what a funder can realistically consult: shallow reviews, forum and Twitter write-ups, yearly organizational reports, grant applications, and personal connections. Is that enough of a view of the field? Probably not, and a funder making the individually rational call on that signal is not necessarily making the call that serves the field. There is precedent for how this goes. Research on science funding describes a squeeze that operates from two directions at once. Ex ante review, which determines what gets funded, favors proposals that are not especially speculative and that are bold in the direction of what the field already believes, since those are the proposals a reviewer can assess quickly and score with confidence. Ex post evaluation, which determines what counts as a successful grant, favors novel results, and those generally require speculative, high-risk experiments that change prevailing beliefs rather than confirm them. Because the two criteria point in opposite directions, the space of work a researcher can realistically propose is squeezed from both sides, and the cheapest response, or even the sensible one, is to quietly self-censor whole topics rather than defend them. I am not claiming that exact dynamic will replicate here. But some version of research being squeezed toward a highly correlated direction seems likely, and without any view of the whole research landscape, there would be no way to notice it is happening.

That reflex is not confined to grant proposals. It would attach to any map of the field that became canonical, which is the failure Forethought files under epistemic misalignment: once a map is what funders and hiring committees consult, the rational move is to look legible on it, and what is easy to represent starts crowding out what is true. A small and correlated field makes that a stronger Goodhart target than a large one would, not a weaker one. It is the most serious objection to what (6) proposes, and the appendix answers it.

• Funders decide the most while seeing the least: their picture of the field comes from shallow reviews, forum posts, annual reports, applications, and people they already know.
• Research on science funding suggests the criteria pull in opposite directions: grants favor proposals that are bold in the direction of what the field already believes, while success afterwards is judged on novel results that overturn those beliefs.
• Squeezed from both sides, a researcher’s cheapest response is to quietly self-censor whole topics rather than defend them.
• Each funding decision can be individually sensible and still push the field toward a single correlated direction.
• Without a view of the whole landscape, nobody would notice that narrowing while it was happening.
• The same reflex would attach to any map that became canonical: once a map is what funders consult, looking legible on it becomes the rational move, which is the strongest objection to what (6) proposes.

5. No external check, a handful of buyers, and no way to measure the correlation

AI alignment is not a field with external signals the way computer science or civil engineering are: most of its concerns are about systems that do not exist yet.

The field’s epistemic processes are not a support function. They are close to being the product.

Most direct safety work happens inside a handful of frontier labs and a few independent organizations. What the surrounding field mainly supplies is direction, critique, and prioritization. Its technical contributions are marginal. Its epistemic ones are not: directly through the discourse, indirectly through the talent the frontier labs absorb.

On the market side, there are a few frontier labs, a few independent safety orgs that supply the frontier labs, and a few governments. The nodes read the same forum, share advisors, and hire from one another; newer entrants such as national AI safety institutes staff themselves from the same pool. The talent pipeline compounds this: a small number of mentors’ taste is replicated into each cohort, a shared curriculum forms the canon, and a few venues host most of the deliberation.

Concentration is partly load-bearing: it is how a small field moves fast, maintains trust, and filters work. The convergence may even be correct; perhaps the dominant agendas deserve their current weight. In a normal field, false consensus eventually collides with reality; here it may not. That leaves provenance as the only remaining test: did several funders converge on the same bets independently, or because the bets share three advisors?

Nobody currently measures this. There is no concentration index for the field’s collective portfolio, no funder-overlap statistic, no way for any individual funder to know how correlated their bets are with everyone else’s. The pre-2008 financial system is the cautionary parallel: every institution looked diversified on its own books, and the correlation lived in a counterparty network no one had mapped. History never repeats, but it often rhymes.

• Alignment cannot wait for reality to settle its disagreements, because its claims are about systems that do not exist yet.
• That makes the field’s collective judgment most of what it actually produces, rather than a support function around the research.
• The buying side is small and interconnected, so research directions end up correlated whether or not anyone intends it.
• Concentration is not automatically bad, but nobody measures it here, so no funder can tell how concentrated the field’s bets already are.

6. Starting points and a general direction for what collective epistemic infrastructure could look like

The collective version is not mysterious, and it does not have to be a map. It starts with a data layer: continuously updated records of research output, funding flows, which organizations work on which agendas, and which venues host what. Not snapshots. The field changes faster than any annual review can track, which is part of why the annual reviews lost their charm (8).

On top of the data layer, we can build a tool layer that makes understanding and evaluating research cheaper for humans, without handing either over to the models. The asymmetry of (1) is that generation got cheap while evaluation stayed expensive; the fix is to multiply scarce evaluator attention, not to delegate the gate to the frontier models that produced the volume. (2) already shows how that loop closes.

The test for corpus-level analysis is speed. Today, anyone who wants to know where the field is heading either runs a three-month project with four FTEs or trusts someone who did. Seeing the field’s direction should be a query, not a research program. Funding data deserves the same standard: the concentration and overlap measures of (5) should be standing instruments anyone can consult, not bespoke studies commissioned after the worry has surfaced.

The labor for this already exists, and (3) described it: the residual cohort, trained and motivated and unplaced, ingesting new research, distilling it, situating it within the field. On first reading this is the worst answer on the table. The scarce thing is evaluator attention, and the proposal is to route the volume through people who have never been evaluated themselves, the same people whose output is straining the filters. Against a model that at least applies one standard consistently, it looks like a downgrade with a mission statement attached.

What that reading misses is why the cohort has no track record. It is not that they were assessed and found lacking. Almost nothing they produce is assessed at all publicly; they write in volume into a system with limited capacity to read them, and the missing record about their quality is a large part of why they stay unplaced, along with limited upstream opportunities. A structured channel changes that: a defined slice of the corpus to cover, a cadence, and experienced people reviewing what comes out. The reviewer’s time then buys three things where it used to buy one. Feedback the newcomer can act on. A public record of whose judgment holds up, which is what the field’s filters currently assess in private and only for the few who get far enough to be assessed at all. And the distillation itself.

That is the leverage, and it is why this is not the same as handing the reading to the models. Mentor attention spent on a submission today disappears into a yes or a no. Spent this way it compounds: it trains someone, it produces a record, and it scales across far more of the corpus than the mentor could have read directly. The work orients the cohort in return and pays down the research debt of (8).

Whether it converts is a question about the protocol rather than the premise. What fraction of the output gets reviewed, how a reviewer’s verdict is recorded and weighted, what happens when two reviewers disagree, how much of the corpus a distillation has to cover before anyone consults it: those choices decide whether mentor attention actually turns into collective epistemic quality, or only into a better-trained cohort. The second is worth having on its own. Only the first pays for the build.

And none of it works as a static artifact. It needs an active community and a channel: a standing place where new work gets absorbed as it arrives, and where the point is helping people find understanding and share their own.

There is modest precedent for funding work of this kind: worldview-investigation and forecasting projects were supported when the open question was AI timelines, those efforts were early, and recently, Forethought has written about strategic awareness and collective epistemics as a category. The combination of research automation and field-building residual cohorts makes the next year or two the period in which this stops being early.

That paper is also the nearest thing to a design document for a build like this, and it names the two ways such a build fails: work shaped to look good on the map, and one map becoming unquestionable. Both apply to what is described above, not only to the general program, and (5) makes both worse here rather than better. The appendix takes the objection at length, along with the partial answer available to it.

• Start with a live data layer: who is working on what, funded by whom, and published where, updated continuously rather than once a year.
• Add tools that make research cheaper for people to read and judge, instead of handing the judging over to the models.
• Keep the measures standing and queryable, so learning where the field is heading is not itself a three-month research project.
• Staff it with the people field-building has already trained, and run it as an active community rather than a finished artifact. On its face that is the worst option available: routing the volume through the field’s least-credentialed readers.
• But they lack a record because nothing reads their work, not because it was read and found wanting. With experienced reviewers on a cadence, the same mentor hours buy feedback, a public record of whose judgment holds up, and the distillation itself. Whether that becomes collective epistemic quality or only a better-trained cohort depends on the protocol.
• Expect the two failures the same template names: work shaped to look good on the map, and one map becoming unquestionable. A small, correlated field is where both take hold most easily, not least. The appendix works through the objection and what can be done about it.

A field’s strategic self-awareness is a product of its collective epistemic infrastructure.

7. What happens if nothing is built

Sorted by confidence: the first items are visible trends continuing, the middle ones need assumptions, the last are long shots included for their size.

• Entry gets harder for anyone outside the network. Overloaded evaluators close open channels — referrals replace open calls, cold outreach stops working, venues raise bars. This is individual triage, not policy.
• Evaluation retreats to channels where trust is already established. Talks, conversations, vouching: the channels where provenance is cheap to check. It is the standard response to a market where quality is hard to observe — costly signals take over, and the costly signal here is trust.
• The field’s existing concentration gets worse. The trust network becomes the fallback filter, and (5) argued it is already concentrated. The cheap field-switching from (1) then accrues mostly to insiders.
• Quality gets judged by proxy rather than by content. People who cannot read everything judge by affiliation, endorsement, and familiar framing. All of these correlate with the existing network.
• Duplicated work and unstaffed gaps both go unnoticed. With no live record of who is working on what, several groups can spend a year converging on the same problem while other problems sit untouched. Neither the overlap nor the gap is anybody’s job to see.
• Criticism gets more expensive than the claims it would correct. Rebutting work means reading it closely, and reading is the scarce thing. Weak claims then go unchallenged rather than refuted, which is not the same as being accepted, but has the same effect on what newcomers read as consensus.
• The field’s memory of what was already tried thins out. Which approaches were attempted and abandoned, and why, lives mostly in the heads of people who were there. As they move on, the record leaves with them, and later cohorts re-run the same reasoning without knowing it is a repeat.
• Alignment research slows down relative to capabilities research. Capabilities is evaluated cheaply — benchmarks, loss curves, products. Alignment’s filter is human judgment, per (5). The same volume shock taxes one field and barely touches the other, and automation widens the gap by accelerating whatever is generation-limited. A hygiene problem becomes a race problem. I hold this loosely, since lab-internal safety work has more empirical feedback than the surrounding field.
• Research bends toward work that is easy to evaluate. Overloaded gates pass legible, incremental, agenda-adjacent work and stall pre-paradigmatic work. If important problems sit in the second category, the field gets pointed away from them without anyone deciding it. This may matter more than the slowdown.
• Course correction gets slower than the thing it is correcting. When an agenda quietly loses its rationale, nothing surfaces that fact on any particular date. Funding continues, cohorts keep being trained into it, and the correction arrives whenever enough individuals drift away on their own judgment.
• The tail case: the field ends up confidently wrong exactly when it matters. Correlated judgment plus weak feedback makes an uncaught, substantially wrong consensus possible here in a way it is not in most fields. I am not sure what probability to attach to it, and it could well be non-trivial. Included because this is the failure mode that feeds directly into the existential risks the field exists to reduce.
• The benign branch: the field adapts on its own, as other fields have. Physics absorbed arXiv; biology absorbed cheap sequencing by growing new evaluation professions. Verification and distillation could become respected specializations here too. The catch is speed: those adjustments took a decade or more, and it is not obvious this field has one.

8. Is it worth it?

It is not going to cost much, in absolute terms or relative ones: a data pipeline, a few standing tools, a channel, and stipends for people the field has already paid to train. Field-building alone has absorbed roughly $320 million. The parts described here would cost a small fraction of that, and unlike a cohort or a conference, infrastructure keeps paying out for as long as the field uses it. And it goes straight at the problem in (4): funders have the most leverage and the worst view of the field, and this is the cheapest way to improve the view. Since, by (5), the field’s collective judgment is most of what it produces, better judgment raises the value of everything else. It may be the highest-leverage money the field can spend.

Chris Olah and Shan Carter named the underlying condition years ago: research debt. Distillation is undersupplied, so papers stay harder to read than they need to be, and the cost compounds with volume. In alignment, the symptom is concrete. The once-annual comprehensive review of the field’s literature was discontinued largely because no single person could read it all anymore, and its best-known successor is titled, candidly, a “shallow review.” The sense-making layer was already struggling with volume, even before the surge.

The $320 million figure comes from Elena Ericheva’s A genealogy of AI safety: how directions are born, and how they die, a solo reconstruction of two decades of the field’s directions and funding flows: careful, rigorous, and done by hand. That provenance makes the point by itself: this is what it currently takes to get a view of the field, and even then the result is a snapshot, not something live. Ericheva was not trying to build epistemic infrastructure. But the skill set and the process are exactly the ones the infrastructure needs, which means the people who can build the live version already exist.

• The potential costs are trivial, since an ecosystem that demands this work and a community that can supply it already exist.
• The potential upsides are many: better research progress, and more clarity about where the field stands.
• And if we do not build it, the downsides are very high: in the worst case, a field unable to prevent the catastrophic events it exists to prevent.