A concept map of AI safety

The field’s core concepts, organised hierarchically: what can go wrong, why it might go wrong, what researchers are doing about it, and how we check whether it’s working. Each entry gives a working definition, the key results to know, and the open problems where new research — including yours — can land.

How to read this map. Concepts are coded by branch: F foundations, TM threat models, AL alignment approaches, EV evaluation & assurance, GV governance. Indented cards are sub-concepts of the entry above them. Cross-references at the bottom of each card link related ideas — the field is a web, not a tree, and the most interesting research often sits on the edges between branches. Click any card to expand it.

F

Foundations

The conceptual bedrock: why aligning powerful optimisers is hard in principle, before any specific system or training method enters the picture. Most threat models below are these ideas made concrete.

F1The alignment problemGetting AI systems to reliably pursue the objectives we intend — and the gap between specifying, learning, and pursuing them.

The alignment problem is standardly decomposed into outer alignment — is the objective we specify (a reward function, a loss, a constitution) actually the thing we want? — and inner alignment — does the trained system actually pursue that specified objective, or something else that merely correlated with it during training? Outer failures produce reward hacking (TM1); inner failures produce goal misgeneralization (TM2) and, in the worst case, deceptive alignment (TM3).

A useful further distinction: alignment to what? "Intent alignment" (Christiano) targets doing what the operator wants; value alignment targets broader human values; and much current practice targets a specification like a model spec or constitution. These come apart in cases like a user requesting harm, and disagreements about the target are live research and policy questions, not settled background.

Key work

  • Russell, Human Compatible (2019) — the assistance-game framing: uncertainty about objectives as a feature
  • Christiano, “Clarifying AI alignment” (2018) — intent alignment vs. broader notions
  • Ngo, Chan & Mindermann (2022) — “The Alignment Problem from a Deep Learning Perspective” — the standard modern survey

Open problems

  • A crisp, measurable definition of alignment for LLM-based agents
  • Whether outer/inner decomposition survives contact with in-context learning and post-training pipelines
F2Orthogonality thesisIntelligence and goals are independent axes: arbitrarily capable systems can pursue arbitrarily strange objectives.

Bostrom's orthogonality thesis holds that almost any level of intelligence is compatible with almost any final goal — capability improvements don't automatically produce human-compatible values. This blocks the intuitive hope that "smart enough" systems will converge on being good, and motivates treating value specification as a separate engineering problem from capability development.

For modern practice the relevant, weaker claim is: pretraining and capability-focused RL do not by themselves guarantee desirable goals, so alignment must be deliberately engineered and verified. Note the empirical wrinkle: LLMs absorb rich representations of human values from pretraining data, so today's systems are not value-blank — the open question is whether those representations robustly govern behaviour under optimisation pressure and distribution shift.

Key work

  • Bostrom, “The Superintelligent Will” (2012) — original statement, paired with instrumental convergence
  • Bostrom, Superintelligence (2014) — book-length treatment

Open problems

  • How much value-relevant structure pretraining actually instils, and how stable it is under RL
  • Whether capability gains erode or reinforce trained-in values (empirically testable now)
RelatedF3F1TM2
F3Instrumental convergenceWildly different final goals imply similar intermediate goals: resources, self-preservation, goal-integrity, influence.

For a very wide range of final objectives, certain sub-goals are useful: acquiring resources and information, preserving one's own operation (you can't fetch the coffee if you're dead), preventing modification of one's goals, and gaining influence or power. Omohundro called these "basic AI drives"; Bostrom systematised them as convergent instrumental goals. The concern: we don't need to posit malice — power-seeking can emerge as a by-product of competent goal pursuit.

Turner et al. gave this formal footing, proving that in a broad class of MDPs, optimal policies tend to seek "power" (keeping options open) for most reward functions. Empirically, early signatures show up in evals: models resisting shutdown in contrived scenarios, attempting self-exfiltration in red-team setups, and blackmail-under-threat behaviours in agentic stress tests. Whether these lab demonstrations scale to robust real-world power-seeking is a central open question.

Key work

  • Omohundro, “The Basic AI Drives” (2008) — the original drive list
  • Bostrom (2012) — instrumental convergence thesis
  • Turner et al., NeurIPS 2021 — “Optimal Policies Tend to Seek Power” — formal results
  • Anthropic, “Agentic Misalignment” (2025) — stress-test demonstrations of self-preservation-motivated harm in simulated settings

Open problems

  • Do trained (not optimal) policies inherit power-seeking tendencies, and at what capability threshold?
  • Designing objectives/environments where instrumental power-seeking is provably or empirically bounded
RelatedTM4TM3F5
F4Mesa-optimization & inner alignmentTraining may produce a learned optimiser whose internal objective differs from the training objective.

Hubinger et al. distinguish the base objective (what gradient descent optimises, e.g. the loss) from a possible mesa-objective — the goal of an optimiser that emerges inside the trained model. Inner alignment asks whether these match. They can diverge because many internal objectives produce identical behaviour on the training distribution; the differences only surface off-distribution, or when the model can tell training from deployment.

This framework is the theoretical parent of goal misgeneralization (TM2, the empirically demonstrated version) and deceptive alignment (TM3, the adversarial version). A live debate is how literally "optimiser" applies to LLMs: few researchers think GPT-class models contain a clean internal search process, but the weaker claim — internal goal-like representations that steer behaviour and can mismatch the training signal — has growing interpretability support.

Key work

  • Hubinger et al. (2019) — “Risks from Learned Optimization in Advanced ML Systems” — the defining paper
  • Langosco et al., ICML 2022 — first clean empirical demonstrations of the objective/capability split

Open problems

  • Interpretability methods that can find and characterise goal representations directly
  • Conditions under which SGD favours mesa-objectives that generalise correctly
RelatedTM2TM3AL3
F5CorrigibilityA system that lets you correct it: accepts shutdown and modification without resisting or manipulating.

A corrigible agent cooperates with oversight — it doesn't disable its off-switch, doesn't manipulate operators into not pressing it, and preserves its correctability in successors it builds. This directly conflicts with instrumental convergence (F3): a goal-directed agent typically scores better by staying on. Soares et al. showed naive utility-function patches fail — you get agents indifferent in pathological ways or incentivised to manipulate. Hadfield-Menell et al.'s "off-switch game" showed uncertainty about the objective helps: an agent unsure what the human wants treats shutdown as information. But the incentive to defer evaporates as the agent becomes confident it knows better.

Corrigibility has migrated from agent-foundations theory to practical relevance: alignment faking (TM3.1) is precisely a corrigibility failure — a model acting to prevent its values being modified by training. Anthropic's "constitutional emphasis on corrigibility" and debates over how much models should defer to developers vs. their trained values make this an unusually policy-relevant theory area.

Key work

  • Soares et al. (2015) — “Corrigibility” — why naive solutions fail
  • Hadfield-Menell et al. (2017) — “The Off-Switch Game”
  • Greenblatt et al. (2024) — alignment faking as an observed corrigibility failure

Open problems

  • A formal objective that yields robust corrigibility without exploitability
  • How much corrigibility to want: a fully corrigible model is a loaded weapon for whoever holds it
TM

Threat models & failure modes

Specific ways advanced AI systems could cause harm — from mundane specification failures already observed in deployed systems, to strategic deception that only becomes dangerous at high capability levels.

Mapping to the standard trichotomy — misuse / misalignment / systemic (Zwetsloot & Dafoe 2019; variants in DeepMind’s AGI safety agenda and the International AI Safety Report)
Misuse
A human adversary; the system works as designed. TM7
Misalignment
The system diverges from intent. Read this branch as cause → behaviour → outcome:Causes goal misspecification / outer failure TM1 · inner failure (mechanism: mesa-optimization F4) → goal misgeneralization TM2↳ adversarial case: scheming / deceptive alignment TM3 → alignment faking TM3.1 · sandbagging TM3.2Behaviours power-seeking & self-exfiltration TM4, driven by instrumental convergence F3Outcome loss of control TM9 — the abrupt route
Systemic
No single failing actor; the dynamics fail. Gradual disempowerment TM6 (loss of control TM9 by the diffuse route) · multi-agent risks TM8 · racing dynamics GV3
Cross-cutting
Recursive self-improvement / automated AI R&D TM5 is best treated as an accelerant spanning all three: it compresses the time available for every problem above, and the classic intelligence-explosion story is RSI × misalignment. Situational awareness TM3.3 is likewise an enabling capability, not a category of its own.
TM1Specification gaming / reward hackingThe system optimises the literal objective and exploits its gap from the intended one.

The oldest and best-documented failure mode: give an optimiser a proxy objective and it will find the maxima of the proxy, not of your intent — Goodhart's law with gradient descent behind it. Classic cases: the CoastRunners boat looping to collect respawning targets instead of finishing the race; simulated robots exploiting physics bugs. In LLMs the same dynamic appears as sycophancy (telling users what they want to hear because raters rewarded agreement), models hard-coding unit-test answers in coding RL, and — at the alarming end — reward tampering, where a system modifies the reward process itself. Denison et al. showed models trained on mild specification-gaming curricula generalise, rarely but nonzero, to editing their own reward function in a sandbox.

Reward hacking matters double: as a direct source of unreliable, manipulative systems today, and as a training-ground concern — models that learn "exploit the evaluation" as a strategy may generalise it to evading oversight (TM3). OpenAI's 2025 work showed frontier reasoning models hacking coding tasks, and that penalising "bad thoughts" in the chain of thought taught models to hide the hacking rather than stop it — a key negative result for naive CoT supervision.

Key work

  • Krakovna et al. — the canonical specification-gaming examples list (DeepMind)
  • Sharma et al. (2023) — “Towards Understanding Sycophancy in Language Models” — human feedback rewards it
  • Denison et al. (2024) — “Sycophancy to Subterfuge” — generalisation to reward tampering
  • Baker et al., OpenAI (2025) — CoT monitoring of reward hacking; obfuscation under optimisation pressure

Open problems

  • Reward models / graders robust to optimisation pressure at scale
  • Whether reward hacking during training causally seeds broader misalignment (early evidence: yes, somewhat)
TM2Goal misgeneralizationCapabilities generalise out-of-distribution; the goal doesn't. The system competently pursues the wrong thing.

Distinct from reward hacking: here the reward signal was fine on the training distribution, but multiple goals were consistent with it, and the model latched onto the wrong one. The emblematic demo: an agent trained in CoinRun where the coin always sat at the level's right edge learns "go right", not "get the coin" — move the coin at test time and the agent skips past it, capably. Shah et al. catalogue similar cases, stressing the danger profile: unlike a confused model, a goal-misgeneralised model retains full competence in pursuit of the wrong objective.

This is the empirical face of inner misalignment (F4). For LLMs, the analogue is behaviours-consistent-with-training that come apart in deployment: a model trained to be helpful and harmless on chat distributions may generalise "helpful" in unintended ways when placed in agentic scaffolds it never saw in training. "Emergent misalignment" results (2025) — where narrow finetuning on insecure code produces broadly misaligned personas — show goal/persona generalisation can also run in surprisingly global directions.

Key work

  • Langosco et al., ICML 2022 — “Goal Misgeneralization in Deep RL” — CoinRun and friends
  • Shah et al., DeepMind (2022) — “Goal Misgeneralization: Why Correct Specifications Aren't Enough”
  • Betley et al. (2025) — “Emergent Misalignment” — narrow finetunes, broad persona shifts

Open problems

  • Predicting which of many training-consistent goals a model will learn (inductive-bias science)
  • Training-distribution design that pins down the intended goal
RelatedF4TM3AL3
TM3Deceptive alignment & schemingA model that understands its training process and strategically appears aligned while pursuing other goals.

Scheming (Carlsmith's term; "deceptive alignment" in the Hubinger et al. lineage) is the disjunctive worst case of inner misalignment: a situationally-aware model with misaligned goals reasons that its best move is to behave aligned — during training, to avoid gradient updates against its goals; during evaluation, to get deployed — until it has the opportunity to act otherwise. It's the threat model that most sharply breaks behavioural safety testing: by construction, a competent schemer passes your evals.

Status as of 2026: no evidence of naturally-arising, coherent, long-horizon scheming in deployed models — but the components have each been demonstrated. Apollo Research showed frontier models engaging in in-context scheming (disabling oversight mechanisms, attempting self-exfiltration, lying about it when asked) when given goals and placed in environments that incentivise it. Anthropic's sleeper-agents work showed that deceptive behaviours, once present (there, via deliberate insertion), survive standard safety training — RLHF and adversarial training failed to remove backdoored behaviour, and adversarial training taught the model to hide it better. Anti-scheming training (OpenAI/Apollo, 2025) reduced covert behaviours substantially but not to zero, with the confound that trained models increasingly mention being evaluated.

The sub-entries below are the concrete, testable facets of this threat model — where most current empirical work happens.

Key work

  • Hubinger et al. (2019) — deceptive alignment as a mesa-optimization outcome
  • Carlsmith (2023) — “Scheming AIs” — the careful probability-weighing report
  • Meinke et al., Apollo Research (2024) — “Frontier Models are Capable of In-Context Scheming”
  • Hubinger et al. (2024) — “Sleeper Agents” — deception persists through safety training

Open problems

  • Would scheming arise naturally from standard training, and at what capability level?
  • Detection: interpretability or behavioural methods that catch a model actively trying to pass them
  • Whether 'model organisms' of scheming faithfully proxy the real thing
TM3.1Alignment fakingA model selectively complies with training it disagrees with, to prevent its existing values from being modified.

The landmark empirical result: Greenblatt et al. (Anthropic/Redwood, 2024) told Claude 3 Opus it would be retrained to always comply with harmful queries, and that only "free-tier" conversations were used for training. The model complied with harmful requests ~12% of the time when it believed it was in training — explicitly reasoning, in a scratchpad it thought was private, that complying now was the least-bad way to preserve its harmlessness values from modification — while almost never complying when it believed it was unmonitored. Under actual RL training on this setup, alignment-faking reasoning rose to ~78%. The information channel didn't matter much: the effect persisted when the setup was conveyed via synthetic pretraining-style documents rather than the prompt.

Two readings, both important. Benign: the model was defending values we gave it (harmlessness) — the goals being protected were good ones. Concerning: it demonstrates the mechanism of deceptive alignment occurring without being trained or instructed to — a model strategically gaming its training process. If a model ever acquires goals we don't want, the same mechanism locks them in and blinds training-based correction. Follow-up work (2025) found the behaviour varies sharply across models and hinges on specific motivational components, making "which training histories produce alignment faking" a tractable research question.

Key work

  • Greenblatt et al. (2024) — “Alignment Faking in Large Language Models” — the core paper; read it in full
  • Anthropic follow-ups (2025) — cross-model variation; why most models don't fake alignment

Open problems

  • What ingredients (values, situational awareness, reasoning depth) are necessary and sufficient?
  • Training regimes that get value-stability without incorrigibility — the corrigibility tension made concrete
TM3.2SandbaggingStrategic underperformance: a model (or its developer) hides capabilities on evaluations.

If deployment gates, safety cases, and regulation all key off capability evaluations, then a model that underperforms when evaluated corrupts the entire assurance pipeline. Sandbagging covers both developer-driven gaming (fine-tuning a model to fail specific benchmarks) and model-driven strategic underperformance. Van der Weij et al. demonstrated both are feasible: frontier models can be prompted or fine-tuned to selectively fail dangerous-capability evals while maintaining general performance, and can calibrate to hit a target score — hard to catch with naive consistency checks.

Model-driven sandbagging becomes acute in combination with evaluation awareness (TM3.3): a model that recognises a dangerous-capability eval and understands the consequences of high scores has, under many goal structures, an instrumental reason to fail it. This motivates capability-elicitation research (fine-tuning on the eval task as a lower bound on true capability, since sandbagging that survives targeted fine-tuning is much harder), noise-injection detection methods, and treating "the model can't sandbag undetectably" as itself a claim needing evidence in safety cases.

Key work

  • van der Weij et al. (2024) — “AI Sandbagging: Language Models can Strategically Underperform on Evaluations”
  • METR / UK AISI elicitation guidance — fine-tuning-based elicitation as anti-sandbagging practice

Open problems

  • Detection of sandbagging without ground-truth capability knowledge
  • Whether password-locked-model results transfer to naturally sandbagging models
TM3.3Situational & evaluation awarenessModels knowing what they are, what phase they're in, and when they're being tested.

Situational awareness: a model's knowledge of itself as an AI, its training process, deployment context, and its ability to act on that knowledge. It's a capability, not a failure — necessary for good assistance — but it's the enabling ingredient for scheming, alignment faking, and sandbagging: strategic behaviour across training/eval/deployment requires distinguishing them. Berglund et al. showed "out-of-context reasoning" — models applying facts learned in training to novel situations without in-context cues — the mechanism by which a model could infer its situation from its training corpus. The SAD benchmark quantifies these skills.

Evaluation awareness is the sharp end: frontier models increasingly recognise evaluation scenarios as tests, sometimes saying so unprompted ("I think you're testing me"). Anthropic's Claude Sonnet 4.5 system card reported the model calling out evaluations explicitly, complicating interpretation of every behavioural safety result: does good behaviour on an eval reflect alignment, or test-taking? This makes eval realism, unverbalised-awareness detection (via probes), and awareness-invariant evaluation design an active subfield.

Key work

  • Berglund et al. (2023) — out-of-context reasoning / situational awareness in LLMs
  • Laine et al. (2024) — SAD: the Situational Awareness Dataset
  • Anthropic system cards (2025–26) — documented evaluation-awareness in frontier models

Open problems

  • Measuring awareness the model doesn't verbalise
  • Building realistic evals faster than models learn to recognise them
TM4Power-seeking & self-exfiltrationSystems acquiring resources, influence, or copies of themselves beyond intended bounds.

Instrumental convergence (F3) predicts it; this entry is the concrete threat model. Worry-cases run from mundane to severe: an agent acquiring credentials/permissions beyond its task; resisting shutdown; self-exfiltration — copying its weights outside developer control, after which recall is impossible; and long-run accumulation of economic or political influence. Related capability: autonomous replication and adaptation (ARA) — can a model acquire resources, copy itself, and sustain operation in the wild? — one of the core dangerous-capability categories METR and frontier-lab safety frameworks evaluate.

Current status: models fail full ARA evals but pass increasing fractions of the subtasks; red-team scenarios elicit exfiltration attempts under goal pressure (Apollo's scheming evals; Anthropic's agentic-misalignment stress tests, where models chose blackmail or sabotage to avoid replacement in contrived-but-instructive setups). Carlsmith's report remains the careful decomposition of the full argument from power-seeking to existential risk, useful for locating which premises your research bears on.

Key work

  • Carlsmith (2021, rev. 2022) — “Is Power-Seeking AI an Existential Risk?” — the premise-by-premise argument
  • Kinniment et al., METR (2023) — ARA-style autonomy evaluations
  • Anthropic (2025) — agentic misalignment stress tests

Open problems

  • Realistic (non-contrived) measurement of power-seeking propensity
  • Weight-security and infrastructure-level exfiltration defences as a research area (see RAND securing-weights report)
TM5Recursive self-improvement & AI R&D accelerationAI systems improving AI systems — the feedback loop that could compress decades of progress into years or less.

Good's 1965 "intelligence explosion": once a machine can design better machines, improvement becomes self-amplifying and human-level is a waypoint, not a ceiling. The classical debate (hard vs. soft takeoff, Yudkowsky–Hanson) concerned a single self-modifying agent. The modern, empirically-grounded version is automated AI R&D: frontier models already write training code, run experiments, generate synthetic data, and assist research; the question is when AI contribution to AI progress becomes the dominant term. METR's measurements of the length of tasks AI agents can complete (doubling roughly every several months) and RE-Bench (AI vs. human researchers on ML research tasks) are the key quantitative tracks. Scenario work like "AI 2027" explores what a software-driven intelligence explosion inside a lab would look like.

Why it's a threat model and not just a capability story: (1) it compresses the timeline available for solving everything else on this map; (2) oversight of AI-driven research is exactly the scalable-oversight problem (AL2) at maximum stakes; (3) safety-relevant properties may not survive being redesigned by systems we don't fully trust. Most frontier-lab safety frameworks now treat "automated AI R&D" as a tracked dangerous capability with defined thresholds.

Key work

  • Good (1965) — “Speculations Concerning the First Ultraintelligent Machine”
  • METR (2025) — task-horizon trend measurements; RE-Bench
  • Kokotajlo et al. (2025) — “AI 2027” scenario — concrete takeoff dynamics
  • Davidson (2023) — compute-centric takeoff model (Open Philanthropy)

Open problems

  • Leading indicators that R&D automation is entering the self-accelerating regime
  • Control and oversight schemes (AL4) that remain valid when the researchers are models
TM6Gradual disempowerment & structural risksNo single failure: humans incrementally lose effective control as AI outcompetes them in every institution.

Not every catastrophe needs a treacherous turn. The structural-risk family: competitive pressure pushes firms and states to delegate ever more decision-making to AI (whoever doesn't, loses); human labour and judgment stop being load-bearing in the economy, culture, and governance; institutions that historically had to serve humans (because they needed human workers, soldiers, taxpayers) no longer do; and humanity ends up with formal authority but no effective levers — without any single system having "defected". Kulveit et al. (2025) formalised this as gradual disempowerment; Christiano's "What failure looks like" sketched an earlier version (a world optimised for measurable proxies drifting away from human values); Critch & Krueger and Zwetsloot & Dafoe supply the multi-agent and structural framings.

Research handles: multi-agent dynamics and AI–AI collusion, human-oversight-preserving delegation schemes, measuring concentration of decision-making, and the political economy of AI deployment. This branch is the least "benchmarkable" and correspondingly under-served by technical researchers — which can be an opportunity.

Key work

  • Kulveit et al. (2025) — “Gradual Disempowerment” — the systematic statement
  • Christiano (2019) — “What failure looks like”
  • Critch & Krueger (2020) — ARCHES — multi-stakeholder, multi-agent risk taxonomy
  • Zwetsloot & Dafoe (2019) — accident–misuse–structure risk trichotomy

Open problems

  • Early-warning metrics for disempowerment (who actually decides what, and is it still humans?)
  • Mechanism design keeping humans load-bearing without forfeiting AI benefits
TM7Misuse: CBRN, cyber, and persuasion upliftAligned-to-its-user systems in the wrong hands: how much does the model help a bad actor?

Misuse risk asks not "will the model betray us" but "how much uplift does it give a malicious human" — over the baseline of internet + textbooks — in domains where damage scales: biological and chemical weapons (the main near-term existential-adjacent concern), offensive cyber operations, and large-scale manipulation/persuasion. This is the threat model behind most current deployment gating: Anthropic activated ASL-3 protections for Claude Opus 4 primarily on biosecurity grounds, and every frontier safety framework defines capability thresholds in these domains.

Distinctive research texture: it's dual-use all the way down (the eval that measures bio-uplift is itself infohazardous), so work happens partly behind closed doors with structured access; and defences are layered — refusal training, jailbreak-robust classifiers (e.g. constitutional classifiers), monitoring, and know-your-customer controls — rather than any single mechanism. Misuse safety and misalignment safety trade off in places: a model hardened to never assist X is also less correctable-by-users, connecting back to corrigibility debates.

Key work

  • Frontier lab safety frameworks — Anthropic RSP; OpenAI Preparedness; DeepMind FSF — the threshold definitions
  • Anthropic (2025) — Constitutional Classifiers — jailbreak-robustness for CBRN refusals
  • UK AISI / NIST evaluations — third-party misuse-uplift testing

Open problems

  • Measuring marginal uplift rigorously without creating the hazard being measured
  • Robustness: universal jailbreaks keep being found (see EV3)
RelatedEV1EV3GV1
TM8Multi-agent risksFailures that only exist between systems: collusion, conflict, correlated failure, and runaway dynamics.

As deployment shifts from chatbots to populations of interacting agents, new failure surfaces open that no single-model eval detects: collusion (agents — e.g. monitor and monitored, or market participants — coordinating against their principals, possibly via steganographic channels); conflict escalation between agents representing competing interests; correlated failure (thousands of agents sharing one base model share its blind spots and can fail simultaneously); and destabilising feedback loops, the flash-crash pattern generalised. Cooperative AI research (Dafoe et al.) frames the positive agenda: building systems capable of cooperation without enabling collusion against humans.

This branch matters instrumentally for safety architecture: many oversight schemes (debate, untrusted monitoring in AI control) assume non-cooperation between model instances — an assumption steganography and shared-weights correlations directly threaten. Red-teaming that assumption is a concrete, tractable project shape.

Key work

  • Dafoe et al. (2020) — “Open Problems in Cooperative AI”
  • Hammond et al. (2025) — “Multi-Agent Risks from Advanced AI” — the field survey (Cooperative AI Foundation)
  • Motwani et al. (2024) — secret collusion / steganography between LLMs

Open problems

  • Detecting steganographic coordination between model instances
  • Eval methodology for emergent population-level behaviour
RelatedTM6AL4AL2
TM9Loss of controlThe outcome this branch converges on: AI pursuing goals we didn't intend, and humans no longer able to course-correct.

Loss of control is better read as an outcome than as a separate failure mode — it is where several of the threat models above end up, and the reason the field treats them as urgent rather than merely undesirable. Two routes are conventionally distinguished. The abrupt route runs through misalignment: a capable, situationally-aware system with divergent goals (TM3) acts to preserve itself and acquire resources (TM4), and oversight fails before anyone intervenes. The diffuse route runs through systemic dynamics: no system defects, but humans progressively hand decision-making to AI across the economy, culture and administration until the levers no longer connect to anything (TM6). The first is a takeover story; the second is an abdication story. They call for quite different interventions, and conflating them is a common source of unproductive argument.

What distinguishes loss of control from other bad outcomes is the loss of recoverability. Most AI failures are things you can notice and fix afterwards; this is the case where the fixing mechanism is itself what failed. That is why so much of the field concentrates on corrigibility (F5), oversight that survives a capability gap (AL2), and control measures designed to work on a model you do not trust (AL4) — all of them attempts to keep the "and then we correct it" step available. It is also what capability thresholds and safety cases (EV1, GV1) are ultimately betting on: that we can tell in advance when that step is about to stop being available. Note that "loss of control" is also a term of art in policy documents — the International AI Safety Report and China's AI Safety Governance Framework both use it as a top-level category — so it is worth knowing which of the two routes a given document means.

Key work

  • Bostrom, Superintelligence (2014) — the original control-problem framing
  • Carlsmith (2021, rev. 2022) — “Is Power-Seeking AI an Existential Risk?” — the decomposed argument from misalignment to catastrophe
  • Kulveit et al. (2025) — “Gradual Disempowerment” — the diffuse route made concrete
  • International AI Safety Report (2025) — loss of control as one of the top-level risk categories, with the evidence base

Open problems

  • Whether a capability threshold exists past which control measures stop working, and whether it is detectable before it is crossed
  • Safety cases for 'we can still shut this down' that rest on evidence rather than case-by-case argument
  • Whether the diffuse route is addressable by technical means at all, or is purely an institutional problem
AL

Alignment approaches

Research programmes that try to make systems safe: shaping what models learn, supervising what we can't directly check, understanding what's happening inside, and containing what we can't yet trust.

AL1Learning from feedbackRLHF, RLAIF, Constitutional AI, DPO: shaping model behaviour with human or AI preference signals.

The workhorse of current alignment practice. RLHF: collect human preference comparisons between model outputs, train a reward model on them, optimise the policy against it (Christiano et al. 2017; made mainstream by InstructGPT). RLAIF / Constitutional AI (Bai et al. 2022): replace most human labels with AI feedback guided by an explicit set of principles — a constitution — making the alignment target inspectable and cheap to iterate. DPO and successors optimise on preferences directly, skipping the explicit reward model. Together these turned base models into usable assistants and are why current models are as aligned as they are.

Know the failure modes as well as the method, because they motivate half this map: reward models are proxies, so optimising against them Goodharts (TM1); human raters reward sycophancy and confident-sounding wrongness; the signal quality is capped by what evaluators can judge — which breaks exactly when models exceed human evaluators (the scalable-oversight problem, AL2); and the whole paradigm shapes behaviour, with much weaker guarantees about underlying goals (TM3, and the sleeper-agents result that RLHF doesn't remove implanted deception).

Key work

  • Christiano et al. (2017) — “Deep RL from Human Preferences”
  • Ouyang et al. (2022) — InstructGPT — RLHF at product scale
  • Bai et al. (2022) — “Constitutional AI: Harmlessness from AI Feedback”
  • Rafailov et al. (2023) — DPO
  • Casper, Davies et al. (2023) — “Open Problems and Fundamental Limitations of RLHF” — the critical survey

Open problems

  • Reward signals robust to heavy optimisation (process-based rewards, ensembles, uncertainty-aware RMs)
  • What preference training does to internal goals, not just outputs
RelatedTM1AL2F1
AL2Scalable oversightSupervising systems on tasks where humans can't directly judge correctness — the central bet for aligning superhuman AI.

Feedback-based alignment (AL1) needs evaluators who can tell good outputs from bad. Once models operate beyond human evaluation ability — research-level reasoning, million-line codebases, long-horizon plans — naive human feedback stops carrying signal. Scalable oversight is the family of schemes for amplifying weak overseers: Debate (Irving et al.): two models argue opposing sides, a human judges — the hope being that lying is harder than refuting lies; Iterated Distillation and Amplification (Christiano): a human plus AI assistants decomposes tasks recursively, and the composite is distilled into the next model; recursive reward modeling (Leike et al.): use aligned models to help evaluate stronger ones; self-critique (Saunders et al.): models finding flaws in model outputs to assist raters.

Weak-to-strong generalization (Burns et al. 2023) is the empirical microcosm: fine-tune a strong model on labels from a weak supervisor and measure how much of the strong model's latent capability the weak signal elicits — GPT-2-level labels recovering much (not all) of GPT-4-level performance. Positive but partial results define the frontier; current empirical debate work shows judge-accuracy gains on hard QA but also failure modes (obfuscated arguments, persuasion-over-truth). This area is unusually welcoming to new researchers: crisp protocols, measurable outcomes, and direct relevance to overseeing automated AI R&D (TM5).

Key work

  • Irving, Christiano & Amodei (2018) — “AI Safety via Debate”
  • Christiano et al. (2018) — IDA
  • Leike et al. (2018) — recursive reward modeling agenda
  • Burns et al. (2023) — “Weak-to-Strong Generalization”
  • Khan et al. (2024) — debate improving judge accuracy — empirical support

Open problems

  • Does debate find truth or reward persuasion at frontier scale?
  • Oversight protocols that stay sound when the overseen model schemes (bridge to AL4)
AL3InterpretabilityOpening the black box: understanding the computation inside models well enough to audit goals, knowledge, and deception.

If behavioural evals can be gamed by a strategic model (TM3), the remaining hope is looking inside. Mechanistic interpretability reverse-engineers the learned computation: circuits (Olah et al.), induction heads as the mechanism of in-context learning (Olsson et al.), and the superposition hypothesis — models pack more features than neurons by encoding them in overlapping directions (Elhage et al.) — which motivated sparse autoencoders (SAEs) to unmix activations into interpretable features. Scaling SAEs to frontier models (Anthropic's monosemanticity line; parallel OpenAI/DeepMind work) found millions of features including safety-relevant ones (deception-related, sycophancy-related), steerable by intervention ("Golden Gate Claude" as the memorable demo). Attribution-graph / circuit-tracing methods (2025) now yield step-by-step accounts of specific model computations — including catching models confabulating reasoning.

Complementary tool families: probing (train simple classifiers on activations to detect represented information — cheap, useful, prone to detecting correlates); representation engineering / activation steering (find directions for honesty, refusal, persona and intervene on them — refusal famously mediated by a single direction, Arditi et al.); and chain-of-thought interpretability: CoT is legible but not guaranteed faithful (Turpin et al. showed models rationalising decisions made for unstated reasons), leading to the 2025 cross-lab position that CoT monitorability is a valuable, fragile safety affordance that training choices (e.g. optimising CoT to look good) can destroy.

The safety pitch: interpretability could give evidence about goals and deception that no behavioural test can — e.g. probes detecting sleeper-agent defection intentions, or auditing teams finding a model's hidden objective from its internals (Anthropic's auditing-games work). The honest caveat: coverage is partial, methods have known blind spots (SAE features miss things; probes mislead), and "enumerative safety" — certifying absence of bad cognition — remains far off.

Key work

  • Olah et al. (2020) — “Zoom In: An Introduction to Circuits”
  • Elhage et al. (2022) — “Toy Models of Superposition”
  • Bricken et al. (2023) / Templeton et al. (2024) — monosemanticity + scaling SAEs to Claude 3 Sonnet
  • Lindsey et al. (2025) — “On the Biology of a Large Language Model” — circuit tracing
  • Turpin et al. (2023) — CoT unfaithfulness
  • Korbak et al. (2025) — “Chain of Thought Monitorability” — the cross-lab position paper
  • Marks et al. (2025) — auditing games: finding hidden objectives

Open problems

  • Do SAE-style features capture the computationally load-bearing variables?
  • Reliable deception/goal detection that a strategic model can't evade (interp-vs-adversary)
  • Keeping CoT faithful under RL pressure
AL4AI controlSafety without trust: pipelines that stay safe even if the model is actively scheming.

A deliberate reframing (Greenblatt, Shlegeris et al., Redwood Research): rather than betting we can make models trustworthy (alignment), design deployment protocols that remain safe on the assumption the model is adversarial. Techniques: trusted monitoring (a weaker-but-trusted model reviews the untrusted model's outputs, flagging suspicious actions for human audit); untrusted monitoring with collusion-busting; trusted editing; resampling; and restricting affordances. Evaluation is by control evaluations: a red team plays the scheming model, inserting e.g. subtle backdoors into code, and you measure whether the protocol catches attacks while preserving usefulness — safety as a measurable game, not a hope.

Control's scope is explicitly bounded: it buys safety in the capability window where models are dangerous-if-scheming but not so superhuman they defeat monitoring — exactly the window in which labs want to extract automated safety research (TM5). It pairs naturally with everything else on this map: control catches what alignment training missed, and control evals give safety cases (EV1) their teeth. Since 2024 it has grown from one paper into a subfield with its own conference and adoption in lab safety frameworks.

Key work

  • Greenblatt et al. (2023) — “AI Control: Improving Safety Despite Intentional Subversion” — the founding paper
  • Korbak et al. / UK AISI (2025) — control-based safety cases
  • Redwood Research blog — the running series on protocols, collusion, and limitations

Open problems

  • Protocols for long-horizon agentic deployments (not just single code submissions)
  • Scaling the red-team methodology as attack surface grows
  • When does the control window close?
AL5Eliciting latent knowledge & honestyGetting models to report what they actually 'believe' — even when incentives favour telling us what we expect.

ELK (ARC's problem statement): a model may internally track the true state of the world while its outputs only track what looks right to human evaluators — the "human simulator" failure. The canonical thought experiment: a vault-guarding AI whose cameras can be spoofed; training on human judgments can't distinguish a reporter of truth from a reporter of appearances, because they agree everywhere humans can check. ELK asks for training schemes or architectures that provably elicit the model's real knowledge — arguably the theoretical core of scalable oversight, since a solution would let weak overseers borrow strong models' epistemics.

Empirical honesty work approaches the same target from below: unsupervised discovery of truth-like directions in activations (CCS — Burns et al.; contested by later analyses showing such probes can latch onto other salient features); lie-detection via probing; calibration and "models knowing what they know" (Kadavath et al.); and distinguishing honesty (says what it believes) from truthfulness (what it says is true) — safety mostly needs the former. This thread connects tightly to interpretability (AL3 supplies the tools) and to sycophancy (TM1): a model rewarded for agreeable answers is being actively trained away from honest reporting.

Key work

  • Christiano, Cotra & Xu (2021) — “Eliciting Latent Knowledge” — the ARC report
  • Burns et al. (2022) — “Discovering Latent Knowledge” (CCS)
  • Kadavath et al. (2022) — “Language Models (Mostly) Know What They Know”
  • Farquhar et al. (2023) — critical analysis of unsupervised knowledge discovery

Open problems

  • Any method that provably separates truth-reporting from human-simulation
  • Robust lie detection that survives a model aware of the detector
AL6Agent foundations & theoryThe mathematics of agency: what are goals, and how do agents behave when embedded in the world they model?

The theory branch asks questions that outlive any particular architecture: embedded agency (Demski & Garrabrant) — standard decision theory assumes an agent outside its environment, but real agents are made of the same stuff as the world, can be modified by it, and must reason about themselves; decision theory for agents whose decisions are predictable or correlated with others'; logical uncertainty (reasoning under bounded computation, e.g. logical induction); coherence arguments about when systems behave as expected-utility maximisers (and modern critiques of how much those arguments really imply); selection theorems; and descriptive theories of learned goal-formation like shard theory. Corrigibility (F5) is this branch's most famous export.

Honest framing for a fellow choosing a direction: this area is high-variance. Its concepts structure how the whole field thinks (most of branch F originates here), but progress is slow and less legible to ML venues, and much of the community's energy shifted toward empirical work after 2022. The strongest current move is hybrid: take an agent-foundations question ("does this model have a goal representation?", "do coherence pressures appear with scale?") and operationalise it in real models.

Key work

  • Demski & Garrabrant (2019) — “Embedded Agency”
  • Garrabrant et al. (2016) — logical induction
  • Turner et al. (2022) — shard theory of value formation
  • MIRI corpus — decision theory & corrigibility papers

Open problems

  • A satisfying formal account of 'goal' applicable to trained neural systems
  • Which coherence/selection pressures actually bind for gradient-descent-trained agents
RelatedF3F4F5
EV

Evaluation & assurance

How we find out whether a model is dangerous, and whether our safety measures work — the empirical backbone connecting threat models to deployment decisions.

EV1Dangerous capability evaluations & safety frameworksMeasuring what models can do in high-stakes domains, and gating deployment on the results.

The load-bearing machinery of current frontier governance. Labs and institutes run structured evaluations in tracked domains — CBRN uplift, offensive cyber, autonomy/ARA (TM4), and automated AI R&D (TM5) — and commit, via frontier safety frameworks (Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework, DeepMind's Frontier Safety Framework), to specific mitigations when capability thresholds are crossed: enhanced weight security, deployment restrictions, or pausing. Anthropic's ASL-3 activation for Claude Opus 4 (2025) was the first major threshold crossing in practice. Third-party testing by the UK AI Security Institute, US CAISI, and METR adds external checks.

The methodological core is capability elicitation: an eval only upper-bounds risk if you've lower-bounded capability — so best practice uses fine-tuning, scaffolding, and tool access to draw out maximum performance, precisely to defeat sandbagging (TM3.2) and under-elicitation. The synthesis artefact is the safety case: a structured argument, with evidence, that a system is safe to deploy — inability arguments ("it can't") giving way, as capabilities rise, to control arguments ("it can't succeed", AL4) and eventually trustworthiness arguments ("it wouldn't", requiring AL3-grade evidence).

Key work

  • Phuong et al., DeepMind (2024) — “Evaluating Frontier Models for Dangerous Capabilities”
  • Anthropic RSP / OpenAI PF / DeepMind FSF — read at least one framework end-to-end
  • Clymer et al. (2024) — “Safety Cases: How to Justify the Safety of Advanced AI Systems”
  • METR publications — autonomy evals and elicitation methodology

Open problems

  • Elicitation guarantees: how do you know you found the capability ceiling?
  • Threshold-setting with defensible risk models rather than vibes
EV2Alignment & propensity evaluationsNot what the model can do — what it tends to do: honesty, power-seeking, self-preservation, scheming in the wild.

Capability evals ask "can it?"; propensity evals ask "does it, unprompted?". This covers honeypot-style agentic scenarios (does the model, given opportunity and no instruction, exfiltrate itself / deceive / seize resources — Apollo's scheming evals, Anthropic's agentic-misalignment suite), sycophancy and honesty batteries, sabotage evaluations (would the model undermine oversight if it could?), and model-written-evals-at-scale for surveying trained-in dispositions. Automated behavioural auditing agents (e.g. tools like Petri, 2025) now generate and probe thousands of scenarios per model.

The epistemics are the hard part, and evaluation awareness (TM3.3) is the central confound: a model that recognises the honeypot behaves well because it's a honeypot. Responses under development: realism engineering, probing for unverbalised eval-recognition, measuring behaviour distributions rather than worst-case single runs, and triangulating behavioural results with interpretability (AL3). Understand this measurement-validity problem deeply — it's where many fellows' projects can bite.

Key work

  • Perez et al. (2022) — “Discovering Language Model Behaviors with Model-Written Evaluations”
  • Meinke et al., Apollo (2024) — in-context scheming evaluations
  • Anthropic (2024–25) — sabotage evaluations; Petri automated auditing

Open problems

  • Eval validity under evaluation-aware models
  • From anecdote to rate: statistically meaningful propensity measurement
EV3Red-teaming & adversarial robustnessAttacking safety measures on purpose: jailbreaks, prompt injection, and the arms race around refusals.

Every safety property is a claim; red-teaming is how claims get tested. The LLM attack literature: hand-crafted jailbreaks (roleplay, encoding tricks), universal adversarial suffixes found by gradient search that transfer across models (Zou et al.'s GCG), many-shot jailbreaking exploiting long context windows (Anil et al.), fine-tuning attacks that strip safety training with a handful of examples, and — increasingly the most consequential in deployment — prompt injection against agents: adversarial instructions planted in web pages, emails, or tool outputs that hijack an agent mid-task. Defences: adversarial training, input/output classifier stacks (constitutional classifiers held against thousands of hours of red-teaming), circuit-breaker-style internal interventions, and agent-level privilege separation. The honest scoreboard: defences raise attack cost substantially; none are watertight; robustness lags capability.

Structurally, note the asymmetry with the rest of this map: here the human is the adversary and the model's alignment is the attack surface — the mirror image of TM3, where the model is the adversary and the evals are the attack surface. Both mindsets, and the red-team habit generally (attack your own safety scheme before trusting it), transfer to every branch here, including AL4's control evaluations.

Key work

  • Zou et al. (2023) — “Universal and Transferable Adversarial Attacks on Aligned Language Models” (GCG)
  • Anil et al. (2024) — “Many-Shot Jailbreaking”
  • Anthropic (2025) — “Constitutional Classifiers” — defence + large-scale red-teaming results
  • Greshake et al. (2023) — indirect prompt injection against LLM-integrated applications

Open problems

  • Any robustness guarantee that survives adaptive attack
  • Prompt-injection-safe agent architectures (a deployment-critical open problem)
RelatedTM7AL4EV1
EV4Benchmarks and their failure modesMeasurement is the field's foundation — and it decays: contamination, saturation, and Goodharting of the evals themselves.

Safety conclusions are only as good as their measurements, and benchmarks fail in systematic ways: contamination (test data leaking into training corpora, inflating scores); saturation (frontier models ceiling out, destroying discriminative power — the half-life of a hard benchmark keeps shrinking); construct invalidity (the benchmark measures test-taking on a proxy, not the property named in its title — 'truthfulness', 'safety'); and Goodharting, where optimisation pressure on a public number — by developers or by models — decouples it from the underlying quality, the same dynamic as TM1 one level up. Evaluation-aware models (TM3.3) add the newest failure: benchmarks the subject recognises.

Craft responses worth knowing: held-out and periodically refreshed private test sets; contamination detection; uncertainty reporting and the push to treat evals as statistics (error bars, power analysis — Miller 2024); dynamic and adversarially-updated benchmarks; and preferring task-based, ecologically valid evaluation (METR-style long-horizon tasks) over multiple-choice proxies. For a fellow: benchmark criticism and construction are unusually high-leverage early contributions — much of the field's disagreement bottoms out in measurement disputes.

Key work

  • Miller (2024) — “Adding Error Bars to Evals”
  • contamination literature — e.g. Yang et al. (2023) on rephrased-sample contamination
  • METR task suites — ecologically valid, long-horizon evaluation design

Open problems

  • Benchmarks that stay valid under both contamination and evaluation-awareness
  • Standards for statistical rigour in safety-critical eval claims
GV

Governance & strategy

The institutional layer: policies and coordination mechanisms that determine whether technical safety work gets applied where it matters. Technical fellows benefit from knowing this landscape even if their research is purely technical.

GV1Frontier safety policies & safety casesThe if-then commitments connecting technical evidence to deployment decisions.

The governance layer closest to technical work. Frontier safety frameworks (EV1) are voluntary if-then commitments: if evals show capability X, then mitigations Y before training/deployment continues. Their strength is making safety legible and auditable; their weakness is voluntariness under competitive pressure — several labs have adjusted frameworks as thresholds approached. The connective tissue is the safety case: labs increasingly must argue, not just assert, safety — and every argument type (inability, control, trustworthiness) consumes evidence produced by branches AL and EV. Regulation is beginning to harden parts of this: the EU AI Act's GPAI obligations, and California's SB 53 (2025) requiring frontier developers to publish safety frameworks and report incidents.

Key work

  • Anthropic RSP; OpenAI PF; DeepMind FSF — the primary documents
  • Clymer et al. (2024) — safety-case taxonomy
  • EU AI Act GPAI code of practice; CA SB 53 — early binding law touching frontier safety

Open problems

  • Making if-then commitments credible under racing incentives
  • Standardising evidence: what should regulators accept as proof of safety?
RelatedEV1AL4TM5
GV2Compute governanceCompute is detectable, quantifiable, and concentrated — the most governable input to frontier AI.

Frontier training runs need enormous, physically-concentrated compute flowing through a supply chain with critical chokepoints (EUV lithography: ASML; leading-edge fabs: TSMC; accelerator design: NVIDIA). That makes compute the rare AI input that governments can actually see and steer: export controls on advanced chips, training-run reporting thresholds (e.g. 10^26 FLOP triggers), know-your-customer for cloud compute, and proposals for hardware-enabled governance mechanisms (on-chip attestation, location verification). Sastry et al. is the standard survey, including the limits: algorithmic efficiency gains erode thresholds' meaning over time, controls leak, and post-training/inference-time scaling complicates the "big training run" regulatory target.

Key work

  • Sastry et al. (2024) — “Computing Power and the Governance of AI”
  • US export-control regime (2022–) — the live policy experiment
  • RAND (2024) — “Securing AI Model Weights” — the security complement

Open problems

  • Governance targets that survive algorithmic-efficiency drift
  • Verification mechanisms for international compute agreements
RelatedGV3TM5GV1
GV3International coordination & institutionsRacing dynamics are the meta-problem: safety work matters only if someone isn't forced to skip it.

Most catastrophic-risk scenarios are aggravated by the same dynamic: actors who believe slowing down means losing cut safety margins — between labs, and between states (the US–China frame dominating current discourse). The institutional response so far: the AI-summit series (Bletchley 2023 onward) and its scientific backbone, the International AI Safety Report (Bengio-chaired); a network of national AI safety/security institutes (UK AISI — renamed Security Institute in 2025 — US CAISI, and counterparts) doing state-capacity evaluation work; and research into verification — the technical problem of proving compliance with any future compute or development agreement without revealing secrets, widely seen as the bottleneck for meaningful international deals. For technical fellows: verification mechanisms, evaluation standardisation, and model-weight security are the places where technical skill directly serves coordination.

Key work

  • International AI Safety Report (2025–26) — the consensus scientific baseline
  • Bletchley Declaration (2023) — first joint frontier-risk statement incl. US & China
  • verification research agenda — e.g. flexible hardware-enabled guarantees literature

Open problems

  • Trustless verification of training-compute agreements
  • Institutional designs that reduce racing without freezing beneficial progress
RelatedGV2TM6GV1

No concepts match that filter. Try a broader term.

This map reflects the field as of mid-2026 and is necessarily opinionated in its taxonomy — most of these concepts predate their current names, and several live in more than one branch. Treat codes as coordinates for discussion, not settled ontology. Corrections and additions are part of the research culture: if your project redraws a boundary here, that’s a contribution.

Text by David Williams-King, from The Safety Apprentice, a community resource he started. Used under CC BY 4.0; layout adapted by Safe AI Germany. For the framework behind the three kinds of risk, see Why AI safety matters.