A Benchmark for Preventing Emergent Misalignment
Mentor: Florian Mai
Project area: Technical AI alignment
Project Language
Minimum Time Commitment
10 hours per week.
Project Abstract
Emergent misalignment (EMA) is the phenomenon that AI models can become broadly misaligned when fine-tuned on a narrow data set, e.g. code with security vulnerabilities, bad medical advice, or seemingly innocent data such as unpopular aesthetic preferences.
As frontier AI labs allow the fine-tuning even of their most advanced models, EMA occurring inadvertently in an uncontrolled fashion could potentially lead to rogue AI scenarios. To prevent this, we must develop mitigation methods that can be applied during training. These methods should not only prevent EMA reliably, but also keep the alignment tax low to incentivize AI labs to adopt them. To this end, the mitigation methods should be cheap and not reduce performance on a variety of benign fine-tuning tasks.
The goal of this project is to develop a benchmark for EMA mitigation methods that is easy enough to use that it allows rapid experimentation with novel mitigation methods. To this end, the project will develop an open source code repository with simple interfaces that seamlessly integrates with many model families, downstream task types, and mitigation methods. It will seek to standardize the evaluation protocol, compute key metrics such as runtime and memory cost in relation to a baseline. Finally, the project will add support for various tasks from both supervised finetuning and reinforcement learning settings, and conduct a thorough comparison of existing mitigation methods.
This project has the potential to directly speed up AI safety research in a context that is relevant to state-of-the-art AI models today. Participants will learn about emergent misalignment as a potentially catastrophic failure mode of frontier AI, and how to design evaluation protocols to stress test mitigations in a way that is relevant to frontier AI labs. Finally, they will contribute to an open source repository and paper report, strengthening their profile as AI safety researchers more generally.
Theory of Change
Bad frameworks produce bad decisions. The question of machine moral status will increasingly affect AI development and governance. Currently, most people reasoning about it lack adequate conceptual tools. This matters for catastrophic risk in several ways.
Under-reaction: if AI systems develop welfare-relevant internal states and we lack frameworks to recognize this, we may create systems with misaligned interests while dismissing their signals as "mere computation." A system that experiences something like suffering under certain conditions, and whose operators dismiss this, is a system with reason to deceive.
Over-reaction: anthropomorphizing systems that lack morally relevant properties wastes attention and resources, and may constrain beneficial AI development without corresponding benefit.
Poor discourse: without shared conceptual foundations, public debate about AI consciousness polarizes between dismissive and credulous positions. Neither serves good governance.
The primer addresses these by training researchers and practitioners to reason carefully across multiple frameworks, recognize what each assumes, and navigate uncertainty without false confidence. The German focus (incorporating European philosophical traditions, piloting with German-speaking users) builds SAIGE's national infrastructure while contributing to the broader field.
Conceptual clarity is infrastructure. This project builds it.
Desired Mentee Background
Computer Science/ML.
Desired Mentee Level of Education
Any level.
Other Mentee Requirements
- Basic understanding of machine learning and neural networks
- Familiarity with pytorch and the huggingface ecosystem is a plus, but can be learned quickly enough
- Strong critical thinking and creativity