Mathematical Scientist for AI Safety Research

Frontier AI companies are throwing billions of dollars into scaling existing architectures and methods such as next-token-prediction, direct preference optimization (DPO), reinforcement learning with human feedback (RLHF), and reinforcement learning with verified rewards (RLVR). These methods are very powerful, yet fundamentally flawed, resulting in misalignment, sycophancy, systematic biases, and other forms of harmful behavior that are already having severely negative consequences in our society.

LawZero is a non-profit founded by Yoshua Bengio developing a fundamentally new approach, the Scientist AI. We are inspired by the fact that scientific theories are both generally useful and, unlike an untrusted agent, equivariant to the consequences of their use (for a broad overview, see our blogpost). We aim not only to build a novel, safe-by-design system, but to construct a theoretical blueprint for safe AI systems, which are at once capable and come with probabilistic safety guarantees, under finite data and compute constraints.

We are seeking an outstanding theoretical researcher to contribute to the foundational and theoretical efforts here working with Yoshua Bengio and our world-class team of researchers and engineers.

Key responsibilities

• Boldly go where no one has gone before in exploring new AI system architectures, theoretical frameworks, and new safety mechanisms in search of a blueprint for safe AGI and corresponding safety guarantees..
• Work closely with Yoshua Bengio and other leading researchers to analyze and refine the formal requirements needed to safely build and deploy the Scientist AI, including proving probabilistic safety guarantees under finite data and compute constraints; and, as the system matures, helping our research teams with empirical validation.
• Analyze, evaluate and contribute to the existing theoretical work already undertaken at LawZero by Yoshua Bengio and leading researchers, including the writing and dissemination of scientific papers.
• Read vigorously to stay up-to-date with relevant AI safety and machine learning research and deeply understand the risks posed by frontier models, including misalignment, reward hacking, instrumental goals, etc.
• Scrutinize and contribute to LawZero’s broader research agenda in order to guide and prioritize experiments to answer critical safety questions efficiently, in a way that does not prevent the highest level of capability.
• Share your knowledge about AI Safety and help researchers and engineers at LawZero better design and implement new ML algorithms and experiments.

Skills and qualifications

• A PhD in machine learning, computer science, mathematics, statistics, or an adjacent area relevant to the Scientist AI research program.
• A track record of research, especially in the formalization and fruitful technical analysis of subtle natural-language intuition.
• Strong interest and ideally experience with technical AI safety, including an exceptional ability for abstract systems thinking.
• A deep understanding of the mathematical underpinnings commonly used in the design and analysis of probabilistic machine learning, especially for neural networks and language models.
• The perseverance to work through hard and ill-defined problems, and to use confusion and dead-ends as challenges to dig deeper and further understanding.
• Clear communications: the ability to articulate complex ideas verbally and in writing to both technical and non-technical audiences.
• The ability to work in a collaborative environment and contribute to collective research goals.

Nice to have:

• The ability to regularly travel alongside Yoshua and team members to conferences is a big plus, as it will significantly accelerate the work!