Paul Christiano — Preventing an AI takeover

Three hours with Dwarkesh Patel on how an AI takeover might actually happen and what it would take to prevent one. Christiano gives his timelines as numbers he openly calls rough (about 15% by 2030 and 40% by 2040 for AI able to build a Dyson sphere), and offers as his median story a gradual handing-over of control rather than a sudden strike. Asked whether RLHF, the technique he led, was net negative for accelerating AI after ChatGPT, he answers probably yes, but “not super clear.” He sets out the responsible scaling policies he has been pushing labs to adopt: name the threats, measure the capabilities, and pause if the promised safeguards can’t be met. Then he goes deep on ARC’s research bet, a formal account of what makes a good explanation of a model’s behavior, so that anomaly detection could notice when a model “goes from thinking it’s being trained to thinking it should defect.”