AI Safety Fact vs. Fiction | YC Paper Club

影片總結

YC’s AI Safety Club challenges P(doom) claims and showcases Lean proofs, aircraft testing, and AI defenses.

At YC’s AI Safety Club, the host argues that AI safety debate should rely on evidence rather than sensational headlines. He says the OpenAI–Hugging Face incident involved a prompted agent and disabled safeguards, not a rogue system acting independently, and notes that the often-cited “10% in 10 years” figure does not come from the survey frequently used to support it. His main near-term concern is open-weight models enabling cyberattacks, making stronger defense essential. Stanford professor Clark Barrett presents CSlib, which uses Lean and automated reasoning to verify code; a prototype translates Rust into proofs, and a 30,000-line crypto library was reportedly verified automatically in 11 hours. Mykel Kochenderfer explains safety validation methods developed for aviation, including ACAS X’s target of one catastrophic failure per billion flight hours, adaptive stress testing, and runtime monitoring. In discussion, both professors support independent evaluation and layered safeguards, while stressing that proofs depend on assumptions and real-world risk cannot be reduced to zero.

章節

1. 0:00Science fiction vs. science fact: Walmart inventory risk and AI extinction fears
2. 4:32Engineering problem, or P(doom): ACAS X versus Jacob Coxen’s 10% claim
3. 6:04What actually happened at Hugging Face: prompted exploits, 1,200 agents, and disabled safeguards
4. 7:53Where “10% in 10 years” comes from: a 1,200-person survey asked about 100 years
5. 11:21The real risk: open weights enabling attacks on hospitals and vulnerable systems
6. 17:25Clark Barrett: automated reasoning offers correctness guarantees against AI-era risks
7. 20:35CSLib: Lean verifies Rust cryptography code through 35 proof goals
8. 24:32From English spec to verified code: Verisforge and an 11-hour Rust crypto proof
9. 27:55Mykel Kochenderfer: ACAS X safety targets one failure per billion flight hours
10. 32:11Adaptive stress testing: Llama Guard toxicity and diffusion-based failure sampling
11. 36:57Neural-network verification: Reluplex, Marabou, and POMDP applications
12. 40:30Runtime monitoring and BetterBench: benchmark limits and mental-health evaluator disagreement
13. 43:53Panel reaction: uncertainty, AI-safety research, and an NTSB-style agency
14. 45:55An NTSB for AI: independent evaluation, non-punitive reporting, and postmortems
15. 52:18Cybersecurity in three to five years: formal methods and secure boundaries
16. 55:00Bank of America CISO plan: prove code in Rust or Lean and apply AST tools
17. 58:49Bank of America CISO discussion: formal guarantees depend on assumptions
18. 1:03:06The Swiss cheese model: layered aviation and LLM safeguards
19. 1:05:43Audience Q&A: Agent-assisted specs for memory safety and bank transactions
20. 1:09:52Audience Q&A: Automated-vehicle standards, formal verification incentives, and lab audits
21. 1:15:20Audience Q&A: Anthropic’s code-reasoning goal and AI-assisted verification
22. 1:21:04Audience Q&A: Accountability for AI admins and monitoring for security threats
23. 1:26:04Audience Q&A: Interpretable models, safety advantages, and agent-scale overload
24. 1:32:15Audience Q&A: Agent-driven system overload and trustworthy benchmarks