Inside the Race to Measure Frontier Intelligence

Video summary

Vals CEO Rayan Krishnan argues independent evaluations caught Llama 4 underperforming and help enterprises measure AI’s real-world value.

Vals CEO Rayan Krishnan tells a16z’s Erik Torenberg, Ben Horowitz, and Jennifer Li that public AI benchmarks are increasingly easy to optimize for, making independent, private evaluations essential. He cites Meta’s Llama 4: it looked impressive on public tests but underperformed on Vals’s held-out benchmarks. Vals runs rapid pre-release testing, builds evaluations for finance and coding, and created a recursive self-improvement index using proxies for the work involved in training a model’s next version. As AI agents tackle longer tasks, evaluations must assess more complex outputs alongside cost and latency. Krishnan says one Fortune 10 company’s coding-tool limits reshaped engineers’ workdays, while Vals’s own token-maxing experiment reached about $1.5 million in monthly token costs—10 times employee salaries. He argues evaluations can help firms choose tools, demonstrate ROI, and give policymakers evidence on risks, while governments set and enforce rules and independent evaluators test whether models meet them.

Chapters

1. 0:00Why Vals Exists: Rayan Krishnan Says Public Benchmarks Could Not Track New Models
2. 2:52The Llama 4 Disaster: Vals’ Private Tests Contradicted Public Scores
3. 4:05Inside the 6-Hour Pre-Release Window: Vals Uses Distributed Tests and Steve
4. 6:00Evaluation Limits: Defining Workplace Standards and Avoiding Pay-to-Pass Benchmarks
5. 10:00The Recursive Self-Improvement Index: Vals Uses Proxies Across Model-Building Stages
6. 13:13Beyond Capability: Long-Running Agent Tests, Richer Criteria, and Updated Case Law
7. 16:58When Token Spend Could Eclipse Salary Spend: A Fortune 10 Firm Raised Limits to $300
8. 19:19Private Repos vs Public Benchmarks: Valsmith Tests Models on Company Code
9. 22:40How Vals Uses Vals: A $1.5 Million Token Experiment Found Efficient Coding Tools
10. 24:48Policy: Rayan Calls for Evidence-Based Standards and Third-Party Testing
11. 28:32Alignment and Government Oversight: VALS Briefs Officials on Model Risks
12. 33:30Geopolitics of Evals: Shared Standards for Sovereign AI and Recursive Self-Improvement
13. 37:14Future Benchmarking: VALS Expands Cyber Tests to Cloud and Grid Infrastructure

What the full analysis includes

13 chapters · 61 key points · 13 section summaries · mind map

The remaining 58 key points, section summaries and mind map are in the full analysis. Analyze this video to see everything →

This video is 40 min. The Free plan analyzes videos up to 30 min; plans from Plus cover videos up to 2 hours.