Let’s be honest. Most AI benchmarks are trash.
They’re basically just memory tests where models spit out training data they’ve already seen. If you’ve spent five minutes on Twitter or LinkedIn lately, you’ve seen the charts. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are all fighting over decimal points on tests like MMLU or GSM8K. But those tests are getting old. They’re leaked. They’re "contaminated."
Then came humanity's last exam leaderboard.
It’s a heavy name. It sounds like a plot point from a mediocre sci-fi novel where a giant computer decides whether to vaporize Earth based on our ability to solve a Rubik's cube. In reality, it’s one of the most sobering reality checks the AI industry has faced in years. It’s a project specifically designed to find the "ceiling" of LLM intelligence by using questions that are so hard, even experts in those specific fields struggle to answer them without a few hours and a lot of coffee.
What actually is humanity's last exam leaderboard?
The project is officially known as Humanity's Last Exam (HLE). It was launched as a massive, crowdsourced effort by the Center for AI Safety (CAIS) and Scale AI. The goal? To create a dataset that models cannot simply memorize.
👉 See also: Elon Musk on Joe Rogan 2024: What Really Happened Behind the Scenes
If you look at the HLE leaderboard, you won't see 99% scores. You’ll see models failing. Hard.
The creators—including Dan Hendrycks, who has been a vocal figure in AI safety and evaluation—realized that if we keep testing models on high school chemistry or Intro to Psych questions, we’re going to hit a wall. We need to know if these things can actually reason through problems that require PhD-level intuition. They gathered thousands of questions from subject matter experts across dozens of disciplines.
Why the name matters
The name isn't just edge-lord marketing. It reflects a legitimate concern: if an AI can out-reason the collective expertise of humanity across every single specialized field, what does "human-level intelligence" even mean anymore?
It’s about the frontier.
Most people think AI is getting "smarter" because it can write a funny poem or debug a Python script. But "smart" is relative. The HLE leaderboard tries to measure the gap between "really good assistant" and "expert-level researcher."
The leaderboard reality check
So, who is winning? Honestly, nobody is "winning" in the traditional sense.
When the initial results for humanity's last exam leaderboard started trickling out, the scores were humbling. While models might hit 80% or 90% on standard benchmarks, they were often tanking into the 10% to 30% range on HLE.
Take a look at the spread.
- o1-preview (OpenAI): Generally considered the heavyweight in reasoning right now. It does better because it "thinks" before it speaks, but even it hits walls.
- Claude 3.5 Sonnet (Anthropic): Often praised for its nuance, yet struggles with the sheer technical density of these questions.
- Llama 3 (Meta): Showing that open-weight models are catching up, but the gap in high-level reasoning persists.
The fascinating thing isn't the rank. It's the delta.
There is a massive chasm between being able to summarize a PDF and being able to solve a niche problem in organic chemistry that requires five steps of deductive logic. Most LLMs are basically "vibe-check" machines. They predict the next most likely word. HLE forces them to actually navigate a logic path where one wrong "word" or "step" makes the entire answer garbage.
The problem with "Contamination"
We have to talk about contamination. It’s the dirty secret of AI development.
Because LLMs are trained on the open internet, they eventually "see" the questions they are tested on. If I write a test today and put it on a blog, by next month, GPT-5 (or whatever comes next) has already read the answer key during its training run.
HLE tries to solve this.
The organizers kept a significant portion of the exam secret. They also asked contributors to provide questions that weren't already online. This is huge. If you can't find the question on Google, the model probably hasn't seen it in its training set. This is the only way to test if the model is actually "thinking" (or simulating thinking effectively) versus just retrieving a memory.
I've talked to researchers who are genuinely frustrated by this. They spend millions training a model, only to find out its "high scores" were just a result of a massive, expensive game of Jeopardy where it already had the cards. Humanity's last exam leaderboard is the first time we've seen a consistent, high-pressure environment that treats the models like students in a room with no internet access.
Why you should care (even if you aren't a coder)
You might be thinking, "Who cares if a bot can't solve a PhD physics problem?"
You should care because this is the yardstick for the next economic shift. If AI can pass the HLE, it means it can do the work of a highly specialized consultant, a research scientist, or a high-level engineer.
We are moving from "AI that helps you write emails" to "AI that helps you discover new materials for batteries."
👉 See also: What is the Control Center on iPhone and How Does it Actually Work?
The leaderboard shows us exactly how far away we are from that reality. Right now, we’re still pretty far. The low scores on humanity's last exam leaderboard suggest that while AI is great at mimicking the structure of expert thought, it still lacks the depth of expert reasoning.
It's sort of like a parrot that knows how to say "E=mc^2" but has no idea why the 'c' is squared.
The crowdsourcing element
One of the coolest things about this specific project was how they got the questions. They didn't just hire a few guys. They opened it up.
Scale AI and CAIS offered prizes. Real money.
They wanted the hardest, most obscure, most "Google-proof" questions possible. This created a diverse set of challenges. You’ve got everything from abstract mathematics to deep-cut legal theory. It’s a testament to human creativity that we can still come up with problems that the most powerful computers on earth find confusing.
What happens when a model finally "beats" the exam?
That’s the "Last Exam" part of the name.
There’s a theory that once a model can pass a test of this caliber—one that is resistant to contamination and requires multi-step expert reasoning—we have reached AGI (Artificial General Intelligence).
But there’s a catch.
📖 Related: Google Wallet iPhone App: Why You Still Can’t Use It Like Android Users Do
Intelligence isn't a single number. Just because a model can solve a complex chemistry problem doesn't mean it has common sense. It doesn't mean it can navigate a physical kitchen or understand the emotional subtext of a broken relationship. The HLE is a benchmark for knowledge and reasoning, not for consciousness or agency.
Still, the leaderboard serves as a lighthouse. It shows the direction.
Navigating the hype
Every time a new model drops, the marketing team will claim it "crushes benchmarks."
Don't believe them until you see the humanity's last exam leaderboard updates.
We are in an era of "benchmark saturation." If a model gets 98% on a test, that test is dead. It's no longer useful for distinguishing between a good model and a great one. We need tests where the average score is 10%. We need tests that make these billion-dollar machines look a bit "dumb."
That’s how progress happens.
Actionable Insights for the AI-Curious
If you're following the development of these models, whether for business or just pure nerd-level interest, here is how you should interpret the scores moving forward:
- Ignore the "MMLU" scores. They are almost entirely contaminated at this point. They are the "participation trophies" of the AI world.
- Look for Reasoning-Trace models. Models like OpenAI's o1 or deep-thinking variants are the only ones that stand a chance on HLE. If a model doesn't have an internal "chain of thought," it's likely just guessing on high-level exams.
- Watch the "Subject Breakdown." Pay attention to where models fail on the leaderboard. If a model is great at math but fails at bio-engineering, that tells you more about its utility than an aggregate score.
- Expect a plateau. We might see scores on humanity's last exam leaderboard move very slowly. This isn't a bad thing. It means the test is working. It’s a real challenge, not a marketing gimmick.
- Contribute if you can. If you are a world-class expert in a niche field, keep an eye on CAIS or Scale AI. The "exam" needs to evolve to stay ahead of the models.
The leaderboard isn't just a list of names and numbers. It’s a map of the current limits of human-made intelligence. By pushing those limits, we find out what's actually possible—and what remains uniquely human.
Keep an eye on those low scores. They are the most honest thing in AI right now.
Next Steps for Implementation:
To get the most out of tracking these developments, bookmark the official Center for AI Safety research page and the Scale AI HLE landing site. When evaluating a new AI tool for your business or research, specifically ask the provider if they have tested against the HLE dataset or similar "private" holdout sets. This prevents you from investing in a model that has simply memorized its way to a high score. Focus your testing on "out-of-distribution" prompts—questions that require the model to apply a concept to a completely new, fictional scenario—to mirror the rigors of the HLE in your own workflows.