Big data is actually pretty dumb. That sounds like heresy in an era where everyone is obsessed with LLMs and neural networks, but it’s the cold, hard truth that Judea Pearl drives home in The Book of Why. Most of our current AI is just fancy curve-fitting. It looks at two things happening at the same time and assumes they might be related, but it doesn't actually understand why.
If you’ve ever felt like "correlation does not imply causation" was a bit of a conversational dead end, you aren't alone. For about a century, the world of statistics basically banned the word "because." Scientists were told to look at the numbers and shut up about the reasons. Pearl, a Turing Award winner and a giant in the world of artificial intelligence, decided that was a huge mistake. He argues that we’ve essentially hobbled our own progress by refusing to give machines—and ourselves—a formal language for causality.
The Book of Why isn't just some dry textbook. It’s a bit of a manifesto. It tracks how we went from ancient superstitions to the rigid "correlation only" era of Pearson and Fisher, and finally to what Pearl calls the Causal Revolution. It's about how we can finally answer questions like, "Would my headache be gone if I hadn't taken that aspirin?"
The Ladder of Causation: Why Your AI is Still a Baby
Pearl breaks down the ability to reason into three distinct levels. He calls this the Ladder of Causation.
The first rung is Association. This is where most AI lives. It’s about seeing. If I see the barometer falling, it’s likely to rain. The machine sees a pattern and makes a prediction. But the machine doesn't know that the barometer doesn't cause the rain. If you sneak up and smash the barometer, it’s not going to suddenly get sunny outside. Most "big data" projects are stuck right here on the first rung. They are great at finding patterns but hopeless at understanding the mechanics of the world.
The second rung is Intervention. This is about doing. If I take this medicine, will my blood pressure go down? This is different from just seeing. You are actively changing the system. This is what Randomized Controlled Trials (RCTs) try to measure. You aren't just observing people who happen to take aspirin; you are forcing a group to take it and seeing what happens.
Then there’s the third rung, the "Holy Grail": Counterfactuals. This is the level of imagining. It involves asking "What if?" Suppose I took the aspirin and my headache went away. A counterfactual question asks: "Would my headache have gone away if I hadn't taken the aspirin?" This is how humans think. It’s the basis of our morality and our legal system. You can’t blame someone for a car accident if the accident would have happened even if they hadn't been speeding. Current AI cannot do this. It has no model of the world to "run" a different version of reality.
💡 You might also like: Starliner and Beyond: What Really Happens When Astronauts Get Trapped in Space
The Men Who Banned "Why"
One of the most fascinating parts of The Book of Why is the history of the "Causal Prohibition." Pearl points the finger at some heavy hitters in the world of statistics, specifically Karl Pearson and Sir Ronald Fisher.
Pearson was obsessed with measurements. He believed that everything we need to know is in the correlation coefficient. To him, saying "A causes B" was just a sloppy way of saying A and B are highly correlated. He wanted to strip away the "metaphysics" of causation.
Fisher took it a step further. He’s the guy who gave us the Randomized Controlled Trial. Fisher’s view was that if you want to know if something works, you have to run a randomized experiment. If you can’t run an experiment, you can’t say anything about causation. Period. This created a massive problem for things like smoking and lung cancer. You can’t ethically force a group of people to smoke for 30 years just to see if they die. Because of the "Fisherian" dogma, scientists spent decades hesitating to say that smoking causes cancer, even as the bodies piled up. They just didn't have the mathematical tools to prove it without a randomized trial.
Pearl’s work provides those tools. He shows how we can use "Causal Diagrams" (or Directed Acyclic Graphs) to extract causal information from observational data. It’s a way to "control" for variables after the fact, using math instead of a laboratory.
Do-Calculus: The Math of Common Sense
How do you actually tell a computer how to think about causes? Pearl invented something called do-calculus.
In standard probability, you have $P(Y|X)$, which is the probability of $Y$ given that we see $X$.
Pearl introduces $P(Y|do(X))$, which is the probability of $Y$ if we intervene and force $X$ to happen.
📖 Related: 1 light year in days: Why our cosmic yardstick is so weirdly massive
These are not the same thing.
Think about the "Ice Cream and Drowning" example. We see a high correlation between ice cream sales ($X$) and drowning deaths ($Y$).
So $P(Y|X)$ is high.
But if we use do-calculus, $P(Y|do(X))$ is low. If we force people to eat ice cream in the middle of winter, it doesn't make them drown. The "do" operator allows us to mathematically strip away "confounders"—in this case, the hot weather that causes both ice cream eating and swimming.
By drawing a simple map of arrows—what Pearl calls a causal diagram—we can see which variables are "backdoor paths" that mess up our data. Honestly, once you see these diagrams, you can't unsee them. You start realizing how many news headlines are just confusing correlation with causation because they haven't accounted for a simple "collider" or "confounder" in their logic.
The Smoking Debate and the Birth of Modern Causal Inference
The struggle to prove smoking caused cancer is a central pillar of The Book of Why. For years, the tobacco lobby used the "correlation is not causation" mantra as a shield. They even had famous statisticians on their side.
They argued there might be a "smoking gene" that made people both want to smoke and also made them more susceptible to lung cancer. Without a randomized trial, how could you prove them wrong?
Pearl shows how researchers like Jerome Cornfield used "sensitivity analysis" to debunk this. They proved that if a smoking gene existed, it would have to be so incredibly powerful—so perfectly correlated with both smoking and cancer—that it was biologically impossible. Later, researchers like Judea Pearl himself and Donald Rubin developed frameworks to handle this kind of data. This wasn't just an academic exercise; it saved millions of lives.
👉 See also: MP4 to MOV: Why Your Mac Still Craves This Format Change
Why This Matters for the Future of AI
We are currently in a "Generative AI" boom. Models like GPT-4 are mind-blowing. But Pearl is somewhat skeptical that they will ever reach true human-level intelligence without a "World Model."
Current LLMs are basically the ultimate rung-one machines. They have read the entire internet and can predict the next most likely word with startling accuracy. But they don't understand the underlying physical or social laws that make those words true.
If you ask an AI how to fix a broken economy, it will give you a beautiful essay based on what economists have written. But it can’t truly simulate a "counterfactual" world where a specific policy is changed unless it has a causal model of how money, labor, and resources actually interact. Without the ability to reason about "why," AI remains a very sophisticated parrot.
Pearl’s vision for the future is a "Causal AI" that can explain its decisions, imagine new scenarios, and understand the consequences of its actions. It’s the difference between a self-driving car that knows "stop signs mean stop" and one that understands "if I don't stop, I will hit that pedestrian because of the laws of momentum."
Actionable Insights: Thinking in Causes
You don't need to be a mathematician to use the lessons from The Book of Why. Here is how you can apply causal thinking to your own life or business:
- Draw the Diagram: Next time you're looking at a problem—like why sales are down or why you're feeling tired—literally draw it out. Put your variables in circles and draw arrows for what causes what. This forces you to identify "confounders" (third factors affecting both) and "mediators" (the mechanism by which A causes B).
- Challenge the "Data": When someone shows you a chart showing a "clear link" between two things, ask yourself: "Is this $P(Y|X)$ or $P(Y|do(X))$?" Would changing $X$ actually change $Y$, or are they both just riding the wave of a third variable?
- Think in Counterfactuals: To evaluate a decision, don't just look at the outcome. Ask: "What would have happened if I had done the opposite?" This helps you distinguish between skill and luck. If the outcome would have been the same regardless of your action, you didn't "cause" the success; you just happened to be there when it happened.
- Look for Colliders: Be careful of "selection bias." For example, if you notice that all the "attractive" people you date are "mean," it might not be that beauty causes meanness. It might be a collider: you only date people who are either very attractive or very nice. If they aren't nice, they must be attractive for you to date them. This creates a fake correlation in your specific "data set" of exes.
Judea Pearl’s work is a reminder that data is just a shadow of reality. To understand the world, we have to look past the shadows and see the objects casting them. We have to be brave enough to ask "Why?" even when the statisticians tell us not to. It’s the only way we’ll ever build machines that actually think—and it’s the only way we can truly understand the complexity of our own lives.
Get a copy of the book. It’s dense in places, but it’ll change the way you look at every "study" or "statistic" you see in the news forever. It’s basically a software update for your brain.
Next Steps for Implementation
- Identify a Correlation: Pick one "metric" you track (e.g., social media engagement and sales).
- Hypothesize the Mechanism: Write down exactly how $A$ is supposed to lead to $B$. If you can't explain the "how," you're likely looking at a confounder.
- Run a Small Intervention: Instead of just observing, change one variable ($do(X)$) in a small, controlled way to see if the effect holds up.
- Audit for Selection Bias: Check if your "data" is limited by a "collider"—did you only look at successful cases? Did you ignore the "dogs that didn't bark"?