The Evidence Problem
Everyone has evidence for their position. Every claim has studies supporting it. Every side has experts on their team.
If evidence is everywhere, and it points in all directions, how do you figure out what's actually true?
The answer isn't to trust no one. It's to develop better filters — ways to assess which evidence is more reliable, which sources are more credible, and which claims deserve more scrutiny.
Types of Evidence
Not all evidence is equal. Here's a rough hierarchy, from weaker to stronger:
Anecdotes
What it is: Personal stories and individual examples.
Strengths: Vivid, relatable, easy to understand.
Weaknesses: Doesn't prove generalizations. Vulnerable to cherry-picking. Memorable outliers distort perception of typical cases.
Use cautiously when: The question is about general patterns. One story doesn't prove a trend.
Expert Opinion
What it is: Claims by people with relevant expertise.
Strengths: Experts have processed far more information than you could. Their judgment incorporates years of experience.
Weaknesses: Experts can be wrong. They can have conflicts of interest. They can speak outside their expertise. Expert consensus can be mistaken.
Evaluate by asking:
- Is this person actually expert in this specific area?
- What do other experts say?
- Are there incentives that might bias their judgment?
Observational Studies
What it is: Research observing correlations in existing data.
Strengths: Can study things you can't ethically experiment on. Uses real-world data.
Weaknesses: Correlation isn't causation. Confounding variables abound. Observational studies are often contradicted by later experimental studies.
Classic trap: "People who do X have better outcome Y." Maybe X causes Y. Or maybe people who do X differ in other ways that explain Y.
Experimental Studies (RCTs)
What it is: Randomly assigning people to conditions and measuring differences.
Strengths: Randomization controls for confounding variables. Establishes causation, not just correlation.
Weaknesses: Can't always randomize (ethical/practical limits). Sample may not generalize. Studies can be poorly designed, manipulated, or p-hacked.
Evaluate by asking:
- Was the sample large enough?
- Was it representative of the relevant population?
- Was the study pre-registered?
- Has it been replicated?
Meta-Analyses and Systematic Reviews
What it is: Analyses combining multiple studies on the same question.
Strengths: Averages out individual study quirks. Larger effective sample size. Reveals overall pattern.
Weaknesses: Garbage in, garbage out — combining bad studies gives bad results. Publication bias affects what gets included. Can be done poorly.
Replicated Experimental Findings
What it is: Findings that have been independently reproduced multiple times.
Strengths: Much more reliable than single studies. False positives don't replicate.
Weaknesses: Takes time for replication to happen. Some fields have poor replication rates.
Key insight: A single study, no matter how impressive, is weak evidence. Replicated findings across multiple independent teams are much stronger.
Evaluating Sources
SIFT Method
Stop: Pause before sharing or believing. Emotional reactions aren't evidence of truth.
Investigate the source: Who's behind this? What's their agenda? What's their track record?
Find better coverage: What do other sources say? Is there consensus or controversy?
Trace claims: Go to the original source. Headlines distort. Summaries mislead.
Source Credibility Questions
Who created this?
- Individual or organization?
- What's their expertise?
- What's their track record?
What's their incentive?
- Who funds them?
- What do they gain from this being believed?
- Do they have conflicts of interest?
What do they cite?
- Are claims linked to sources?
- Are those sources reliable?
- Can you trace back to original evidence?
What do others say about them?
- Do other credible sources cite them?
- What do fact-checkers say?
- Are there substantive criticisms?
Red Flags
No sources cited: Claims without evidence deserve no evidence in return.
Emotional manipulation: Heavy on outrage, fear, or sentiment. Light on facts.
One-sided presentation: Ignores or strawmans counterarguments.
Extraordinary claims with ordinary evidence: Big claims need big evidence.
Conflicts of interest: Funded by parties who benefit from the conclusion.
Not peer-reviewed (for scientific claims): Anyone can post anything. Peer review isn't perfect but filters out obvious problems.
Anonymous or unclear authorship: No accountability for errors.
Website quality signals: Excessive ads, clickbait, unprofessional design. Correlation with unreliability.
Common Misleading Tactics
Cherry-Picking
What it is: Selecting only evidence that supports your position.
How to spot it: Ask what evidence was left out. Look for systematic reviews, not individual studies selected to support a narrative.
Example: Citing three studies showing X works while ignoring twenty showing it doesn't.
Misleading Statistics
Context stripping: "Deaths increased 100%!" (from 2 to 4, in a population of millions)
Relative vs. absolute: "Risk doubles!" (from 0.001% to 0.002% — trivial in absolute terms)
Graph manipulation: Truncated axes make small changes look large.
Survivorship bias: Only looking at successes while ignoring failures.
Correlation/causation confusion: Implying X causes Y when they're just correlated.
False Balance
What it is: Presenting fringe views as equally valid as consensus views.
How it works: "Some say the earth is round, others say it's flat. We present both sides." This creates fake debate where none exists.
How to spot it: Check whether the "controversy" exists among experts or just among non-experts.
Appeal to Study
What it is: Citing a single study as if it settles the question.
Why it's misleading: Individual studies are often wrong. They need replication. One study ≠ scientific consensus.
Proper response: "What does the body of research say, not just this one study?"
AI for Evidence Evaluation
Claim Analysis
Someone claims: "[Claim]"
Help me evaluate:
1. What evidence would support or refute this?
2. What type of evidence is being offered?
3. What would strong evidence look like vs. weak evidence?
4. What should I look for to verify this?
Source Evaluation
I'm reading something from [source].
Help me assess:
1. What kind of source is this? (news, advocacy, academic, etc.)
2. What are their likely biases or incentives?
3. What's their reputation for accuracy?
4. How should I calibrate my trust level?
Study Evaluation
I found a study claiming [claim].
Here are the key details: [study description]
Help me evaluate:
1. What type of study is this?
2. What are its strengths and limitations?
3. Does the conclusion follow from the methods?
4. What would I need to see to be confident in this finding?
Verifying Specific Claims
Is this claim accurate: "[Claim]"
Walk me through:
1. What do we know about this topic?
2. What would I need to verify this?
3. Are there obvious red flags?
4. What questions should I ask?
Building Evidence Habits
Before Believing
- Check the source
- Look for original sources
- Find multiple perspectives
- Assess evidence quality
- Consider what could change your mind
Before Sharing
- Did you read past the headline?
- Have you verified the core claim?
- Do you know the source's reputation?
- Would you bet money this is true?
When You Find Contradicting Evidence
- Don't dismiss it automatically
- Assess quality of each source
- Look for reconciling explanations
- Update proportionally to evidence strength
The Epistemic Humility Principle
Strong thinkers are appropriately uncertain. They:
- Distinguish "I believe X" from "X is certain"
- Hold beliefs with varying confidence levels
- Update when evidence warrants
- Admit when they don't know
- Distrust their own certainty
The goal isn't knowing everything. It's knowing what you know, knowing what you don't, and calibrating confidence accordingly.
What's Next
Even with strong evidence skills, you might misunderstand positions you disagree with.
Chapter 5 covers steelmanning — building the strongest version of opposing arguments. This is how you actually understand positions rather than fighting caricatures.