.png)
🤨 How many of you feel constant pressure to justify every product decision with hard data?
Metrics Pressure
Innovation Paralysis
Finding Balance
Core Message:
Data is a vital tool, but not a panacea. Let's explore looking beyond the numbers.
What We'll Cover
My Evolving View on Data
Personal story from intuition to data-driven and beyond
Hidden Pitfalls
The dangers of 'Data-Driven Everything'
Finding Balance
Intuition, Experience & Data together
Validation Through Action
Building to learn + Real example
AI & Evaluation
Looking ahead to quality over metrics
My Evolving View on Data
Smartisan Technology
A startup focused on mobile and IoT devices acquired by ByteDance
My product journey started back in 2015 at Smartisan Technology, where intuition drove our product decisions.
Focus:
- Design
- User Experience
- Founder Vision
Data Usage:
Minimal. Decisions driven by intuition.
Award-Winning Hardware and Software Design
We created unique hardware – winning the first iF Gold Award for a mainland Chinese smartphone – and incredibly thoughtful software features.
Handshaker
I worked on products like Handshaker, a file manager for Android on macOS. Built on our assumption of user need, not hard data.
Result:
Despite minimal data justification, it received great feedback from users and media.
Culture:
- Systematic
- Process-heavy
- Intensely data-focused
Focus:
Systematic, Process-Heavy, Intensely Data-Focused
My Experience Journey:
Embraced It
Grew Frustrated
Questioned
Key Challenges:
Quantifying Individual Contribution:
How do you measure a PM's impact vs. Sales on B2B? How do you quantify long-term strategy?
Isolating Variables:
Was it your feature or seasonality/other factors? (Correlation ≠Causation)
Evaluating the Unimplemented:
Predicting impact of innovation is often guesswork.
Consequences:
- Focus on easy metrics (clicks) over value (satisfaction)
- Analysis Paralysis / Resource Drain
My Core Belief Now
- Not all requirements can be effectively measured by data.
- Not all valuable actions link directly to easily quantifiable benefits.
- Relentless data focus may be more about justification than great products.
"Tactical diligence masking strategic laziness."
Finding the Balance: Intuition + Data
The Limits of Data (It's Not Just Me)
Andrew Chen (Andreessen Horowitz)
From 'Why data-driven product decisions are hard (sometimes impossible)'
Past vs. Future
Data measures yesterday, product builds for tomorrow.
Speed of Decision
Intuition = Fast; A/B Tests = Slow.
Majority Rules
Tyranny of the Majority (Kills niche/innovation).
Causation Issues
Correlation ≠Causation.
Intangibles
Hard to Measure: Brand, delight, trust.
Analysis Paralysis
Cost/time > benefit.
The Bottom Line:
Data doesn't tell the whole story, and sometimes it actively misleads us.
Beyond the Dashboard
Real-World Complexity vs. Quantification
Quantification is vital, but it can't be everything. Why? Because the real world, and real user experiences, are far too complex to be perfectly captured by numbers alone. If pure quantification worked, business would be predictable, and AI could do our jobs entirely. Is that the reality we see?
Market unpredictability
Customer emotions
Innovation leaps
These human factors defy perfect quantification
Karri Saarinen, founder of Linear
From Twitter"As companies scale... they start flying with the dashboard instead of flying by looking out of the window because the former is easier to do in scale... Measurement like number of doors installed or times the door is used never gives you anything about how good it is... It's almost as if you have to stop thinking for a minute, and just focus on the experience."
Focus on the holistic experience, the "quality without a name."
Finding the Balance: It's Not Either/Or
Is data useless? Absolutely not. It's about how we use it.
"Distinguish between Data-driven decisions (you can acquire, study, and debate facts and numbers... fairly confident in your choice... relatively easy to make and defend) and Opinion-driven decisions (you have to follow your gut and your vision... without sufficient data... always hard and always questioned)."
Fadell warns against trying to turn opinion-driven problems (like overall product vision or market direction) into data-driven ones. You'll drown in inconclusive data – analysis paralysis.
Use Data-Driven For:
(Execution-Focused)
- Optimization (A/B tests, funnels)
- Monitoring known flows
- Technical performance
Use Opinion-Driven For:
(Strategy-Focused)
- New product direction / Vision
- Strategic planning
- Complex / long-term impact features
The Key to Success:
Knowing which type of decision you're facing and using the appropriate tools.
Use data as input, not the decision-maker
Especially for vision, innovation, and long-term decisions
A Decision Framework
Validation Through Action
How do we really know if ideas are good?
We often seek certainty before we build.
Jason Fried of 37signals
From Twitter"There's really only one real way to get as close to certain as possible. That's to build the actual thing and make it actually available... Real usage on real things... is the only way to validate anything... you don't know, you won't know, you'll never know until you know and reflect back on something real... The best way to find out, is to believe in it, make it, and put it out there."
"The most effective way to validate ideas is often to build and launch, not just plan endlessly."
This aligns with Andrew Chen's advice, especially for startups or new initiatives:
Prioritize intuition over waiting for perfect data
Make fast decisions
Favor big moves over tiny increments
Why? Because in uncertain territory, speed of learning through action often beats slow, data-heavy deliberation.
We are in the golden age of build.
Even if you don't know how to code, these tools make it possible to build your idea!
Example: Learning by Doing
My recent experiment: "File Visualizer"
It turns documents into beautiful interactive websites, like the one hosting this presentation.
https://file-visualization.vercel.app/- Built prototype in ~1 week ("vibe coding")
- Launched quickly
- Result: Immediate, real-world usage & feedback
Faster learning loop than months of pre-analysis.
File Visualizer
Looking Ahead: AI & Evaluation
AI raises deeper questions about value and evaluation.
If AI can reliably evaluate based purely on quantifiable data...
...is the job/task itself at risk of automation?
The harder a role/contribution is to evaluate with simple metrics...
...the more it likely relies on uniquely human skills (judgment, creativity, complex problem-solving).
Measuring AI's True Impact
Ethan Mollick (Professor at the Wharton School)
From One Useful Thing"The traditional metrics of time saved or costs reduced may miss the more transformative impacts of these systems - their ability to generate novel insights, synthesize complex information, and enable new forms of problem-solving. Moving too quickly to concrete KPIs, and leaving behind exploration, will blind companies to what is possible. Worse, they encourage companies to think of AI as a replacement for human labor, rather than exploring ways in which human work can be boosted by AI."
Look beyond simple efficiency. Focus on augmentation.
Meaning in the Age of AI
Dario Amodei (CEO of Anthropic)
From Machines of Loving Grace"In any case I think meaning comes mostly from human relationships and connection, not from economic labor. People do want a sense of accomplishment, even a sense of competition, and in a post-AI world it will be perfectly possible to spend years attempting some very difficult task with a complex strategy, similar to what people do today when they embark on research projects, try to become Hollywood actors, or found companies."
AI handles
routine tasks
Humans focus on
creative challenges
Connection &
complex human work
Key Takeaways
Core Message: Data is an important tool, but not a panacea.
Acknowledge Data's Limits
Biases, blind spots, correlation ≠causation.
Value Intuition & Experience
Especially for strategy.
Prioritize User Experience
Look beyond metrics.
Use a Balanced Approach
Context matters.
Validate Through Action
Build, launch, iterate, learn.
Look Beyond Efficiency with AI
Focus on augmentation.
Thank You!
The Goal
Build Things That Matter
Create products that resonate with real human needs and experiences
Maintain Balance
Use data wisely as a tool, but never forget the human experience behind the numbers
Connect with me
Joe (Beiqiao) Hu
Product People. It's time to build!