When to Ignore the Research (Yes, Really)

I used to treat research like a referee’s call. If a study said something, that settled it. Then I watched a team spend three months rebuilding an onboarding flow because “a study found users prefer fewer steps.” The study had 38 participants, all of them college students, and the product was accounting software for small-business owners. Completion rates dropped. The research wasn’t fake or sloppy, but it wasn’t about them. Here’s how I decide now whether a finding deserves a place in my decisions or a polite goodbye.

Small samples are fine until they aren’t

Small studies get a bad rap, and often unfairly. Ten user interviews can show you a usability problem that matters, because if five of ten people get stuck on the same button, you don’t need a statistician. Qualitative research is built to find problems, not measure how common they are.

The trouble starts when a small sample is asked to do a big job. Watch for these:

A percentage built on a tiny base. “Sixty percent of respondents preferred option B” sounds decisive. If there were ten respondents, that’s six people. Swap two of them, and you get a different headline. When you see a precise number, ask what it’s a percentage of.

A small sample claiming to predict behavior. Ten people can tell you why something confuses them. They can’t tell you how many of your thousands of customers will click, buy, or cancel.

A tiny effect found in a small group. Small studies tend to overstate effects, and the ones that get published and shared are the ones with dramatic results. If a 25-person study reports that a headline change boosted engagement by 40%, be suspicious. Real effects are usually smaller and messier.

Representative matters more than big

A huge sample of the wrong people is worse than a small sample of the right ones. Ten thousand responses from a survey about workplace tools don’t help you if respondents were mostly tech employees at large companies and your customers are plumbers and dental office managers.

Before you lean on a study, check who was in it:

• Who were they? Age, job, income, region, experience level. Not every difference matters, but some matter a lot.
• How were they found? People who answer surveys or join studies are different from people who don’t. Volunteers tend to be more engaged, more opinionated, and more online.
• What were they doing? Saying what they’d do in a survey is a long way from doing it with their own money and a busy afternoon.

A good gut check is to picture one of your real customers reading the study’s participant description. Would they see themselves in it? If not, treat the findings as an interesting hypothesis rather than a conclusion.

A quick test I use

When someone sends me research and says, “we should act on this,” I ask three questions.

1. Would I expect this to be true for my audience, regardless of the study? If the finding fits what I already see in my own data and conversations, it’s a useful confirmation. If it contradicts everything I know, the burden of proof is high.
2. What would it cost to be wrong? Changing a button color is cheap to undo. Overhauling pricing or cutting a feature isn’t. The bigger the bet, the more evidence I want, and the more it needs to come from people like mine.
3. Can I test it myself? Often the best use of outside research is as a prompt for a small experiment of your own. Run it with your audience and let your data settle it.

Ignoring research isn’t ignoring evidence

This is the part people get wrong. Setting aside a flimsy or mismatched study doesn’t mean going with your gut. It means giving more weight to better evidence: your own analytics, customer conversations, sales calls, support tickets, and the experiments you can run this week.

Your own audience’s behavior will almost always beat a stranger’s study about someone else’s audience. A study of 40 students can’t outvote 4,000 of your actual customers.

What to say instead of “that’s not valid”

If you’re the person pushing back in a meeting, “I don’t trust that study” lands badly. Try something like: “This is interesting. Who was in it? I’d want to see if it holds for our customers before we change anything.” You’re not dismissing the work, just asking whether it applies.

Most of the time, people are relieved. Nobody loves being the one who bet the roadmap on a 12-person survey.

Take research seriously but take it seriously too. Ignore or heavily discount a study when the sample is too small to support the claim it’s making, when the people in it don’t look like your audience, when the effect sounds too good to be true, or when being wrong would be expensive and you can test it yourself instead.

And when you do set a study aside, say why. That habit alone will make you a better decision-maker than most people who just cite the latest headline.