Sample Size and Power

I get this question all the time: “How many survey respondents do we need?”

The short answer: as many as resources allow.

Here’s the long answer:

There are two aspects that play into how big your sample size should be: statistical significance and representativeness. Today, let’s talk about statistical significance.

For example, let’s find out if American bears and kids like the same foods. We collected data from 10 bears and 10 kids and found the following:

Okay, so these results are little off-kilter, but bear with me. For now, let’s just focus on the numbers: It looks like there is a
difference in their food preferences, but there just aren’t enough responses to begin to test if this difference is significant.

Statistical significance is really about power, that is, the ability to detect meaningful relationships between groups or variables. The two things that contribute to power are the extent to which the relationship exists (it looks like bears and kids might like different foods) and the size of the sample. While we can’t really control the size of the relationship, we do have some control over the size of the sample. Usually, a bigger sample size means more power: you’re more likely to find the relationships that exist between variables with more responses. In this example, there isn’t enough power to find this difference in a statistical significance test.

Some tests of statistical significance are naturally more powerful than others. Jacob Cohen, a statistician well-known for his work in power calculations and estimations, has already produced a lot of documentation on how many respondents you need to achieve medium power at the 95% statistically significant level (the level commonly used in statistics) for some common statistical analyses:

• Independent samples t-test (two groups) – 64 people in each group (128 total, the groups don’t have to be exactly equal but the closer they are in size, the better)
• Correlation – 85 people

In most cases, around 100 respondents will be enough to find statistically significant relationships. Even if you have fewer than this, you may still be able to find those relationships if they are considerable in size. In our example, if we had 100 bears and 100 kids with the results below, we’ll be much more likely to detect the significant difference in their food preferences:

So, it really all comes down to this: the more responses you can get, the more likely you are to find statistical significance. But going back to our survey on food preferences of kids and bears, there’s something else that’s fishy about our results. It seems like they came from a very limited group of respondents…

… For more on what that means, stay tuned. In our next post, we’ll talk about the relationship between sample size and representativeness.