Six rules of thumb for determining sample size and statistical power

This short guide from the Poverty Action Lab presents six rules very clearly, with helpful diagrams to explain or reinforce the points.  Four of them are relevant for planning quantitative data collection even if you’re not using a control group. 

This resource and the following information was contributed by Patricia Rogers.

Authors and their affiliation

Poverty Action Lab

Year of publication

2018

Type of resource

  • Guide

Key features

The guide presents six ‘rules of thumb’ that affect sample size and statistical power. Two of them relate specifically to random assignment (whether the treatment group and control group are the same size, and whether randomization is done at the level of individuals or larger units such as schools or villages) but the other four apply more generally to sampling in order to produce quantitative estimates. 

  • Rule of Thumb #1: A larger sample increases the statistical power of the evaluation.
  • Rule of Thumb #2: If the effect size of a program is small, the evaluation needs a larger sample to achieve  a given level of power.
  • Rule of Thumb #3: An evaluation of a program with low take-up needs a larger sample.
  • Rule of Thumb #4: If the underlying population has high variation in outcomes, the evaluation needs a larger sample.
  • Rule of Thumb #5: For a given sample size, power is maximized when the sample is equally split between the treatment and control group.
  • Rule of Thumb #6: For a given sample size, randomizing at the cluster level as opposed to the individual level reduces the power of the evaluation. The more similar the outcomes of individuals within clusters are, the larger the sample needs to be.

The rules are illustrated with useful diagrams to explain or reinforce the points.

Who is this resource useful for?

  • Commissioners and managers of evaluation
  • Evaluation users
  • Evaluators
  • Those involved in evaluation capacity strengthening

How have you used or intend on using this resource?

I could imagine using this as part of the training materials for a skills development class in quantitative data collection and analysis.  While the concepts would probably be included in a statistics text book, the format of this guide is likely to be clearer.
The material could also be used to work with an evaluation advisory group or steering group to help them understand design choices and tradeoffs.

Why would you recommend it to other people?

It presents some important  ideas very clearly in terms of the clear terminology, the diagrams and the uncluttered design.

Source

The Abdul Latif Jameel Poverty Action Lab (J-PAL) (2018). Six rules of thumb for determining sample size and statistical power. Retrieved from: https://www.povertyactionlab.org/sites/default/files/resources/2018.03.2...

0
No votes yet
Rate this Resource:
This resource is useful for:
A special thanks to this page's contributors
Resource Suggested By
Director of BetterEvaluation/ Professor of Public Sector Evaluation, Australia and New Zealand School of Government.
Melbourne.

Comments

Bismark's picture
Bismark Fiador
Rating: 
0

This material is very useful and self explanatory

Alice Macfarlan's picture
Alice Macfarlan
Rating: 
0

Thanks, Bismark. It's great to hear you found the resource useful!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.