Low-cost convenience samples are surprisingly good for estimating association in tobacco studies
August 14, 2018
A team from the UNC Gillings School of Global Public Health found that data collected from low-cost convenience sampling in research experiments was comparable to that collected from state-of-the-art and costly probability sampling. The finding is useful for behavior scientists interested in gathering timely data on a budget.
Dr. Noel Brewer
Dr. Michelle Jeong
Michelle Jeong, PhD, postdoctoral fellow, and Noel Brewer, PhD, professor of health behavior at the Gillings School, co-authored the study, “Similarities and Differences in Tobacco Control Research Findings From Convenience and Probability Samples,” published July 20 in the Annals of Behavioral Medicine. Both researchers are members of the UNC Lineberger Comprehensive Cancer Center.
Probability sampling – collecting samples through methods such as random-digit dialing and addressed-based sampling, which requires a substantial investment in resources and funding – is the gold standard in behavioral science studies. Convenience sampling, such as that offered by the web-based Amazon Mechanical Turk (MTurk) operation, requires considerably less time and money, but scientists have perceived it to be less precise.
“Our paper is timely given the rise in use of online research with convenience samples, which we anticipate will become even more popular,” said Jeong, first author on the paper. “This study verifies the utility of these studies and will prove useful for researchers who need to collect data in a timely manner and cannot wait for the resources needed for probability sampling.”
Researchers examined tobacco behaviors by surveying 5,000 smokers and non-smokers via telephone, as well as 5,000 with an online survey from MTurk. The national telephone survey cost nearly $620,000 (almost one million dollars, when including overhead costs), while the MTurk study cost only around $17,000. Jeong said a national probability survey could take months, while MTurk returns data in as little as a week.
“In the early stages of research, behavioral scientists often have to settle for cheap convenience samples,” said Brewer. “It turns out that these experiments give the same answers as ones that are fancier and far more expensive. Why spend almost a million dollars on a survey when you can get the same answers for a few thousand?”
Jeong said the study’s results show that convenience sampling would be a reliable first step for preliminary or pilot studies for experiments and correlational studies, opening the door to meaningful research experiments from young faculty members with little funding.
“In the end, MTurk is a useful way to get preliminary data for experiments,” said Jeong. “Based on those results, a researcher may either forego the full project or feel confident that they can replicate those findings with a national probability sample to get the whole picture.”
Jeong said that while these findings are significant, it is important to remember that, as these two samples are different, researchers would want to replicate any new findings with a national representative sample and data set.
“This is a good way to test the feasibility of a project, she said. “Because the experimental findings are the same, you can deduce that if you don’t find something in this sample, you aren’t going to find it in a large-scale study. Those results can help you refine what you’re looking for and decide if you’re going in the right direction.”
Contact the Gillings School of Global Public Health communications team at sphcomm@listserv.unc.edu.
