Predicting Dropout Rates for Students Completing Online Surveys

An upcoming paper in Cyberpsychology, Behavior and Social Networking examines participant drop out rates in online surveys1.  In the study, 1963 undergraduates completed one of six online surveys through SurveyMonkey.

The six surveys varied a bit.  Two were personality inventories combined with measures of anticipations related to an upcoming Valentine’s Day.  One was primarily an examination of the construct validity of various impulsivity scales.  Two were related to an examination of emotionally evocative images.  The last examined personality differences related to birthdays.

From this data, the author developed a regression equation to predict dropout rates: y’ = .10 + .02x, where x represents the number of items on your survey in hundreds.  From this, I created this handy chart:

I only extended this chart out to 550 because the maximum number of items examined empirically was 535, in the second emotionally-evocative-images study.  Predicting beyond that point may be unwise.

The formula is interesting in its own right.  The author experienced an average of 10% participant drop-out after the consent form averaged across studies, with a 2% increase per 100 items.  The authors interpreted this to mean that researchers should be more careful when conducting in-person studies to give more opportunity to stop participation.

Of practical importance is the fact that this study gives researchers some guidance on how many participants they will really need to recruit in order to get a desired power level in online survey research.  For example, if you needed 200 participants for adequate power, and your survey had 300 items, it’s estimated that you would actually need to recruit 239 to meet your desired numbers due to dropout.  Even so, it’s important to remember that this is not a cure-all; researchers should still critically evaluate why participants are dropping out.  If participants are not missing at random, the validity of research conclusions may be threatened.

It should be noted that there was some variability in response rates between the surveys, and it’s unclear what the precise cause of this might have been.  For example, the participation rate in the first study by the end of the 243-item survey was still around 90%, while the participation rate in the sixth study at the end of 152-item survey was around 70%.  Thus, there are likely other study-level characteristics influencing participation rate – hopefully future research in this area will provide additional clarity.

1. Hoerger, M. (in press). Participant dropout as a function of survey length in Internet-mediated university studies: Implications for study design and voluntary participation in psychological research. Cyberpsychology, Behavior and Social Networking : 10.1089/cyber.2009.0445 [↩]