An upcoming paper in Cyberpsychology, Behavior and Social Networking examines participant drop out rates in online surveys. In the study, 1963 undergraduates completed one of six online surveys through SurveyMonkey.
The six surveys varied a bit. Two were personality inventories combined with measures of anticipations related to an upcoming Valentine’s Day. One was primarily an examination of the construct validity of various impulsivity scales. Two were related to an examination of emotionally evocative images. The last examined personality differences related to birthdays.
From this data, the author developed a regression equation to predict dropout rates: y’ = .10 + .02x, where x represents the number of items on your survey in hundreds. From this, I created this handy chart:
The formula is interesting in its own right. The author experienced an average of 10% participant drop-out after the consent form averaged across studies, with a 2% increase per 100 items. The authors interpreted this to mean that researchers should be more careful when conducting in-person studies to give more opportunity to stop participation.
Of practical importance is the fact that this study gives researchers some guidance on how many participants they will really need to recruit in order to get a desired power level in online survey research. For example, if you needed 200 participants for adequate power, and your survey had 300 items, it’s estimated that you would actually need to recruit 239 to meet your desired numbers due to dropout. Even so, it’s important to remember that this is not a cure-all; researchers should still critically evaluate why participants are dropping out. If participants are not missing at random, the validity of research conclusions may be threatened.
It should be noted that there was some variability in response rates between the surveys, and it’s unclear what the precise cause of this might have been. For example, the participation rate in the first study by the end of the 243-item survey was still around 90%, while the participation rate in the sixth study at the end of 152-item survey was around 70%. Thus, there are likely other study-level characteristics influencing participation rate – hopefully future research in this area will provide additional clarity.Footnotes: