Technical Difficulties Decrease Learning, Motivation in Training

DISCLOSURE: One of the authors on this paper is a doctoral candidate that was previously in one of my classes.

In a cleverly-designed study on the experience of technical difficulties in online training, Sitzmann, Ely, Bell and Bauer (2010)1 made free Microsoft Excel training available online through “community sites” – this likely means Craigslist, Backpage, and other similar classified ad websites.  530 adults ultimately signed up for the study.

After people signed up, they began a four-hour four-module course – but with an unknown catch.  Participants were randomly assigned to one of several conditions crossing planted technical difficulties (i.e. participants would encounter errors that made it appear as if the website was poorly designed) in a random number and variety of modules.  This isn’t described clearly, but I suspect it means there were fifteen conditions: no errors, errors in 1, 2, 3, 4, 1 and 2, 1 and 3, 1 and 4, 2 and 3, 2 and 4, 3 and 4, 1 2 and 3, 1 2 and 4, 1 3 and 4, and 1 2 3 and 4.  Using hierarchical linear modeling (HLM), Sitzmann et al. examined the impact of technical difficulties on several outcomes:

• Attrition was 8% higher in the first module when technical difficulties were present, but there was no (or a much smaller) effect in later modules (a rise from 46% to 54%).
• Learning was lower in modules where trainees encountered technical difficulties, though the effect was quite small (a 3% decrease in test scores).
• People who reported greater motivation to complete the training were more likely to learn from it (though the effect was again fairly small), but there was no interaction between motivation and technical difficulties, i.e. regardless of their motivation to complete the training, technical difficulties were equally demotivating.
• People who reported greater motivation to complete the training were more likely to persevere (the effect was a bit larger this time), and there was in fact an interaction between motivation and technical difficulties, i.e. people with higher motivation to complete the training were more likely to persevere in the face of technical difficulties.

The unusual sample for this study is both a strength and a weakness.  The authors capture “real” trainees (people with a reason to complete the training – in this case, their own initiative), which is probably more generalizable than college students.  But it also makes it more difficult to explain what did happen.  The attrition rate in Module 1, even for trainees not experiencing technical difficulties, was 46%.  There are many possible explanations for this: 1) perhaps they did not judge the training to be professional enough, 2) perhaps they became bored, 3) perhaps they didn’t like the material, 4) perhaps they decided the material was not appropriate for them, 5) perhaps they decided the material was too simple.  Each of these would have different implications for the relationships they found and might implying missingness not at random.

As far as practical implications for online training, this suggests that every effort should be made to avoid technical difficulties, else you might demotivate your employees from completing the training (or perhaps worse, motivate them to complete the training but put in less learning effort than they otherwise would have).

1. Sitzmann, T., Ely, K., Bell, B., & Bauer, K. (2010). The effects of technical difficulties on learning and attrition during online training. Journal of Experimental Psychology: Applied, 16 (3), 281-292 DOI: 10.1037/a0019968 [↩]