If you need further evidence of the enthusiasm and engagement that Second Life can enable, just observe this labor-of-love: an appropriately-timed Valentine’s music video captured in Second Life, singing a love song between two auditors.
The video was a group effort from various accounting bloggers and the staff of the MACPA. A great amount of time was obvious spent – production values (in Second Life terms) are pretty high! It is full of accounting in-jokes, so don’t be surprised if a few go over your head, but it is definitely worth checking out.
Scientific publishing culture is such that typically only studies with well-thought out theoretical backing and empirical evidence supporting that theory tends to get published. The reason is not necessarily that studies without empirical support are worthless – rather, the problem is that there is limited journal space available. Studies that find what they set out to find are seen as more valuable than those that don’t.
The problem is that all these unpublished studies, despite providing potentially useful information, are just that – unpublished, and difficult to find. Imagine this situation:
- Researcher A uses Technique to address the Problem
- Technique fails to address the Problem
- Researcher A tries to publish results, but they are negative, and cannot do so
- Researcher B uses Technique to address the Problem
And the cycle continues. In the social sciences (like psychology), things aren’t quite so clear cut, since we more typically are trying to explain more subtle processes. But the problem is the same – unpublished studies are useful, but difficult to locate.
This led to me creating filedrawer,org, an online unpublished papers database in psychology. It’s fairly empty, but it fills a need. It essentially allows researchers to make their unpublished papers available to a wider audience. There is no peer review, because it is just an online database. It’s called “filedrawer” as a tribute to the so-called file drawer effect: the problem that unpublished (although still valid) studies usually bias meta-analytic effect estimates away from zero.
In the computational and life sciences, a new journal designed for such data has been created: the Journal of Serendipitous and Unexpected Results. JSUR aims to put such studies through peer review, and if their methods are sound, publish them. In Biology, the Journal of Negative Results in Biomedicine has been around since around 2002, and the Journal of Negative Results has been publishing such data in ecology and evolutionary biology since 2004.
Where are the peer-reviewed journals in the social sciences? Well, as far as I can tell, they don’t exist. I’m not really sure why. I guess filedrawer.org will have to do for now…
I recently got into a discussion with Flemming Nielson on Anja Patulski Nielson’s Weblog about what evidence is available to support the use of virtual worlds for training and education. Flemming argued that a number of high-profile case studies are available (especially from IBM); I argued that there is little scientific support. But I soon realized that all I knew about was the lack of scientific support in my home research area, industrial/organizational psychology, the application of psychology to business principles specifically. What about the wider psychological literature, especially relating to education? What evidence is there that virtual worlds produce learning outcomes as strong as traditional classrooms and instructional methods?
A simple search for [“Second Life” or “virtual world”] produced the following psychological studies relevant to my question. Criteria for inclusion in this list:
- Considers adult learning in some regard (college courses, training and development, includes lab studies)
- Collects empirical data (research volunteers, trainees, or students)
- Is in English (I have to be able to read it)
- Can be retrieved easily (2-3 articles that might have otherwise fit in this list would have taken several days of inter-library loan to retrieve)
- Is indexed by APA’s PsycINFO
- Has been peer-reviewed
- Discusses true 3D virtual worlds (not just massively multiplayer games or multi-user learning systems). It should be noted that there is some confusion in the scholarly literature as to what exactly a virtual world is. Virtual worlds differ primarily from massively multiplayer games in that there are no set objectives in a virtual world – Second Life qualifies, World of Warcraft does not.
- Some effort must have been made to prove virtual worlds as a reasonable alternative or complement to traditional education/training practices. I would have included “unreasonable” as well, except that no study study set out with this objective.
My initial search turned up a couple hundred, but few of those met these qualifications. Every study I could locate in psychology within those guidelines is in the following list:
- De Lucia, Francese, Passero & Tortora (2009)1 examined the reactions of 26 students participating in 3 virtual SL lectures. Unfortunately, evaluation data is never recorded, so we don’t really know how Second Life compared to traditional instruction. They did find that participants had some degree of a “sense of presence,” but that sense is not connected to any real outcomes (like learning).
- Jarmon, Traphagan, Mayrath & Trivedi (2009)2 conducted a qualitative study that I think is typical of this literature; a virtual world was used as part of an assignment for 5 graduate students to examine communications, and all sorts of stories and personal explanations were collected. It’s fantastic that SL worked well in this context; the question that this study cannot answer is “will it work in other contexts?”
- Jamaludin, Chee and Ho (2009)3 randomly assigned to groups which role-played in the virtual world. While I agree this is certainly a good educational use for Second Life that takes advantage of the technology, no effort is made to compare it to traditional in-person role-playing. Without it, there’s no way know if SL is a reasonable substitute for traditional instruction.
- Lester and King (2009)4 is one of the better designed studies I saw. An online course held in Second Life was compared with a traditional course in terms of both student reactions and grades. Scores were similar across all grades, which is promising, although standard deviations (and standardized effect sizes) were not reported, so it is impossible to say just how similar they really were.
- Good, Howland and Thackray (2008)5 is another qualitative study examining the use of SL in a classroom setting. No comparison group, no statistics. Just a story, albeit an interesting one.
- Edirisingha, Nie, Pluciennik & Young (2009)6 describe yet another qualitative study. This one is not done through interviews, however, but through the analysis of chat logs. Again, interesting, but doesn’t help address my core question.
- Herold (2009)7 is yet another qualitative study of SL, this time for a Media Studies course in Hong Kong.
- McVey (2008)8 conduct a qualitative study of 10 students, and then generalizes that to several “lessons” of education regarding Second Life.
- Livingstone, Kemp & Edgar (2008)9 takes a similar qualitative approach.
- Dickey (2005)10 is the oldest reference I could find, with a discussion of two case studies of classes using Active Worlds as their primary course delivery mode. Again, interesting, but limited in generalizability.
The especially depressing thing about these studies is that most decry poor study design and the overabundance of theoretical work in the area, saying data is the only path to understanding the best way to use virtual worlds. So where is that data? On the bright side, a number of unpublished dissertations appeared in my list as well, which is promising – younger researchers have often had stats more recently. All that we can do is hope. Frankly, If anyone reading this has a virtual world they’d like to “prove” is effective, just let me know, and my team will freely help you evaluate it (it really is that easy!).
Several bits of this evidence come from the same place – a special issue of ALT-J, Research in Learning Technology, volume 16, number 3. If you want further reading, that would be a good place to start.
- Delucia, A., Francese, R., Passero, I., & Tortora, G. (2009). Development and evaluation of a virtual campus on Second Life: The case of SecondDMI Computers & Education, 52 (1), 220-233 DOI: 10.1016/j.compedu.2008.08.001 [↩]
- Jarmon, L., Traphagan, T., Mayrath, M., & Trivedi, A. (2009). Virtual world teaching, experiential learning, and assessment: An interdisciplinary communication course in Second Life Computers & Education, 53 (1), 169-182 DOI: 10.1016/j.compedu.2009.01.010 [↩]
- Jamaludin, A., Chee, Y., & Ho, C. (2009). Fostering argumentative knowledge construction through enactive role play in Second Life Computers & Education, 53 (2), 317-329 DOI: 10.1016/j.compedu.2009.02.009 [↩]
- Lester, P.M. & King, C.M. (2009). Analog vs. Digital Instruction and Learning: Teaching Within First and Second Life Environments Journal of Computer-Mediated Communication, 14 (3) : 10.1111/j.1083-6101.2009.01449.x [↩]
- Good, J., Howland, K., & Thackray, L. (2008). Problem-based learning spanning real and virtual words: a case study in Second Life ALT-J, 16 (3), 163-172 DOI: 10.1080/09687760802526681 [↩]
- Edirisingha, P., Nie, M., Pluciennik, M., & Young, R. (2009). Socialisation for learning at a distance in a 3-D multi-user virtual environment British Journal of Educational Technology, 40 (3), 458-479 DOI: 10.1111/j.1467-8535.2009.00962.x [↩]
- Herold, D. (2010). Mediating Media Studies – Stimulating critical awareness in a virtual environment Computers & Education, 54 (3), 791-798 DOI: 10.1016/j.compedu.2009.10.019 [↩]
- McVey, M. (2008). Observations of expert communicators in immersive virtual worlds: implications for synchronous discussion ALT-J, 16 (3), 173-180 DOI: 10.1080/09687760802526673 [↩]
- Livingstone, D., Kemp, J., & Edgar, E. (2008). From Multi-User Virtual Environment to 3D Virtual Learning Environment ALT-J, 16 (3), 139-150 DOI: 10.1080/09687760802526707 [↩]
- Dickey, M. (2005). Three-dimensional virtual worlds and distance learning: two case studies of Active Worlds as a medium for distance education British Journal of Educational Technology, 36 (3), 439-451 DOI: 10.1111/j.1467-8535.2005.00477.x [↩]