Postmortem: Game-based Assessment Conference 2019

In the long tradition of postmortems for video game releases within the game development community, I decided to write up a brief one for the game-based assessment (GBA) conference I hosted on August 22-23, 2019. Its title was Game-based Assessment: An Interdisciplinary Workshop Integrating Organizations, Education, and Assessment. You can still browse its website, which includes the full program plus slides from presenters that gave us permission to share them. You can also still view the recording of the YouTube live streams from either Day 1 sessions or Day 2 sessions of the conference. The conference brought together I-O psychologists, management personnel, educational researchers, educators, GBA vendors and consultancies, organizational consultants, data scientists, GBA designers, and others to discuss the current state-of-the-art in GBA.

The purpose of postmortems is to reason through how we ultimately ended up with the final product we did. I will hazard a guess that this is a pretty unfamiliar idea to most I-O psychologists (i.e., my field). I-Os are typically trained in scientific methods rather than engineering methods, so the mindset is much more of a “consult the literature and do what it says” approach than a “consult the literature and build a workable solution by combining what it says with your expertise and design methods.” In the real world, I-Os often end up using design methods but typically don’t realize that’s what they are doing (e.g., gold-standard survey development is an inherently iterative design process, but I doubt you’ll even find the word “iteration” in a book on it).

As a bit of background, this conference was funded by a National Science Foundation workshop grant under the Science of Organizations program. The general purpose of NSF grants is to promote basic research. Thus, the overall goal of our grant was to bring together these diverse perspectives to promote a more explicitly interdisciplinary approach to GBA research, versus the multidisciplinary approach currently common. In the current approach, researchers more often have a strong basis in their home field and just enough passing expertise in related fields to build the GBA they need for their project, or to adapt some pre-existing game. I’m trying to encourage more integration, both across academic-practitioner lines and across fields, to ultimately build better GBAs in terms of validity, fairness, and quality of gameplay.

In terms of organizing the conference, challenge one was finding a venue. We ended up at the Graduate Minneapolis, which is adjacent to the University of Minnesota Twin Cities East Bank campus, for both a hotel block and for the conference itself, which ended up being fine, but it was my first experience working with a full corporate-style “events team.” I had three or four different points-of-contact within the hotel (honestly, still not sure), and it wasn’t always clear which person was responsible for the thing I had a question about. We were also on a tight timeline; although we applied for conference funding in November 2018, the winter shutdown of the federal government also shut down NSF, and it took many months for everyone to catch up. We didn’t get our funding notification until around March, which was not a lot of time to prepare for a conference to be held in August. Things were made worse by the 3-week-or-so delay between funding notification and actual delivery of funds, so I was negotiating with hotels in good faith that the money would actually appear in my grant account in time to pay them.

Processes involving money were in general quite a surprise. If you’ve never tried to book a hotel/event combo before, it goes roughly like this: 1) you work with a sales manager to put together a quote and purchase order, 2) the purchase order is a “high estimate” and includes the cost of all of your hotel block’s rooms in the event too few sell, 3) the purchase order must be approved by your organization, which commits them to that max amount, 4) a new manager takes over and writes up “work orders” which may or may not actually match the numbers the sales manager gave you, and in our case was two managers, because the event manager and the tech manager were different people, 5) those work orders get modified before and during the event, as if for example you need an extra plate of food or an extra microphone, that updates the totals for the work order, although such request filters back through the kitchen or on-site tech managers, also new people, 6) all of that gets reconciled into a final invoice, and 7) the invoice is then paid for by the same entity that signed the purchase order, which isn’t you, so it requires coordinating between whoever is responsible for such things at your organization and whoever is responsible for it at event space, which is usually yet another manager you’ve never talked to before or possibly an “accounts receivable agent.”

So as you might guess, the layers of bureaucracy were frankly a bit astounding – and this was just to get the space and food ordered! A mistake-in-hindsight that I made was to budget most of our money toward speaker fees and graduate student travel and leave most of the logistical support to my unpaid self plus a couple of lightly paid graduate students. In hindsight, we should have funded two fewer people to travel and hired an event planner with that money. I even found myself two days before the conference printing name badges on the printer in my office. That is honestly not the sort of stress one needs right before hosting a 100+ person event!

I was also responsible for running the camera/live-streaming system while hosting, which in hindsight, is also something I should’ve hired someone to do. On top of the last-minute scramble to figure out software and equipment, it left the camera unstaffed any time I got up to speak, which as the sole host, is not infrequent. One surprise was just how easy live-streaming ended up being; we used Open Broadcaster Software, which is a free platform and allows you to mix-and-match whatever video and audio streams you want. I ended up tapping into the room’s mic system using a Shure X2U (which allows you to plug a microphone XLR cable into a computer via USB – and you basically get a free high-quality microphone with the linked bundle), so we got nice crisp audio from the speaker, and splicing that onto the video feed from a Logitech C920S streaming webcam set up on a 10″ tabletop tripod from a table near the front. That’s it. If you already have a webcam you like, that makes the entire streaming solution about $200 since YouTube streaming is free, including archived recordings. The OBS software is also drag-and-drop; you basically click “add video from webcam” and drag the video feed wherever you want it in a window that represents the composite video that will be streamed. It was all dramatically easier than when I managed the online live streaming of the Leading Edge Consortium the first time they did it, around 7 years ago.

The event itself I think went well. We organized the schedule around stakeholder groups: a symposium for organizational sciences research, a symposium for educational research, a symposium for vendor research, and a symposium for graduate student research that won a competition we judged, plus three keynote sessions and one general+small group discussion session. I’ve had a lot of requests to run the conference again next year (we’ll see!), so I’m taking that to mean that people thought it was useful.

I was extremely happy with the Graduate Student Top Research Symposium. As mentioned earlier, we had a competition; students submitted abstracts for consideration of inclusion as a speaker, but graduate student funding was not contingent upon acceptance. All five of our winners gave great presentations and really showcased high-quality work – better than or at least on par with what the main lineup of presenters had to offer. The grad students tended to be more explicitly interdisciplinary, on top of being closer to “state-of-the-art” in general, so this is a tradition I’d really like to continue.

There were several interesting moments at the conference that I felt had to be carefully navigated. One of the interesting things about interdisciplinary environments is that you bring not just different expertise but also different values, traditions, and methods. What seems cutting edge in one field may be decades old in another. For example, after one organizational sciences presenter described how their GBA prototype took a year to put together and a huge amount of coordination, a person from education from another discipline commented “but you could’ve had a couple of students put that together in a weekend.” This sounds harsh but reflects just a fundamentally different way to approach the problem; in the organizational context, especially close to HR, everything is slow, steady, and carefully considered. In education, there’s a much greater willingness to experiment and move quickly, and work out the details later. Neither approach is objectively better than the other but rather a matter of opinions and tradition.

An interesting idea to come out of the conference is what I am tentatively calling GBAJam (no relation). Game jams are a common way for small teams of game designers and developers to prototype new ideas with little risk. The archetype for a game jam is a 24-hour jam, where everyone locks themselves in a room from noon Saturday to noon Sunday with the goal of creating a working game prototype by the end of the jam, which is then judged either by chosen judges or the community. An idea popped up that we could run a game jam for GBA; for example, we could construct small teams of either I-Os or education folks plus game designers plus developers, give them a 1-hour introduction to assessment (i.e., constructs, scale construction techniques, etc) and then compete to create a valid GBA, which we’ll test with real data, perhaps even MTurk studies funded by jam sponsors. Still early stages, but it’s an idea I’m excited about.

In any case, I think the event overall went well and am happy we made the connections and in-roads we did. I was pretty exhausted with the idea of another conference next year, but as I’m getting further away from it, the idea is not quite so horrible. So we will see. In the meantime, I hope a lot of good connections came out of the event and triggers a lot of new, meaningful research!