Even If Job Applicants Cheat, Online Testing May Still Increase Job Performance
When hiring with online tests (a concept called unproctored internet testing [UIT]), one of the biggest worries is that test-takers will cheat. A home computer is just about as “unsecured” a testing environment as possible, so test-takers have many options to deceive their potential employers: looking up answers on the Internet or getting a friend to complete the test for them are just two options. When people cheat, their test scores no longer represent their ability level, which reduces the average job performance of those that ultimately get hired. One remedy for this is something called “verification testing,” where job candidates are re-tested at a later stage of the selection process: for example, they might complete an intelligence test online and, after passing it, take a parallel version of that test in person. But many organizations don’t want the extra hassle of verification testing, because it takes more time and effort on the part of both the organization and the job candidate. So if people are cheating, can the organization still come out ahead by using online testing?
In an upcoming article in the International Journal of Selection and Assessment, Landers and Sackett1 simulate the conditions under which mean job performance of those hired will be different when job applicants are cheating. This includes both positive and negative changes to job performance, relative to the selection system used before UIT was adopted. The key is recruitment; if online testing enabled an organization to reach a large pool of applicants, it can be more selective, improving mean job performance.
from Landers & Sackett, 2012
You can see a sample of the researchers’ findings in the table here. The top left corner (0.24) represents the baseline. With the criterion-related validity chosen for simulation, job performance is 0.24 standard deviations higher than the previous system when no one is cheating and the applicant pool is not increased. We can compare the other values in the table to this baseline. For example, if 10% of your applicant pool is cheating but you double the size of your applicant pool, online testing results in a job performance level of 0.32 (over a 33% increase in job performance!). However, if 50% of your applicant pool is cheating, doubling the applicant pool will result in lower job performance over your old system (about a 46% decrease!).
Perhaps more disturbingly, some simulation conditions resulted in not only lower than baseline job performance but negative job performance. In other words, under some conditions, using online testing resulted in lower job performance than hiring at random.
This points to a pressing need for future research to identify what percentage of applicants we would expect to cheat and other simulation parameters. Unfortunately, dishonest behavior is one of the most difficult areas of study in psychological research. While in most studies, the biggest concern in this area is participant apathy (i.e. bored undergraduates not paying attention), people are outright working against you when trying to determine how dishonest they are. We can only hope this research spurs further work in this area.
- Landers, R. N., & Sackett, P. R. (2012). Offsetting performance losses due to cheating in unproctored Internet-based testing by increasing the applicant pool. International Journal of Selection and Assessment, 20 (2), 220-228. DOI: 10.1111/j.1468-2389.2012.00594.x [↩]
An upcoming article in Academy of Management Perspectives by Aguinis, Suarez-Gonzalez, Lannelongue and Joo1 investigates the accuracy of citation counts as a measure of how impactful an academic’s work in both in and out of academia. Short answer: while citation counts reflect the extent to which an academic’s work affects the work of his/her peers, they do not reflect the extent to which that work affects the world at large.
Business schools in particular place a great deal of importance on citation counts, often equating them with the importance of the scholar. A researcher with a citation count might be headhunted from another business school in order to increase the prestige of the hiring institution.
The problem is that citation counts do not necessarily capture the actual importance of a person’s work in anything beyond scholarly circles. If the purpose of science is to help the world (and I’d argue that it is), then citation counts capture something altogether different: they reflect how valuable other scientists view your work to be in supporting their own ideas. This is not really what we want to know when we ask, “how important is this scholar’s work?”.
Many fields (especially business-related fields) worry that their research is not adopted by those that could benefit from it most. Often, research never makes it beyond journal articles and into practice. So if we are really concerned with identifying the most “impactful” scholars, do citation counts capture that? Do highly cited authors have a bigger impact on the world than less-cited authors?
To determine how much exactly citation counts reflect larger impact, the authors found the number of citations to the top 550 most cited authors in the Academy of Management. They searched Google for these authors, using their full name with quotation marks to pull up a list of Internet references to that author. They then reviewed the first 50 pages of results to see how many actually referred to the author of interest. If more than 5% referred to someone else, they dropped that author from analysis. This resulted in a final database of 391 scholars.
In that database, the count of citations on Google did not correlate highly with citation counts: correlations ranged from .152 to .260 depending on whether or not you include .edu domains. In a multiple regression analysis, when controlling for years since earning the doctorate and the number of articles published, the number of citations did not predict substantial incremental variance in Google listings among non-.edu domains (delta-R2 of about one half of one percent).
In summary, if we believe the number of references in Google to be an accurate metric for capturing impact on the world at large, citation counts do not reflect this value. Impact is clearly a more complicated construct that we typically consider it; future work should investigate better ways to capture this. It’s also worth noting that while this approach works for business, where research results should be directly adopted by managers, it would not work so well for fields where there are several steps between research and adoption. For example, just because a nuclear physicist does not appear much on Google doesn’t mean that his work didn’t help build a nuclear power plant.
At the least, my 17000 results in Google put me around #300 of the 391 most-cited authors in the Academy of Management. Not too bad for 3 years out!
- Aguinis, H., Suarez-Gonzalez, I., Lannelongue, G., & Joo, H. (2012). Scholarly Impact Revisited Academy of Management Perspectives, 26 (2), 105-132 DOI: 10.5465/amp.2011.0088 [↩]
Grad School Series: Applying to Graduate School in Industrial/Organizational Psychology
Starting Sophomore Year: Should I get a Ph.D. or Master’s? | How to Get Research Experience
Starting Junior Year: Preparing for the GRE | Getting Recommendations
Starting Senior Year: Where to Apply | Traditional vs. Online Degrees | Personal Statements
Alternative Path: Managing a Career Change to I/O | Pursuing a PhD Post-Master’s
Interviews/Visits: Preparing for Interviews | Going to Interviews
In Graduate School: What to Expect First Year
Rankings/Listings: PhD Program Rankings | Online Programs Listing
So you want to go to graduate school in industrial/organizational (I/O) psychology? Lots of decisions, not much direction. I bet I can help!
While my undergraduate students are lucky to be at a school with I/O psychologists, many students interested in I/O psychology aren’t at schools with people they can talk to. I/O psychology is still fairly uncommon in the grand scheme of psychologists; there are around 7,000 members of SIOP, the dominant professional organization of I/O, compared to the 150,000 in the American Psychological Association. As a result, many schools simply don’t have faculty with expertise in this area, leading many promising graduate students to apply elsewhere. That’s great from the perspective of I/O psychologists – lots of jobs – but not so great for grad-students-to-be or the field as a whole.
As a faculty member at ODU with a small army of undergraduate research assistants, I often find myself answering the same questions over and over again about graduate school. So why not share this advice with everyone?
I’ve decided to return to this series because a few questions have come up from students that I realized I didn’t cover here. This week, I’d like to cover another important decision: should I go to graduate school at an online institution or a more traditional on-campus program?
This is actually part of one of the continuing “big arguments” in the field of education, so there aren’t many clear answers just yet. There is some evidence from the Department of Education that web-based courses are no less effective than in-person courses. That is, there’s reason to believe that two courses, similarly designed, one online and one in-person, will be essentially the same in their ability to teach you content. But the studies that the DOE summarizes are generally all undergraduate or laboratory studies, offering little insight into a) graduate courses or b) complete programs (not individual courses).
The experience of graduate school is certainly going to be different at these two categories of institution. One of the major benefits from brick-and-mortar graduate school is literally immersing yourself in the academic environment: being a part of a cohort of graduate students with similar experiences that you socialize with, interacting intensively face-to-face with professors about your academic achievement and career goals, gaining networking contacts that you will call on for the rest of your career, and generally learning about the culture of the profession.
Most of that is lost in an online environment. You’re not going to go out for drinks after a difficult exam with your classmates. As you likely already learned as an undergraduate in psychology, frequency of interaction and shared traumatic experiences are some of the best ways to ensure relationships form between people. This simply doesn’t really exist in an online program. While you might get to know people on discussion boards, it’s not quite the same. Think of it like the difference between your in-person friends and your “Facebook friends.”
The casual interaction with others in an academic environment also is beneficial developmentally. Completing a graduate program at home, you almost always have time to sit and think about your answers, to carefully consider your responses, and to put a lot of time and effort into producing the best answer possible. And this is certainly a valuable skill in an I/O career – but it’s not everything. If you ever plan to use your I/O degree in the “real world,” you’ll need experience coming up with answers on the fly and responding to/interacting with other experts. Many graduate students find that their first academic conference presentation, where they must respond to random questions from interested parties about their research results, is eye-opening in terms of the sudden pressure to think on their feet. Most students have already practiced this skill in their courses, and still find it challenging. For example, in my first-year Master’s-level Personnel Psychology course, I have students lead discussion for over an hour on a set of several journal articles. Without that kind of practice, I’d be a bit worried – and this directly translates into the kind of work you’d need to do .
I often find that students are considering an online program because they want to balance graduate school against a job. Let me be absolutely clear: this is a terrible idea. I fully expect my graduate students to be studying and working on research 40-60 hours per week on top of any teaching responsibilities. Teaching, at its most intense, should be a commitment of 10 hours per week. That is the maximally permissible distraction. If you plan to hold an outside job to support yourself during graduate school instead of teaching, you should be working less than 10 hours per week. Most part-time jobs don’t permit this and “strongly encourage” employees to increase their hours, so it’s generally not a good idea to have such a job while in graduate school. Remember, you’re in graduate school to prepare for your career. Every class you take, every bit of research you conduct, is now precisely targeted at giving you better opportunities later. Distracting yourself from that goal in any way will only hurt you in the long run.
More practically, there is some question as to the quality of online programs. One 2010 report by SIOP found several disturbing features of online I/O programs. For example, most online I/O programs don’t report who their faculty are. Of the PhD programs identified offering online I/O graduate degrees, only one program (Walden University) did report this, and of the 22 faculty, only two (2!) held I/O PhDs. That opens many questions about the expertise in I/O of those offering these degrees. Master’s programs had, on average, 1.5 I/O faculty. Only one online I/O Master’s program required a written thesis, which is necessary for anyone hoping to progress into a PhD program.
In the annual SIOP Survey reported here, most employers additionally had negative or neutral opinions about students coming from online programs. For example, respondents tended to respond positive to, “I tend to negatively evaluate a résumé if I notice that the applicant earned his or her graduate degree online.” and “I feel that there IS a meaningful difference in the quality of training that one receives in an online graduate degree program in I-O Psychology versus a traditional, in-person program in I-O Psychology.”
A more recent 2012 report by Rechlin and Kraiger found in an experimental study of I/O consulting firms that applicants from online programs tend to be evaluated more negatively than those coming from brick-and-mortar institutions by those making hiring decisions of I/Os. They discovered this by presenting resumes of effectively identical candidates (but with different names, distracting information, etc) crossing several degree characteristics. They found that those with online degrees were less likely to be asked for an interview, less liked to be hired, and likely to get a lower starting salary offer.
So what this really comes down to is priorities. Are you just trying to get the degree/credentials as a stepping stone for some other career goal, or are you trying to gain experiences that will help you create an I/O career? If you just want the degree, either type of program is probably fine. But if you’re trying to build a career within I/O psychology, at least for now, a brick-and-mortar institution is likely to put you on a superior trajectory, with better training, better opportunities, and better earning potential.