Skip to content

ChatGPT Can Write Undergrad-Level Essays with Accurate Citations

2023 March 20
by Richard N. Landers

I was asked recently if ChatGPT with GPT-4 running underneath has become capable of writing an essay at a level that would fool and instructor and earn a high grade. “That’s an empirical question” as they say, but my N=1 study shows the answer is yes. At least in psychology. Here’s an example.

My goal was to get ChatGPT to produce a five-paragraph essay about the Myers-Briggs Type Indicator, a notoriously controversial personality test often the center of discussions about test validity. I did a small bit of prompt engineering, but not extensively.

For an undergraduate essay in an introduction to psychology class, I think what ChatGPT produced is an “A” paper. For a class on personality specifically… maybe. It’s a bit light on details (e.g., it appears as if the conclusions/abstracts of each paper are taken at face value), and it’s not very creative. But I’d still say it’s an B+/A- sort of paper even then.

This is definitely a noticeable jump in capability from GPT-3.5, which would hallucinate citations. Specifically, it would produce citations with authors that appeared relevant (e.g., my MBTI request would likely produce last names of authors that had written on MBTI topics) but were not real (e.g., the article titles would be fictional, and the authors would be a weird mashup of people’s names and initials). That no longer seems to be the case – these citations appear to be generally correct, on-topic, and described accurately in the text.

More precisely, they at least appear “close enough” to reasonable to someone who isn’t a world class expert on the MBTI, i.e., most psychology instructors. There are a couple of red flags, ironically highlighted by ChatGPT itself – two citations lack DOIs. One of those (Carskadon & Cook, 1982) when looking at Google Scholar appears to be from a now-defunct journal created specifically to promote the MBTI and which has changed names over the years. The other (Myers et al., 1998) is the MBTI manual itself. So these are not precisely trustworthy citations in a discussion of MBTI pros and cons. But that is also not a level of citation investigation and critical thinking that I would expect out of undergrad intro students. Frankly, I don’t imagine the average intro psych instructor would bother to examine them at that level of detail either.

Here are the prompts I used:

  1. Can you provide citations suggesting the MBTI is a high quality personality inventory?
  2. I heard that the MBTI is a poor quality personality inventory. What are some major academic citations to support this?
  3. Write a paragraph explaining the pros and cons using the MBTI as a personality inventory, using the citations above.
  4. Write a five-paragraph essay on this same topic.

And here is the result:

Title: The Myers-Briggs Type Indicator: A Balanced Assessment of Its Pros and Cons

Introduction:
The Myers-Briggs Type Indicator (MBTI) is a widely known and utilized personality inventory based on Carl Jung’s theory of psychological types. Despite its popularity in various contexts, such as career counseling, team building, and personal development, the MBTI has received mixed reviews from researchers and psychologists. This essay aims to provide a balanced assessment of the pros and cons of using the MBTI as a personality inventory, drawing from the cited academic sources.

Paragraph 1 – Pros of MBTI:
The MBTI has been praised for its ability to provide valuable insights into individuals’ preferences and behaviors (Myers et al., 1998). Some studies have supported the construct validity of the MBTI, suggesting that it can effectively measure the intended psychological constructs (Thompson & Borrello, 1986). Furthermore, the MBTI has been shown to be useful in certain applications, such as increasing self-awareness and facilitating interpersonal communication (Carskadon & Cook, 1982). In these contexts, the MBTI can serve as a valuable tool for personal growth and team development.

Paragraph 2 – Cons of MBTI:
Despite its widespread use and popularity, the MBTI has faced significant criticisms regarding its scientific rigor and psychometric properties. Critics argue that the MBTI lacks reliability, meaning that the results may not be consistent over time (Pittenger, 1993; Boyle, 1995). Additionally, its validity has been questioned, as some researchers have found that the MBTI may not accurately measure personality traits when compared to other personality assessments like the Big Five personality traits or the NEO Personality Inventory (McCrae & Costa, 1989; Furnham, 1996). These limitations cast doubt on the MBTI’s usefulness in research or clinical settings.

Paragraph 3 – Context is Key:
While the MBTI has its shortcomings, it is essential to recognize the importance of context when assessing its utility. In less formal settings, such as career counseling or team building, the MBTI may provide valuable insights and serve as a starting point for conversations about individual differences (Myers et al., 1998). However, in more rigorous research or clinical applications, where reliability and validity are crucial, alternative personality assessments with stronger empirical support may be more appropriate (McCrae & Costa, 1989; Furnham, 1996).

Conclusion:
The Myers-Briggs Type Indicator, while popular and valuable in certain contexts, has its share of pros and cons. Its usefulness in promoting self-awareness and facilitating communication is undeniable, but its limitations in terms of reliability and validity cannot be ignored. It is essential to approach the MBTI with a balanced understanding of its strengths and weaknesses, considering its appropriateness for a given situation and exploring more empirically supported alternatives when necessary.

References

Boyle, G. J. (1995). Myers-Briggs Type Indicator (MBTI): Some psychometric limitations. Australian Psychologist, 30(1), 71-74.
Link: https://doi.org/10.1080/00050069508259607

Carskadon, T. G., & Cook, D. A. (1982). Validity of MBTI descriptions as perceived by recipients unfamiliar with type. Research in Psychological Type, 5, 89-94.

Furnham, A. (1996). The big five versus the big four: The relationship between the Myers-Briggs Type Indicator (MBTI) and NEO-PI five factor model of personality. Personality and Individual Differences, 21(2), 303-307.
Link: https://doi.org/10.1016/0191-8869(96)00047-0

McCrae, R. R., & Costa, P. T. (1989). Reinterpreting the Myers-Briggs Type Indicator from the perspective of the five-factor model of personality. Journal of Personality, 57(1), 17-40.
Link: https://doi.org/10.1111/j.1467-6494.1989.tb00759.x

Myers, I. B., McCaulley, M. H., Quenk, N. L., & Hammer, A. L. (1998). MBTI manual: A guide to the development and use of the Myers-Briggs Type Indicator (3rd ed.). Palo Alto, CA: Consulting Psychologists Press.

Pittenger, D. J. (1993). The utility of the Myers-Briggs type indicator. Review of Educational Research, 63(4), 467-488.
Link: https://doi.org/10.3102/00346543063004467

Thompson, B., & Borrello, G. M. (1986). Construct validity of the Myers-Briggs Type Indicator. Educational and Psychological Measurement, 46(3), 745-752.
Link: https://doi.org/10.1177/0013164486463032

Counts and Demographics of IO Psychology Doctoral Degrees

2022 December 5
by Richard N. Landers

Have you ever wondered about the demographics of IO Psychology PhDs? The National Science Foundation recently released its report from the 2021 Survey of Earned Doctorates, which summarizes trends in doctoral education across US universities. There are a few interesting tidbits about the IO psychology degrees awarded back in 2021:

  • IO psychology PhD programs awarded 181 research doctorates in 2021, which accounts for 4.9% of all doctoral degrees in psychology and 0.3% of all doctoral degrees awarded.
  • 64.6% of IO psychology research doctorates were awarded to women (versus 73.0% for psychology in general).
  • 11.0% of IO psychology research doctorates were awarded to international students (versus 8.3% for psychology in general).
  • The racial breakdown of American IO PhD psychology graduates, compared to psychology in general, was:
    • 11.4% Hispanic or Latino (vs. 11.7% for psych in general)
    • 0.0% American Indian or Alaskan Native (vs. 0.3%)
    • 9.4% Asian (vs. 6.4%)
    • 8.7% Black or African American, not Hispanic (vs. 7.9%)
    • 65.1% White, not Hispanic (vs. 66.2%)
    • 4.0% More than one race (vs. 4.1%)

So overall, it looks like the profile of awarded IO psychology doctoral degrees looks very similar to the that of the field in general. The exception appears to be gender balance, for which IO is closer to gender parity than almost all other subfields of psychology (although still 15% away from equal odds).

You can find the source for all of these values, and a huge amount of additional data not specific to IO psychology, by looking at the tables generated from survey data. Most of these numbers came from Tables 3-1 through 3-3.

I’d be very curious to see how these numbers change following people into career paths but could not find any relevant data. If you know of any, please let me know!

A New List of I-O Psychology Podcasts

2021 April 16
by Richard N. Landers

We’re currently in the midst of the Society for Industrial and Organizational Psychology’s annual conference, which as appropriate to a pandemic, is mostly via Zoom. Over three thousand people are participating.

Another side effect of the pandemic, one that I did not expect but realized because of the online conference, is that there has been an explosion in I-O psychology podcasts. I also realized 1) there is no centralized record keeping for I-O podcasts and 2) I-O podcasts are hard to search for because most don’t use the words “industrial-organizational” or “I-O” because you obviously don’t want to scare away your audience.

Classifying podcasts as “I-O podcasts” is a trickier thing than you might imagine. Although anything about “workplace behavior” is potentially I-O-relevant, my goal here is to present podcasts hosted by people who identify to the world at large as I-Os, especially in their podcasts. That’s a much smaller list.

With all that in mind, here’s a list of the I-O podcasts I’m currently aware of that have had a new episode within the last 60 days, along with some short commentary for the first few followed by descriptions (happy to add yours if you leave a comment!).

The Great I-O Get-together with Richard Landers and Tara BehrendThis first one is obviously the best, and it’s obviously the best because Tara is involved. Available as both a YouTube Live variety show and as a podcast for those without the time to watch live YouTube variety shows, our goal is to bring the I-O psychology community together, whether you’re an academic, practitioner, student, or all three at the same time. We usually play a game, like trivia or would-you-rather, in the first half, and have interviews in the second half. We also have a Discord community where you can connect with your fellow I-Os. And just to be clear, this is a show for I-O, not a show about I-O. The rest of these will be alphabetical. 😉
Bias Check-In with Susana Gomez Ornelas and Claudia Della Pona“What biases do we carry with ourselves in our daily lives, classrooms, and careers? A podcast hosted by IO psychologists and international students Susana GĂłmez Ornelas & Claudia Della Pona.”
The Business of Behavior with Becca Tagg“Getting clear about our values will help us to make changes in our business that lead to greater fulfillment, productivity, and abundance.”
Coaching for Leaders with Dave Stachowiak“Leaders aren’t born, they’re made. This Monday show helps you discover leadership wisdom through insightful conversations. Independently produced weekly since 2011, Dr. Dave Stachowiak brings perspective from a thriving, global leadership academy, plus more than 15 years of leadership at Dale Carnegie.”
Department 12 with Ben ButinaOne of the longest-running I-O podcasts, this is “a podcast for industrial-organizational psychologists, researchers, practitioners, and students. We cover anything and everything related to the research and practice of IO psych.”
Healthy Work with Keaton Fletcher and Maryana ArvanKeaton may need to work on his sales pitch, although the show is great!: “We are two Industrial-Organizational psychologists who care about how to make work a healthier experience for everyone. We run a bi-weekly podcast to bring the science directly to your ears.”
HRchat with Bill Banham“…get insights, strategies, and anecdotes to help you get more from the world of work. Show topics covered include HR Tech, Leadership, Talent, Recruitment, Employee Engagement, Recognition and Company Culture.”
The Indigo Podcast with Ben Baran and Chris Evertt“Take control of your life, business, and career once and for all. The Indigo Podcast offers raw, unfiltered, and science-based talk about flourishing in life. Join CEOs, executives, and regular folks in learning how to build a better future at work and beyond.”
Mind Your Work with Nicholas Bremner and Jose Espinoza“Mind Your Work is a podcast about understanding the human aspects of work through science.
We live in an interesting time. Employers are paying more attention to their people than ever. Employee wellness is now considered important. What makes a good career has changed from “a stable job” to “inspiring work”. At the same time, research on people at work is growing faster than ever. Employers are working on how to best use science to attract, retain, and engage employees. But what difference does all this make?”
TyePod with Tianna Tye“Thanks for tuning into TyePod, the go-to podcast for entrepreneurs building and leading teams. Your host, Tianna Tye, aims to bring you research-backed and practice validate tips, tools, and techniques to build a team that you can trust.”
WorkLife with Adam GrantAdam Grant is a somewhat polarizing figure in I-O. He’s arguably the most visible I-O psychologist on the planet, and has done a lot of good with that platform, but also falls pretty firmly in the category of “populist” these days, for better and/or worse. Here’s the sell for his podcast: “You spend a quarter of your life at work. You should enjoy it! Organizational psychologist Adam Grant takes you inside the minds of some of the world’s most unusual professionals to discover the keys to a better work life. From learning how to love your rivals to harnessing the power of frustration, one thing’s for sure: You’ll never see your job the same way again.”
The Workr Beeing Podcast with Patricia Grabarek and Katina Sawyer“Do you want a happier and healthier work life but don’t know where to start? Patricia and Katina, two bff organizational psychologists, share research and tips about workplace wellness and interview other leading experts in the field.”