NSF Report Flawed; Americans Do Not Believe Astrology is Scientific
A report from the National Science Foundation recently stated that a majority of young people believe astrology to be scientific, as reported by Science News, Mother Jones, UPI, and Slashdot, among others. Troubling if true, but I believe this to be a faulty interpretation of the NSF report. And I have human subjects data to support this argument.
What the NSF actually did was ask the question, “Is astrology scientific?” to a wide variety of Americans. The problem with human subjects data – as any psychologist like myself will tell you – is that simply asking someone a question rarely gives you the information that you think it does. When you ask someone to respond to a question, it must pass through a variety of mental filters, and these filters often cause people’s answers to differ from reality. Some of these processes are conscious and others are not. This is one of the reasons why personality tests are criticized (both fairly and unfairly) as valid ways to capture human personality – people are notoriously terrible at assessing themselves objectively.
Learning, and by extension knowledge, are no different. People don’t always know what they know. And this NSF report is a fantastic example of this in action. The goal of the NSF researchers was to assess, “Do US citizens believe astrology is scientific?” People were troubled that young people now apparently believe astrology is more scientific than in the past. But this interpretation unwisely assumes that people accurately interpret the word astrology. It assumes that they know what astrology is and recognize that they know it in order to respond authentically. Let me explain why this is an important distinction with an anecdote.
It wasn’t until around my sophomore year of college that I discovered the word “astrology” referred to horoscopes, star-reading, and other pseudo-scientific nonsense. I had heard of horoscopes before, sure, but not the term astrology. I had, as many Americans do, a very poor working vocabulary to describe scientific areas of study. Before that point, in my mind, astrology and astronomy were the same term.
I did not, however, think that horoscopes were scientific. I simply did not know that there was a word for people who “study” horoscopes. If you’d asked me if astrology was scientific before college, I would have said yes – because to me, astrology was the study of the stars and planets, their rotations, their composition, the organization of outer space, and so on. Of course, in reality, it isn’t. Astronomy is a science. Astrology is the art of unlicensed psychological therapy.
When I saw the NSF report, I was reminded of my own poor understanding of these terms. “Surely,” I said to myself, “it’s not that Americans believe astrology is scientific. Instead, they must be confusing astronomy with astrology, like I did those many years ago.” Fortunately, I had a very quick way to answer this question: Amazon Mechanical Turk (MTurk).
MTurk is a fantastic tool available to quickly collect human subjects data. It pulls from a massive group of people looking to complete small tasks for small amounts of money. So for 5 cents per survey, I collected 100 responses to a short survey from American MTurk Workers. It asked only 3 questions:
- Please define astrology in 25 words or less.
- Do you believe astrology to be scientific? (using the same scale as the NSF study)
- What is your highest level of education completed? (using the same scale as the NSF study)
After getting 100 responses (a $5 study!), I first sorted through the data to eliminate 1 bad case from someone who entered gibberish when responding to the first question. Then I tried to replicate the findings from the NSF study by looking at a bar chart of #2 for the remaining 99 people. It was very similar to what NSF reported, as shown below.
Across the sample, approximately 30% found astrology to be “pretty scientific” or “very scientific.” This is lower than the NSF report found (42% for “all Americans”), but this is probably due to the biases introduced by MTurk in comparison to a probability sample of US residents – MTurk users tend to be a little more educated and a bit older. Still a pretty high proportion though.
Next, I went through and coded the text responses to identify who correctly differentiated between astrology and astronomy. 24% of my sample (24 of 99 people) answered this question incorrectly. And given the biases of MTurk, I suspect this percentage is higher among Americans in general. Some sample incorrect responses:
- Astrology is the study of the stars and outer space.
- Astrology is the study of galaxies, stars and their movements.
- the study of how the stars and solar system works.
- Astrology is the scientific study of stars and other celestial bodies.
These are in stark contrast to the correct responses:
- Trying to determine fate or events from the position of the stars and planets
- Astrology is the prediction of the future. It is predicted through astrological signs which are influenced by the sun and moon.
- Astrology is the study of how the positions of the planets affect people born at certain times of the year.
- The study the heavens for finding answers to life questions
For those statistically inclined, a one-sample t-test confirms what I suspected: if people generally did not have trouble distinguishing between astrology and astronomy, we would not have seen such an extreme number of incorrect answers: t(98) = 17.032, p < .001. There is definitely some confusion between these terms.
Next, I created the bar graph above again, but split it by whether or not people got the answer correct.
Quite a big difference! Among those that correctly identified astrology as astrology, only 13.5% found it “pretty scientific” or “very scientific”. Only 1 person said it was “very scientific.” Among those that identified astrology as astronomy, the field was overwhelmingly seen as scientific, exactly as I expected. This is the true driver of the NSF report findings. Both an independent-samples t-test and a Mann Whitney U test (depending on what scale of measurement you think Likert-type scales are) agree that the differences in science perceptions between those responding about astronomy and those responding about astrology is significantly different (U = 119.00, p < .001; t(97) = 10.537, p < .001). Massive effect too (d = 2.48)! Thus I conclude that it is invalid to simply ask people about astrology and assume that they know what that term means.
Various media reports have noted that the NSF report discussed how 80-99% of Chinese respondents in a 2010 Chinese study reported skepticism of various aspects of astrology, interpreting this difference as evidence supporting the decline of US science education. Instead, I suspect that this difference only reveals that in Chinese, the words for astronomy and astrology are not very similar. 86.5% is right in the middle of the range of findings from the Chinese sample, although I expect a broader US sample than MTurk would probably be a little lower. But at the least, I feel comfortable concluding that we are safe from Chinese scientific dominance for at least another year.
So why might this effect have been worse for young people? My guess is that the long-term effect is exactly the opposite from what the NSF reports. I suspect that young people are in fact more skeptical of astrology than ever before – and I believe this skepticism is driven by reduced exposure in youth. Young people just aren’t as likely to hear the word astrology in connection to horoscopes anymore, and probably less likely to hear about horoscopes at all because virtually no one reads newspapers anymore (which I also suspect is where most people were historically first exposed to them; I remember learning of horoscopes for the first time in the Tennessean many years ago as a child in Nashville). As people age, they’re more likely to hear the term “astrology” and say, “Oh, astrology means horoscopes? That’s obviously fake! Nothing like astronomy!” Perhaps the demise of print journalism isn’t as bad as we thought!
If you’d like to take a look at my data from MTurk, it is available to download here. Overall, I think the lesson from this is quite clear: more NSF funding for social scientists to prevent these problems in the future!!
Update 2/18: Thanks to @paldhaus and @ChrisKrupiarz, I discovered a European Commission report corroborating my findings, available here (see p. 35-36). In the Commission survey of the EU, a “split ballot” approach was used, asking half of respondents about how scientific “horoscopes” are whereas the other half were asked about how scientific astrology was. 41% of those in the EU identified astrology as scientific, whereas only 13% identified horoscopes as scientific. Since this was a 2005 study, it it surprising NSF has not altered their methods since.
| Previous Post: | Is Age-Related Mental Decline Not As Bad As We Think? |
| Next Post: | Do Recommendation Letters Actually Tell Us Anything Useful? |
Trackbacks and Pingbacks
- Most young Americans think astronomy is science | Stats Chat
- NSF Report Biased, Expert Says: Americans Don’t Think Astrology is Scientific | The Horoscope Reader
- Tesettur Giyim Trend | Did people confuse ‘astronomy’ with ‘astrology’ in the NSF study?
- Weekly Roundup: History of Science Videos & NSF Report | Darin Hayton
- NSF Report Biased, Expert Says: Americans Don’t Think Astrology is Scientific | Horoscope Homepage
- NSF Report Biased, Expert Says: Americans Don’t Think Astrology is Scientific | The One Horoscope
- Astrology ≠ Astronomy | Star Crossed | Whiskey Priest
- NSF Report Biased, Expert Says: Americans Don’t Think Astrology is Scientific - Technology Org
- Geloof Amerikanen in astrologie berust op begripsverwarring | Astroblogs
- I’ve Got Your Missing Links Right Here (22 February 2014) – Phenomena: Not Exactly Rocket Science
- Astrology: What it is and why it is still attractive - Development
Pretty cool resource for the quick study. Although, I’m not sure if your study is elegant or eloquent or how it might affect or effect me, but I’m sure the answers are their or there…I just have to look for them. 🙂
When I saw the headline I came to the same initial conclusion as you. For a long time I also thought astrology and astronomy were interchangeable. I also applaud you for doing some quick research to support your interpretation! Good work, and happy valentine’s day!
Great short study. I was having problems believing the original poll myself (although being very familiar with the quality of science education in students currently graduating from Texas high schools, I was afraid there might be an element of truth to it.)
But I think you nailed it. Long ago I realized that the statements/questions, “You’re an amateur astrologer?” and “So, you study the stars with a telescope,” really meant that my friends had no idea of the difference between the two words.
One caveat to your findings, I have met people who understand that ‘astrology’ is connected to horoscopes AND describe it in terms of what you listed as ‘incorrect descriptions’. So I see there as being a bit of a flaw, or an assumption, in your reasoning.
Many see astrology as the study of celestial bodies with precognition simply being an application or side effect, and ‘astronomy’ as simply a half blind version of astrology.
I absolutely agree; my “study” is certainly limited to the extent that I accurately interpreted their provided definitions. But that is why I uploaded and linked my data set; if you have an alternate analytic strategy that takes this into account, please feel free to re-analyze and share what you find.
Interpreting human subjects data is always fraught with potential misinterpetations; all we can do is try to be as accurate as possible. 🙂
awesome analysis. I think quite a few of us had the same hunch, but very nice job putting your hypothesis to the test.
you should consider posting the figures and data to figshare.com
I want to believe that this explains things, but the questions on the NSF survey did in fact explicitly reference horoscopes.
http://img.gawkerassets.com/img/19fdeysgu5je5jpg/ku-xlarge.jpg
Interesting! A couple of ideas come to mind. First, do you think survey-takers actually read the first sentence past “horoscope”? If they were just scanning quickly (looks like 267 pages of survey questions?), they might not have even connected the first and second parts of that question (which is why psychology relies on scale measures rather than single questions). Second, did they see the term ASTROSCI when taking this? That is quite a prime. Third, perhaps US MTurk users are just overwhelmingly better educated about science than the general US populace? Seems doubtful but also possible!
This statement isn’t right: Among those that correctly identified astrology as astrology, only 10% of the sample found it “pretty scientific” or “very scientific”.
Parsing your sample’s data, among those that correctly identified astrology as astrology, 13.5% found it “pretty scientific” or “very scientific”. 10 of 74 respondents.
Across the total sample, 10% BOTH correctly identified astrology as astrology AND found it “pretty scientific” or “very scientific”. 10 of 99.
Ah! You are correct – I wrote the 10% first and added the “among” part later, mixing up my conditional and conjoint probabilities (it’s still 10% of the sample, but it’s 13.5% of the group I was referencing). Thanks, fixed now!
I’m not sure how much of an improvement this is (if any)…
Your personal example depends on not knowing that there are 2 astrology questions asked in the GSS.
http://publicdata.norc.org:41000/gss/documents//WLPR/GSS2010_Ballot3_AREA_English.pdf
ASTROLGY: Categorical (Single)
Now, for a new subject. Do you ever read a horoscope or your personal astrology report?
ASTROSCI: Categorical (Single)
Would you say that astrology is very scientific, sort of scientific, or not at all scientific?
If you horoscopes are unscientific then the first question could clue you that astrology is too. Of course people can know neither word.
As I responded to another commenter earlier, this assumes that survey takers read the questions carefully enough to notice this. My guess is that as soon as they hit the word “horoscope”, they respond and move to the next question. It also presupposes that respondents infer that “horoscope” and “personal astrology report” are intended to mean the same thing. But if you didn’t already know the word “astrology,” that is probably not what would happen.
I think your response to HairBolus is too glib, considering problems with your own study.
You’re implicitly assuming that your pool of 100 test subjects from Mechanical Turk is representative of the general population. Consider that they all own or have access to computers with internet (only ~80% of the population), that they have heard of MTurk (bias towards technical savvy), that they have the patience or interest to complete your survey, etc. For instance, what stats do you have on the number of people who looked at your survey and decided not to do it? I’ll bet there’s a bias there.
You’ve not got any questions to test for possible bias: education levels, for instance, or urban/rural, or age.
I don’t know what you mean by “glib” exactly, but my response to HairBolus was an extension of my response to Eric Odell, who brought up the same issue, so I did not go into as much depth the second time around. As I have already described there and also a bit in the article itself, this study is certainly biased as far as it describes the US population and the MTurk population is different from that population. I don’t claim otherwise. However, there’s a growing body of work to suggest that US MTurk samples are reasonably representative of the US in general. Various experimental psychological effects have been replicated, and the range of intelligence, personality scores, etc, are not very dissimilar.
More critically, however, the limitations you are describing would actually suppress the effect I found. If MTurk participants are generally better educated, and if education is correlated with misunderstanding this concept, we would expect attenuation in the relationships I observed – the mathematical effects of range restriction on observed correlations are well known. Thus, your argument suggests that the use of MTurk may have resulted in an underestimate of the bias described here.
Finally, as I described above, I did collect education level. If you’d like to analyze it and report your findings, feel free: the dataset is linked above.
I find the authors’ tone offensive for two reasons: this line “I discovered the word “astrology” referred to horoscopes, star-reading, and other pseudo-scientific nonsense.”
A self admitted psychologist should not be impugning anything as pseudo scientific nonsense.
I point you to John Nelson, radio astronomer for RCA in the 1950s. Nelson used planetary positions (astrology) to accurately predict disruptions in shortwave radio transmissions.
And another scientist who is quoted in support of astrology, sir Issac Newton, who reportedly replied to Edmund Halleys’ criticism of his belief in Astrology “but sir, I have studied the matter and you have not”
“A self admitted psychologist should not be impugning anything as pseudo scientific nonsense.”
Actually, this precisely what a psychologist should do.
Upon seeing the initial news, I simply knew this had happened. Actually, when I was 9, this happened to me too.
There are further explanations as to why respondents who accurately defined astrology still decided it was “scientific”. Astrology uses the language and toolsof astronomy. Astrologist don’t claim that the Earth is the center of the Universe, that there are Celestial Spheres, etc. and generally have some precision in describing their practice (though many use out-of-date ephermeris which disregards precession). Given that, a person who doesn’t believe the central tenet of astrology – that somehow planets and stars affect our fate – could still characterize its practice as somewhat scientific.
The reason astrology confuses so many people is because more often than not they don’t have the good fortune of looking at the whole picture. My own experiences over the course of several years have revealed that my understanding of human interactions improved considerably once I factored in numerology, western astrology and chinese astrology. For instance, just take the day on which you were born. Then go and find your birth number here:
https://groups.google.com/forum/#!msg/sharing-mails/XczHuKCpHqg/UxyHfc-ju7YJ
Now make a list of your best friends and match your birth number with theirs. You will find that most of them are best matches or good matches according to the above page. Coincidence? Sorry but you would have to be an ignorant fool to write that off as mere chance.
Adnan, the page you linked describes nothing but coincidence.
There are only 9 birth numbers, so there is roughly a 11% chance that a person will be a particular number. There are 3-5 “best matches” for every one of these categories. If we assume that birth day is pretty much random (no more people born on any particular day than another), that means any particular friend of yours has a 33%-55% chance of being a “best match”. If you had 5 friends, it is virtually guaranteed that at least one (and probably more) of them will be a “best match”.
If anything, this is an excellent demonstration of how astrology takes advantage of people who don’t understand probability. This document creates the illusion of meaning by taking advantage of people who lack this understanding.
First, I agree that there are some problems in the questions asked. In the assessment industry, you have what is called a bivariance problem. The low performing group gets it wrong and the high performing group also gets it wrong, but because the question was poorly asked. Like “The Earth revolves around the sun.” To the highly educated, the Earth and Sun orbit a point that is (at least mathematically) distinguishable from the Sun itself, so the answer is “no”.
On the other hand, the fact that many people don’t know the difference between Astrology and Astronomy, I think, is part of the problem.
I think that not knowing the difference between Astrology and Astronomy, and how this could potentially impact the interpretation of the findings, is the point Dr. Landers has made.
Whether or not the fact that people cannot distinguish between the two is problematic is a separate question in my opinion (although not necessary unimportant). The issue here is that these results suggest indiviuals do in fact know the difference and endorse Astology as scientific.
But still, at the end of the day, it seems that the NSF findings are tenuous. If, in fact, individuals did not know the difference between the two, it is difficult to discern what their true feelings or beliefs are. And frankly that’s the flaw. Although I have not read the primary article, it seems as if there is nothing stating that these individuals knew the accurate definition of Astrology or knew that Astrology and Astronomy are conceptually different. Thus, it is a potential threat to the validity of the study, a fact supported by Dr. Landers’ findings.
Psychometricians advise survey developers to create items that are within the expected reading comprehension level of participants. Thus, if I ask people how often do they protect their epidermis to mitigate the carcinogenic effect of ultraviolet rays and assume that a low behavioral frequency rate means no one cares about skin cancer, I might find different results with 1) a more colloquial question or 2) defining the terms for the survey respondants.
1. Unfortunately for your argument, the study actually did tie horoscopes to astrology in the question preceding the question you attack, which may be why they didn’t define the term. See GSS Codebook, 1972-2012, quoted in other comments above. So the respondents answered knowing that astrology was tied to horoscopes.
2. In responses to critics, you assume that people read over the questions quickly. But this is an in-person survey, which is done in the home, and I think they take like an hour and a half to conduct it (don’t hold me to the last bit). This is not like MTurk where people are rushing to get paid, and the mostly middle-aged women who do the interviewing are the best-trained in the business.
3. You speculate that young people do not read horoscopes any more, but in the study the 18-29 age group was the MOST likely to read horoscopes, not the least likely. So again, the evidence contradicts your speculation.
4. If one believes that the stars influence our lives, then some of your qualitative language about astrology being a science is not inconsistent with the actual meaning that astrologists attribute to their field. E.g., “Astrology is the scientific study of stars and other celestial bodies” sounds like what astrologers believe. So saying this does not mean that they confused the word astrology with the word astronomy, but rather might mean that they falsely believe that astrology is a science.
5. Bottom line, probably some people confused the two, but your study does not show this or how frequent this is. The effect is probably not large. And you make some non-trivial mistakes in speculations and reasoning.
James Lindgren, JD, PhD
The text of the previous question read “Now, for a new subject. Do you ever read a horoscope or your personal astrology report?”
Here is the relevant question: If I asked you “Do you ever read a horoscope or your personal lindgrenology report?”, would you assume that horoscopes and lindgrenology are synonymous? I am suggesting that you would not, nor would those that completed the NSF study.
The bottom line is that single-item scales of psychological constructs are flawed. This is well-known in psychology. An NSF-sponsored report should not be making conclusions drawn from such data. Unlike those interpreting the NSF report, I do not pretend to be giving an unbiased estimate of “what Americans think.” I am merely highlighting an alternative statistical explanation for the effect that cannot be ruled out given the data available.
I will also point out that your statement that “the effect is probably not large” and that people completing the NSF survey are not “rushing to get paid” is, in your own words, speculation. I find your general position quite logically inconsistent; it is not reasonable to criticize something as speculation while contributing no data (i.e., speculating).
Certainly those who read horoscopes cannot be assumed to be unaware of what astrology is.
In the 2012 study, 37% of those 18-29 both read horoscopes and believe that astrology is at least sort of scientific. This is higher than any other age group. The average for all age groups is 26%.
James Lindgren
JD, PhD
You seem to be trying to imply a causal relationship between responses to one question in responses to another question (i.e. that because one number is high and another number is high, one must have caused the other). This is not logically valid (see http://en.wikipedia.org/wiki/Correlation_does_not_imply_causation).
Mr. Landers,
I will second Jim Lindgren’s critiques of your study. The questions asked on page 1729 of the National Opinion Research Center survey were extremely clear and quite unambiguous:
1062. “Now, for a new subject. Do you ever read a horoscope or your personal astrology report?”
1063. “Would you say that astrology is very scientific, sort of scientific, or not at all scientific?”
Confusing astronomy and astrology during the interview is indeed possible in some unique cases, but hardly the sizable issue that you make it out to be.
The Pew Research Center’s Religion & Public Life Project survey in 2009 calculated that 25% of the U.S. population believes in astrology. The more recent Harris poll in 2012 calculated it at 26%.
Jonathan Hall
I assume from your response that you did not read my reply to Mr. Lindgren. I will direct you at that response before replying further. Your statement implies that people would learn new vocabulary from reading a survey question and then incorporate that learning in their response to the following question. That is an unwise assumption.
Richard:
I redid your study and corrected some of the errors noted by commenters above.
You can speculate that it wouldn’t matter if you had asked the actual questions used in the GSS/NSF study, but the evidence is now otherwise.
It’s discussed in my post at the Washington Post website, where I blog:
http://www.washingtonpost.com/news/volokh-conspiracy/wp/2014/02/18/did-people-confuse-astronomy-with-astrology-in-the-nsf-study/
Properly done–using the actual NSF questions and probing further to explore what the respondent means by the “study of the stars”–reveals that perhaps only one of the 108 respondents in my replication of your study was confused between astrology and astronomy.
For example, a respondent defined astrology:
“Astrology is the science of the stars.”
When I asked how people would gain information about astrology, this respondent answered:
“I am not really sure because I don’t really believe in horoscopes.”
Obviously, this person is not confusing astrology with astronomy.
Almost all the other cases are the same:
2. DEFN: “study of stars”
INFO: “it tells the future and your personality traits.”
3. DEFN: “I believe it’s the study of the stars. study of the alignment of the stars and planets.”
INFO: “Maybe the stars can tell the future?”
We all make mistakes. Certainly I do.
While it is kind of you to respond so promptly to criticisms, instead of getting huffy with Mr. Hall and those of us who agree with a quite plausible position, perhaps you should have been a little less certain that you would have received the same results if you had used both of the actual NSF questions, rather than just two different questions that you made up yourself.
That question-framing affects results is well-settled in the survey research literature.
James Lindgren, JD-PhD
Professor of Law
Northwestern University
I am only huffy when I am asked to repeat myself. 🙂
From your reply, and from your post on your blog, I think you may be forgetting the timeline. You came here to criticize my conclusions without any data of your own to contribute. I pointed out, quite rightly, that your assertions did not change my data. I then provided a plausible explanation addressing your criticisms. Only after that point did you collect your own data.
Only after all that had occurred did you make this statement: “You can speculate that it wouldn’t matter if you had asked the actual questions used in the GSS/NSF study, but the evidence is now otherwise.”
Now, with the new information you have provided, I agree that we now don’t need to speculate! This is an exciting development, although I disagree that “mistakes” have been made. That is the very nature of science – self-correction. Perhaps it is common in law for there to be a “right answer”, but science is a bit more fluid than that.
I did notice that you did not provide your data, and the SSRN report is unrelated to your MTurk study. Would you provide your data set please, as I did?
I’m curious if you were able to obtain any information regarding your speculation on the difference between the Chinese survey and the US survey.
No – that’s unfortunately not directly testable given the data available, or the data I would be able to collect (because I do not speak any Chinese languages). The traditional way to do that sort of comparison in Psychology would be to use a technique called back-translation – where you ask independent experts to translate a survey from one language to another, then a new set of experts to translate it back without seeing the original. If everything agrees 100%, you have the first piece of evidence to suggest that the two surveys actually ask the same questions. But even then, there are still doubts. If this was within my research program, that would be a good angle to approach next – but it isn’t. Perhaps Jim will take this up though? This appears to be close to his core interests.
1. I appreciate the more conciliatory tone.
2. I sent you my data last night by email and gave you permission to post it here if you wish.
3. The Europoean study does indicate that it matters how the question is framed, which is part of what I was arguing. Leaving aside the foreign language issue, the European study does indeed show some lack of understanding of astrology when astrology is not tied to horoscopes; it does not show whether people would misunderstand astrology in the NSF study where astrology is tied to horoscopes and personal astrology reports. (I couldn’t find any indication in the European report whether other questions might have cued respondents to the meaning of astrology, so I’ll assume they didn’t.)
4. If you would like, I would be happy to join you in another small-scale replication of my study (which was a modest expansion of your study), where we can both ask open-ended follow-up questions, and I would be happy to give you first crack at the data. So long as the two actual NSF questions are asked first and they are followed by open-ended questions probing what astrology means, I would expect to obtain similar results to those I obtained yesterday. Besides the definitional question, I think it might be good to have questions about how to acquire information about astrology and how people might use astrology in their lives.
James Lindgren
Hmm. My initial reaction is that I don’t think the study you’re describing would be productive at this point – I think we’ve both demonstrated (perhaps unsurprisingly) that framing is important when asking questions about “astrology” and that the NSF questions are at least somewhat flawed as measures of belief in astrology (in different ways, but both inflating the negative perception of American science education). The problem may not be as severe as I indicated, but you’ve also identified an additional wording flaw that is influencing the results to an unknown degree. Exploring these issues seems preferable to straight replication at this point.
Were this my research program, I think I would next pursue one of four other approaches as a more valuable next step:
1) to experimentally assign respondents to one of multiple phrasings and observe differences on some relevant outcomes,
2) to ask multiple questions about how scientific multiple dimensions of astrology are (e.g. asking about horoscopes, star-reading, tarot cards… and whatever else astrology involves, as separate questions),
3) to replicate the GSS wording but changing “astrology” to a scientific-sounding nonsense word (i.e. my lindgrenology example before) and see if the endorsement effect on the second question remains, or
4) to identify the dimensionality of how people perceive things to be “scientific”, since I think your data demonstrate this concept to be multi-dimensional (and poorly measured as a single question).
1. It’s not really your research program–or mine! So maybe we shouldn’t go too much further.
2. I think Qualtrics can handle randomization pretty easily between ballots, but I haven’t yet done it there.
3. I think I get the point of your idea #3 above, but I’m not sure which NSF question you want to revise with a nonsense word. I think at this point I’ll pass on that idea, but I am open to being persuaded if you feel strongly. Interestingly, the NSF’s reading horoscopes question doesn’t have wording problems, but it does raise a question whether they are read for entertainment or for belief.
Just thought I’d point out that there’s something important missing from your (admittedly informal) study. You ask people to define “astrology” and you ask them if they think “astrology is scientific.” All very good.
But do you ever ask them what they mean by “scientific”? I’d guess that you’ll find a lot of surprises and variations there — even among very well educated people.
That’s a great point – but it does change the scope of the research here. The NSF study (and my little one) essentially ask, “Do you believe these words to represent scientific fields of study, however it is that you define scientific?” When you mean something specific by “scientific”, this becomes much more complicated.
So I absolutely agree that personal definitions of “scientific” are going to vary by person… but that is an entire research question unto itself. But this is also why psychologists don’t recommend single item scales – when you ask the same question 10 different ways, you can be more confident that the shared response patterns between those questions actually tells you something meaningful.
Drs. Landers & Lindgren,
Since I was at least partially responsible for adding in the question of the European study, I’d be willing to fund an MTurk study with a split ballot like the EU study with these questions:
1. Do you ever read a horoscope or your personal astrology report? [Yes/No]
2. Would you say that astrology is very scientific, sort of scientific, or not at all scientific?
&
1. Do you ever read a horoscope or your personal astrology report? [Yes/No]
2. Would you say that horoscopes are very scientific, sort of scientific, or not at all scientific?
and then send you the data. (This is not my area of expertise, but am curious how Mturk works.)
Chris
One more quick question that is more out of curiosity. I assume one of the advantages of the GSS study is that it’s the same question asked over many years. Hence, the wording is pretty much set in stone when the study first started, right? Not using the GSS as an example since arguably the wording isn’t an issue, but have you ever had to deal with a situation where you wanted a long term study but the wording, over time, made less sense either due to language changes or other factors?
I think that would certainly be an interesting contribution to this conversation – please let us know what you find. If you haven’t used MTurk before, I’d warn you to be sure that you don’t require Master Workers (this is a very narrowly defined group), restrict to US respondents only, and a couple of quality restrictions (I usually set >95% approval rate plus at least 50 HITs completed). However, I did want to mention that one of the things that’s come up in my emails back and forth with Jim is that the scientificness scale I used actually is also used in the GSS (hence my confusion), but on the questions about how scientific specific (real) academic fields are perceived. So when asking about medicine, sociology, etc., the 4-point scale was used – when using the astrology scale, the 3-point scale was used. That means the scales are not comparable across questions; thus it is not clear how much scale choice is affecting responses either. No easy fixes here. 🙂
Your design does better get at another issue than either of ours – what about people who think astronomy and astrology are synonymous or related, but think the astronomy side is scientific (planets definitely have paths!) and the horoscope side isn’t (but they don’t predict anything about your life!)?
As far as longitudinal research goes, this type of benchmarking research is uncommon in Psychology – the general thought is that it doesn’t really reveal much that can be used to predict human behavior (there are better methods, like experimentation, if that is your goal). There are also downsides to having large numbers of researchers mine the same data set for whatever little publishable nuggets they can find. So I’ve never personally faced the problem you’re describing.
Having said that, when specific measures exist over long periods of time (used for many years across multiple studies), they do need to be updated periodically – for example, some old personality tests asked questions about things that people previously did but now generally don’t (e.g. questions about general preferences between types of radio programs, for example). When items are changed, there are ways to sort of calibrate old phrasing with new phrasing, but you can never be sure they are equivalent.
Of course, in science, you can never be sure of anything. 🙂
Chris: Your question about the GSS being tied to existing question wording is absolutely true. They try very hard not to change questions, so that trends can be found. Psychologists tend to look more often at how people think and feel, while sociologists like to track changes in beliefs and opinions over time. And the GSS usually gets about a 70% response rate!!!
I actually have 7 questions on one ballot of the 2014 GSS (if you have to buy a spot on the GSS, it usually costs over several thousand dollars a question for every 700 respondents)
My PhD at the U. of Chicago was in Sociology with specializations in Survey Research and Statistical Methods, so I trained with the people who designed and ran the survey. I even took a course on questionnaire design.
If you are serious about using MTurk, you should give me a call (773-294-9043), 10am to 10pm US Central Time (Chicago). I would second Landers’ advice: US only, 95% (or 98% or 99%) approval rate on prior jobs. BTW, the two big markets for MTurk workers are the US and India.
Unless you are a modestly adept HTML programmer, I think it much easier to conduct the survey at Qualtrics. I believe that you can randomize your experimental question there as well, which you can’t do at MTurk (if you do it solely at MTurk, I think you would have to do 2 separate surveys and then manually drop the people who took both surveys, despite your telling them not to).
So you set up the study at MTurk, recruit workers there, and send them to Qualtrics. At the end of your survey at Qualtrics, you give them a code word to enter at MTurk. They go back to MTurk, enter the code word and submit. Then you pay them. And then you go to Qualtrics to download your data.
At Qualtrics, I think you can do small studies (several hundred respondents) on free accounts.
I find that on short surveys with few respondents, you can get your sample at MTurk paying 11 cents to 31 cents. If you are doing a survey with more than 1,000 respondents, paying 51 cents pretty much assures that you will get them. Paying less than 50 cents can lead to slow responses if the survey is longer or if you need more than 1,000 respondents.
The advantages of MTurk are speed, administration, cost, and pretty good data quality, esp. compared to using students or convenience samples among friends. The sample skews as somewhat better educated and younger, but it has a substantial number of ethnic minorities. So it’s probably worse than Rasmussen, definitely worse than Pew, and obviously much worse than the GSS. But compared to decades of research on psych students in college, it is miles better.
So instead of being scientifically illiterate, folks are apparently just plain old regular illiterate.
That’s comforting.
Astrology is not just horoscopes, and astrology is not a totally unscientific field, says this scientist.
Few would dispute that astrology has an end result that is not scientific, assuming they fully understood “scientific”. That the position of celestial bodies affects humans to the extent described in horoscopes is provably false by the standard scientific method of falsifiable prediction. But there is much science in the means taken to get there. Before an astrologer in ancient times warned of the obvious stupidity of financial ventures when the moon is in Aquarius (pseudo-science), the traditional astrologer would also determine precisely when that would be (science).
Kepler published astrological calendars that forecast planetary positions, the weather, and political events. The first of these was completely scientific according to Tycho Brahe’s data, and Kepler’s own laws of planetary motion. The second is performed in a scientific manner today, and the third was achieved by Kepler keeping abreast of political intrigue, which I consider unscientific (sorry PolySci people). So despite my consideration that astrology does more harm than good (to both gullible impressionable people AND the “sky sciences”, those familiar with the normal use of the term astrology and its history might be aware that astrology does have some scientific aspects. “Astrology is not scientific at all” is actually incorrect (in this ESA rocket scientist’s opinion). Astrology was initially developed throughout the world as a means of identifying the correct planting and harvesting seasons by the movement of the constellations (star recognition by analogy to imaginative but recognizable shapes). So the pseudo-science of astrology had a completely scientific foundation.
In the seventh century, Isidore of Seville stated that astrology had two parts: one was scientific, describing the movements of the sun, the moon and the stars, while the other, making predictions, was theologically erroneous.
I agree entirely.
Staltz, George, Richard, and Christopher are correct in raising the issue of the definition of “scientific” as a wild card in this discussion. If “scientific” means ‘based on demonstrable, observable and replicable phenomena’, well of course most people who spend any time at all discussing astrology with their friends will hear about a ‘typical Taurus” or “she’s such a Virgo” — and will likely be able to cite examples of individuals who match the norms of their sun sign. This observation of supposed truths of astrology is not infrequent, and may thus be thought of as replicable – plus certainly demonstrable as we can all point to friends whose personalities fulfill astrological expections. Doesn’t that qualify as pretty scientific? These are observed phenomena, not rank unsubstantiated faith or beliefs. Neither survey took this into consideration.
Interesting analysis. maybe you’ve addressed this in the comments already but I haven’t read all of them. I think there is one thing you’ve overlooked. Yes, the magnitude of the belief in astrology may be over estimated, but since this is a longitudinal study, what’s more important is the change over time. Unless it can be demonstrated that people’s misunderstanding of the terms astrology and astronomy was less common in years past (or for some reason changes over time), your assessment can’t explain the uptick in the belief in astrology. That, to me, seems like the more interesting question. That said, sharp thinking on this definition question though.
I would say that if you don’t know what you’re measuring with a question, changes over time are difficult if not impossible to interpret. Why do you think “changes in astrology belief” is a more plausible explanation than “changes in understanding of the term astrology”? That sounds like an assertion to me, and not something we have evidence for either way. The GSS doesn’t use a validated scale of astrology belief, so we really don’t know what these questions actually measure.
In my interactions with other commenters after posting this, it also became obvious that Americans do not have a consistent definition of “scientific” either. So changes over time in belief in astrology, in belief in science, in definitions of astrology, in definitions of science, or in definitions of “scientific” could all have led to changes over time. We just don’t know. At this point, I don’t think that these GSS questions are a valid way to draw conclusions about what Americans “believe”, in any regard.
What you’re saying is that there is noise in the data created by the ambiguity of the question. Fine. There is always noise. Is it great enough to swamp the variation in the results shown? If so, you would then expect to see an unpredictable trend in the historical data. But we don’t see that. Results are relatively consistent from 1985-2002 with a bump up in 2004 and a trend downwards since. For confusion to be the driver, then that confusion had to have remained consistent for 8 surveys, then decreased considerably (to cause the uptick in 2004 of those who said that astrology was not at all science). What would cause this confusion level to change so much? and what is causing it to increase again since (producing the downward decline since 2004)?
If this were a one time survey, or we only had a couple of historical surveys to work with, then I think your argument is grounds for dismissing the study. But with the historical data we have, assuming consistency of methodology, that noise can be accounted for.
This confusion of terms is indeed a factor, but I argue its a subset of the cause speculated to in the report – science education may be increasingly insufficient. If so, that not only explains the increase in number of people who know what astrology is and still think it is scientific, but so too the number of people confused by the terms in the first place. and that is really the take home message here, there simply aren’t enough people like us who can understand, let alone discuss, this kind of an issue.
Your explanation may very well be correct. But we simply have no way to know given current data. Any construct tapped by this question that is correlated with time could be responsible for the over time variation we’re seeing, which is not all that extreme. As a result, it’s not reasonable to conclude science education is specifically the problem. Even if we were to assume the question is valid (and I don’t think it is), the variation we’re seeing over time could be a consequence of many other causal forces.
Here is one possibility. People tend to seek out information that confirms their opinion and use that information to strengthen their opinion – this is well-known phenomenon called confirmation bias. With access to the Internet, it is much easier to seek out other people with extreme opinions that match our own when our self-concept is threatened. So despite improving science education, a person that believes in astrology might be challenged in school, say to themselves “I bet my science teacher is wrong”, and find a whole community of astrologers online that support their viewpoint and discredit the science teacher. Not amount of “improved science education” will change such a person’s opinion. And not coincidentally, 2004 is right about when the Internet became a convenient source of (mis)information. And it’s only getting worse as the Internet continues to grow.
In this case, the differences we see between countries over time could be interpreted instead as cultural differences – cultures where people are taught there is virtue in blindly following authority figures are going to see a smaller effect because they are less likely to question their teachers, and not because their science education is any better.
This is the danger of making policy decisions based upon incomplete data. Maybe a resulting solution will fix the problem, or maybe it’ll just waste a lot of money. There’s just no way to know (yet).
Having had a whole year to muse on the ignorance of others, I am less troubled now by all but one of my concerns I had in February 2014. I probably overestimated the fraction of respondents who would confuse “astrology” with “astronomy”, as I didn’t know that the immediately preceding question strongly implied that “astrology” is related to “horoscopes”. I also hadn’t appreciated that a major component of the real data here is the trend of answers to this question over the years (spanning 33 years)
My original concern remains, however. The headline-grabbing distillation that “the majority of young people now believe that astrology is scientific” is not a conclusion that I can draw from the results. Or rather what the results that should have been obtained would indicate. It seems the NSF doesn’t “know” the answer themselves, because it isn’t offered as a possible response. I was not clear on that last point a year ago, so please allow me to try again.
I am tarring the conclusions that astrology draws from the effect of the position of the stars at one’s birth to human social traits decades later as utter nonsense – or “utter nonscience”, if you like. But as pointed out by others above, there was a scientific basis to the ancient founding of astrology. There is still a pure science (the maths) and natural science (celestial observation, planetary motion etc.) component to “astrology”. Please refer to my previous post here (Feb 22, 2014) for Kepler’s astrology day-job. Galileo and Newton were astrologists too.
Some aspects of astrology, such as deriving patterns from observable data are 100% scientific, whilst others (particularly drawing horoscopic conclusions for tomorrow, based on celestial arrangements decades earlier) are at the 0% mark. I thus claim that the first response “Astrology is not scientific at all” is incorrect. The complete lack of its ability to pass scientific experimental testing means that “Very scientific” is also wrong. Aside from those all-or-nothing answers, we are left with only one response:
“Astrology is sort of scientific” — the least incorrect of the constructive answers.
That would not be my short answer if allowed my own wording. I would prefer to say that Astrology had a scientific foundation in observing the sky, but it draws provably false conclusions that have no scientific basis whatsoever.
I would like to congratulate the youngsters in the random responding Class of 2014. Contrary to the conclusion drawn by the NSF themselves, the 2014 crop of 18-24 year-olds had the highest percentage of correct answers (44%) since the question was first posed, 35 years earlier.
I even found it mildly reassuring that within the lowest quartile for factual knowledge of science, the percentage who chose “very scientific” remained in single digits throughout the millennium (with the single exception of reaching 12% in 2012). during which time around 50% of that group (52%) went for “not at all scientific” (I presume that’s the NSF’s opinion of the correct answer).
The number of 18-24 year olds who responded that astrology is very scientific was only 14%. Considering my claim that astrology is neither completely devoid of science nor fully embraces it, THAT number, 14% is how many youngsters “believe” in this pseudo-science. And 14% is hardly “the majority” as the National Science Foundation’sown summary called it.
Wide fluctuations of the smaller percentages from one survey to the next, probably say more about the science of small samples in statistics than about the scientific content of astrology.
I heard an American TV ‘Talk Show” host say that a survey of people in New York City showed that 44% of them had no grasp of statistical percentages. “Forty-four percent”, he reiterated. “That’s nearly one in three”.
Correction. I referred to results in my previous post just now as the 2014 results. They are from 2012.
http://www.nsf.gov/statistics/seind14/content/chapter-7/at07-13.xls
Go sample millennials