One of my colleagues today gave her class a fairly simple task: pick 5 people that you would assign to go on a mission to explore a new planet. I’ll tell you my list: geologist, engineer, astrobiologist, medical personnel, chemist. It’s not the only list people could come up with. I’m sure everyone reading this would come up with a slightly different list, but I’m sure there would be similarities: you’d pick people who could study the planet.
Want to know what the kids came up with? (keep in mind that these are sophomores and juniors in high school)
Doctor, lawyer, police man, teacher, construction person
Politician, doctor, farmer, army person, cook.
Builder, leader, doctor, teacher, business man
These are just a few examples of the lists, but I think it shows the point. There is not a single scientist among them. Not only that, some of the choices are pretty stupid. Who the heck needs a business man with exploring a new planet?
Maybe the kids were confused on the assignment. Maybe they thought their teacher was asking about colonization instead of exploration. But still, most of the groups had no explanation for why they made their choices. When asked why I would pick a geologist, I could explain that a geologist would be able to understand the composition of the planet. Understanding the composition allows future colonies to know where to planet things, where to mine things, where the ground is stable enough to build things. But these kids could not tell their teacher why they picked a police man to go explore a new planet.
Their lack of ability to explain their decisions shows an amazing lack of thinking things out logically and making decisions based on background knowledge and evidence. And these are kids are the future.
Are you scared yet?
On the plus side, I had an amazing (although overly long as it was the first day of school and I had stuff to do) conversation about starting a class that would be all about getting kids to think critically.
So one of my (many) goals for the semester is to research content and curriculum ideas for a critical thinking class at the high school level. Any ideas on where to start would be appreciated.
NASA recently launched an initiative to capture an asteroid to study and possibly even mine for usable commodities. The technology—still in the speculative stage—would use a solar-powered robot to basically bag-up a small asteroid (around 20 feet in diameter) and move it into a stable orbit around the moon.
Among the experts called to review and voice opinions about the initiative was Dr. Jim Bell, an astronomer and planetary scientist at ASU’s School of Earth and Space Exploration. He voices skepticism about the project, citing nonexistent technology, unknown costs, and the rarity of suitable candidates for capture. This last point is significant; of the 10,000 known asteroids orbiting our sun, only about 370 are small enough to be eligible. Of these, only 14 are in acceptable orbits. In light of these difficulties, Dr. Bell advises “a mix of skepticism and excitement” instead of wild-eyed optimism.
This is how NASA describes the mission:
The asteroid grand challenge complements NASA’s mission to find and capture a near-Earth asteroid, redirect it to a stable lunar orbit and send humans to study it. The asteroid redirect mission is included in President Obama’s fiscal year 2014 budget request for NASA, and leverages the agency’s progress on its Space Launch System rocket, Orion spacecraft and cutting-edge technology development. The mission is one step in NASA’s plan to send humans to Mars in the 2030s.
This weekend was The Amaz!ng Meeting! If you haven’t been, it is an “amazingly” fun gathering of critical thinkers with tons of excellent talks on skeptical, scientific, and thought-provoking topics. I wasn’t able to attend this year, but lived somewhat vicariously through the Facebook posts of a friend and fellow skeptic. The post below inspired this blog entry.
Although I have not seen Harriet Hall speak on this topic, I presume she intelligently articulated her reasons for this stance as she has in her writing (e.g. here and here). I thought I’d take a moment to explain the math behind screening tests in support of the argument that it is risky to over screen for disease.
In classical measurement theory, sensitivity and specificity are statistical measures regarding the outcome of any binary (i.e. two-outcome) test. Let’s consider medical tests in terms of a positive result (diagnosis of disease/condition) and a negative result (free of the disease/condition). Sensitivity is a measure of the “true positive rate” and is the proportion of positive results that are correct positive results. Specificity is a measure of the “true negative rate” and is the proportion of negative results that are correct negative results. Consider a test for pregnancy. There are four possible results as shown in the table below: a. pregnant woman gets the correct positive result, b. non-pregnant woman gets an incorrect positive result, c. pregnant woman gets an incorrect negative result, and d. non-pregnant woman gets a correct negative result. The sensitivity of a pregnancy test is the percentage of time a woman who IS actually pregnant will get the correct positive pregnancy result. The specificity of a pregnancy test is the percentage of time a woman who IS NOT actually pregnant will get the correct negative result.
We calculate sensitivity as the proportion of people who correctly get positive test results. If each cell represents the number of people in the sample with the corresponding test result and actual condition status, then Sensitivity = A/(A+C). Notice that sensitivity only incorporates data from people who actually have the condition.
We calculate specificity as the proportion of people who correctly get the negative test results. When each cell represents the count of people from the sample with a particular test result and condition status, Specificity = D/(B+D). Notice specificity only incorporates data from people who don’t have the condition.
Ideally, we want all medical tests to have 100% sensitivity and specificity, but the reality is that most medical tests do not have that level of precision. However, tests need to be developed balancing both sensitivity and specificity as an increase in one often means a decrease in the other. For example, consider an HIV diagnosis. Sensitivity is particular important as you don’t want to miss diagnosing someone with HIV; you want to help further prevent spreading of the disease and to start the patient’s treatment promptly. On the other hand, if the test gives too many false positives, it would cause a great deal of unnecessary emotional distress. That brings us to the math! If a test has 99% sensitivity, then for every 100 people with the disease, 99 would be correctly diagnosed and 1 would have false negative and a missed diagnosis. If the same test has 99% specificity, then for every 100 HIV-negative people, 99 would correctly get a negative result and one person would get a false positive. Both balancing and maximizing sensitivity and specificity are considerations in the design of diagnostic tests.
Base Rate Fallacy
The so-called “Base Rate Fallacy” comes into play in terms of medical diagnoses. Broadly, the base rate fallacy is when a person makes a judgment of the overall likelihood of an event based on easily accessible knowledge (here: values of sensitivity and specificity) without taking into consideration the prevalence or base-rate of the event. Here, we make the base rate fallacy when take the results of the test and fail to incorporate the actual rate of disease/condition. Rate of Disease = (A + C)/(A + B + C + D)
Let’s consider a fictitious, deadly disease, Disease X. Disease X is very rare and only afflicts .01% of the population. Your doctor has a cheap and very accurate test: it has both 99.5% sensitivity and specificity. Because the test is inexpensive, he or she has decided to administer this test to all of his/her patients (should this scenario be true, it would be time to find another doctor). You think, “This is great! I’d love to know for sure that I don’t have Disease X.” The doctor does the test and it comes back positive. You panic, hire a lawyer to write your will, and start saying your goodbyes because death is surely imminent. After all, the test has 99.5% specificity, doesn’t that mean that you most likely have the disease (a 99.5% chance!). But is death imminent? What is the probability you actually have this disease? After the initial panic subsides, you take a step back and think about the actual probability that you have the disease and discover you made the “base rate fallacy.”
Let’s walk through this. We need to calculate the probability that you have the disease given the positive results of the test. Conventional notation expresses the probability of a disease given a positive test result as P(Disease|Positive). Recall that sensitivity is the probability that a person gets a positive test result given they have the disease. This can be expressed as P(Positive|Disease). The base rate fallacy is the assumption that P(Disease|Positive) =P(Positive|Disease). However, the actual probability of the disease given a positive diagnosis based on the disease’s low base rate (assuming no other risk factors or variables are known) can be calculated. NOTE: This is about to get very “mathy”. If math isn’t your cup of tea, just scroll down for the results. The results illustrate the point and it is not crucial you understand the math.
Using Bayes’ Theorem, the probability of the disease given a positive diagnosis is expressed below.
The probability of a positive diagnosis can be expressed using the rules of probability and the equation above is equivalent to this equation.
Although the Equation above may seem a bit intimidating, we know the values to plug in:
P(Positive|Disease) = Specificity = 99.5% = .995
P(Disease) = Base Rate = .01% = .0001
P(Positive|NoDisease) = 1 – Specificity = .005
P(NoDisease) = 1 – P(Disease) = .9999
Plugging these values into the equation, we get:
MATH IS OVER – HERE ARE THE RESULTS.
Based on the math, given a 99.5% specificity and a .01% base rate, a person with a positive test result has less than a 2% chance of having the disease!!! And just a slight decrease in specificity, the rate goes down further (for 99% specificity there is less than a 1% chance of actually having the disease given a base rate of .01%). Although no one wants to hear that they have 2% chance of having a deadly condition, this is certainly better than the original panic over the 99.5% accuracy test! Given that low base rate diseases can have high rates of false positives, it is prudent for doctors not to over test and cause unnecessary distress.
Although the example I’ve given for the base rate fallacy is in terms of medical diagnostic testing, the base rate fallacy is broadly applicable to many types of scenarios. For example, consider the low rate of terrorists in the US and thus the resulting accuracy of any sort of screening procedures – there will be many false positives. Psychologists have also studied this phenomenon and found that humans tend to fall prey to the base rate fallacy in their decisions and judgments (e.g. Kahneman & Tversky, 1985). Anytime you make judgement about the probability an event based on information at hand while ignoring the base rate of the event, you are making the base rate fallacy. In conclusion, I hope that you will 1) be prudent about interpreting the results of any medical tests and ask questions regarding sensitivity, specificity, and rate of disease and 2) be on a keen lookout for ways the base rate fallacy may influence your own decision making or be used to manipulate you in advertising and the media.