![\"Data](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/data-fallcies-poster-preview.jpeg\")
<\/p>\nData fallacies<\/p>\n
Statistical fallacies are common tricks data can play on you, which lead to mistakes in data interpretation and analysis. Explore some common fallacies, with real-life examples, and find out how you can avoid them.<\/p>\n
<\/p>\n
Download poster<\/a><\/p>\n Cherry Picking<\/p>\n The practice of selecting results that fit your claim and excluding those that don\u2019t. The worst and most harmful example of being dishonest with data.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n When making a case, data adds weight \u2013 whether a study, experiment or something you\u2019ve read. However, people often only highlight data that backs their case, rather than the entire body of results. It\u2019s prevalent in public debate and politics where two sides can both present data that backs their position. Cherry Picking can be deliberate or accidental. Commonly, when you\u2019re receiving data second hand, there\u2019s an opportunity for someone choosing what data to share to distort the truth to whatever opinion they\u2019re peddling. When on the receiving end of data, it\u2019s important to ask yourself: \u2018What am I not being told?\u2019.<\/p>\n Related Reading:<\/p>\n Data Dredging<\/p>\n Data dredging is the failure to acknowledge that the correlation was in fact the result of chance.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n Tests for statistical significance only work if you\u2019ve defined your hypothesis upfront. Historically, this has been a problem with clinical trials where researchers have \u2018data-dredged\u2019 their results and switched what they were testing for. It explains why so many results published in scientific journals have subsequently been proven to be wrong. To avoid this, it\u2019s now becoming standard practice to register clinical trials, stating in advance what your primary endpoint measure is.<\/p>\n Related reading:<\/p>\n Survivorship Bias<\/p>\n Drawing conclusions from an incomplete set of data, because that data has \u2018survived\u2019 some selection criteria.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n When analyzing data, it\u2019s important to ask yourself what data you don\u2019t have. Sometimes, the full picture is obscured because the data you\u2019ve got has survived a selection of some sort. For example, in WWII, a team was asked where the best place was to fit armour to a plane. The planes that came back from battle had bullet holes everywhere except the engine and cockpit. The team decided it was best to fit armour where there were no bullet holes, because planes shot in those places had not returned.<\/p>\n Related reading:<\/p>\n Cobra Effect<\/p>\n When an incentive produces the opposite result intended. Also known as a Perverse Incentive.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n Named from a historic legend, the Cobra Effect occurs when an incentive for solving a problem creates unintended negative consequences. It\u2019s said that in the 1800s, the British Empire wanted to reduce cobra bite deaths in India. They offered a financial incentive for every cobra skin brought to them to motivate cobra hunting. But instead, people began farming them. When the government realized the incentive wasn\u2019t working, they removed it so cobra farmers released their snakes, increasing the population. When setting incentives or goals, make sure you\u2019re not accidentally encouraging the wrong behaviour.<\/p>\n Related reading:<\/p>\n False Causality<\/p>\n To falsely assume when two events occur together that one must have caused the other.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n Global temperatures have steadily risen over the past 150 years and the number of pirates has declined at a comparable rate. No one would reasonably claim that the reduction in pirates caused global warming or that more pirates would reverse it. But it\u2019s not usually this clear-cut. Often correlations between two things tempt us to believe that one caused the other. However, it\u2019s often a coincidence or there\u2019s a third factor causing both effects that you\u2019re seeing. In our pirates and global warming example, the cause of both is industrialization. Never assume causation because of correlation alone \u2013 always gather more evidence.<\/p>\n Related reading:<\/p>\n Gerrymandering<\/p>\n The practice of deliberately manipulating boundaries of political districts in order to sway the result of an election.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n In many political systems, it\u2019s possible to manipulate the likelihood of one party being elected over another by redefining the political districts \u2013 include more rural areas in a district to disadvantage the party that\u2019s more popular in cities etc. A similar phenomenon known as the Modifiable Areal Unit Problem (MAUP) can occur when analyzing data. How you define the areas to aggregate your data \u2013 e.g. what you define as \u2018Northern counties\u2019 \u2013 can change the result. The scale used to group data can also have a big impact. Results can vary wildly whether using postcodes, counties or states.<\/p>\n Related reading:<\/p>\n Sampling Bias<\/p>\n Drawing conclusions from a set of data that isn\u2019t representative of the population you\u2019re trying to understand.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n A classic problem in election polling where people taking part in a poll aren\u2019t representative of the total population, either due to self-selection or bias from the analysts. One famous example occurred in 1948 when The Chicago Tribune mistakenly predicted, based on a phone survey, that Thomas E. Dewey would become the next US president. They hadn\u2019t considered that only a certain demographic could afford telephones, excluding entire segments of the population from their survey. Make sure to consider whether your research participants are truly representative and not subject to some sampling bias.<\/p>\n Related reading:<\/p>\n Gambler’s Fallacy<\/p>\n The mistaken belief that because something has happened more frequently than usual, it\u2019s now less likely to happen in future and vice versa.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n This is also known as the Monte Carlo Fallacy because of an infamous example that occurred at a roulette table there in 1913. The ball fell in black 26 times in a row and gamblers lost millions betting against black, assuming the streak had to end. However, the chance of black is always the same as red regardless of what\u2019s happened in the past, because the underlying probability is unchanged. A roulette table has no memory. When tempted by this fallacy, remind yourself that there\u2019s no rectifying force in the universe acting to \u2018balance things out\u2019!<\/p>\n Related reading:<\/p>\n Regression Toward the Mean<\/p>\n When something happens that\u2019s unusually good or bad, over time it will revert back towards the average.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n Anywhere that random chance plays a part in the outcome, you\u2019re likely to see regression toward the mean. For example, success in business is often a combination of both skill and luck. This means that the best performing companies today are likely to be much closer to average in 10 years time, not through incompetence but because today they\u2019re likely benefitting from a string of good luck \u2013 like rolling a double-six repeatedly.<\/p>\n Related reading:<\/p>\n Hawthorne Effect<\/p>\n When the act of monitoring someone can affect that person\u2019s behavior. Also known as the Observer Effect.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n In the 1920s at Hawthorne Works, an Illinois factory, a social sciences experiment hypothesised that workers would become more productive following various changes to their environment such as working hours, lighting levels and break times. However, it turned out that what actually motivated the workers\u2019 productivity was someone taking an interest in them. When using human research subjects, it\u2019s important to analyze the resulting data with consideration for the Hawthorne Effect.<\/p>\n Related reading:<\/p>\n Simpson’s Paradox<\/p>\n A phenomenon in which a trend appears in different groups of data but disappears or reverses when the groups are combined.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n In the 1970s, Berkeley University was accused of sexism because female applicants were less likely to be accepted than male ones. However, when trying to identify the source of the problem, they found that for individual subjects the acceptance rates were generally better for women than men. The paradox was caused by a difference in what subjects men and women were applying for. A greater proportion of the female applicants were applying to highly competitive subjects where acceptance rates were much lower for both genders.<\/p>\n Related reading:<\/p>\n McNamara Fallacy<\/p>\n Relying solely on metrics in complex situations can cause you to lose sight of the bigger picture.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n Named after Robert McNamara, the U.S. Secretary of Defense (1961-1968), who believed truth could only be found in data and statistical rigor. The fallacy refers to his approach of taking enemy body count as the measure of success in the Vietnam War. Obsessing over it meant that other relevant insights like the shifting mood of the U.S. public and the feelings of the Vietnamese people were largely ignored. When analyzing complex phenomena, we\u2019re often forced to use a metric as proxy for success. However, dogmatically optimizing for this number and ignoring all other information is risky.<\/p>\n Related reading:<\/p>\n Overfitting<\/p>\n A more complex explanation will often describe your data better than a simple one. However, a simpler explanation is usually more representative of the underlying relationship.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n When looking at data, you\u2019ll want to understand what the underlying relationships are. To do this, you create a model that describes them mathematically. The problem is that a more complex model will fit your initial data better than a simple one. However, they tend to be very brittle: They work well for the data you already have, but try too hard to explain random variations. Therefore, as soon as you add more data, they break down. Simpler models are usually more robust and better at predicting future trends<\/p>\n Related reading:<\/p>\n Publication Bias<\/p>\n How interesting a research finding is affects how likely it is to be published, distorting our impression of reality.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n For every study that shows statistically significant results, there may have been many similar tests that were inconclusive. However, significant results are more interesting to read about and are therefore more likely to get published. Not knowing how many \u2018boring\u2019 studies were filed away impacts our ability to judge the validity of the results we read about. When a company claims a certain activity had a major positive impact on growth, other companies may have tried the same thing without success, so they don\u2019t talk about it.<\/p>\n Related reading:<\/p>\n Danger of Summary Metrics<\/p>\n It can be misleading to only look at the summary metrics of data sets.<\/p>\n Get the printable card<\/strong><\/a><\/p>\n To demonstrate the effect, statistician Francis Anscombe put together four example data sets in the 1970s. Known as Anscombe\u2019s Quartet, each data set has the same mean, variance and correlation. However, when graphed, it\u2019s clear that each of the data sets are totally different. The point that Anscombe wanted to make is that the shape of the data is as important as the summary metrics and cannot be ignored in analysis.<\/p>\n Related reading:<\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":" Data fallacies Statistical fallacies are common tricks data can play on you, which lead to mistakes in data interpretation and analysis. Explore some common fallacies, with real-life examples, and find out how you can avoid them. Download poster Cherry Picking The practice of selecting results that fit your claim and excluding those that don\u2019t. The … Continue reading Data fallacies<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":2379,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25],"tags":[32,26,66,58,52,51,59],"class_list":["post-2285","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bookmarked-articles","tag-analytic","tag-data","tag-data-fallacies","tag-data-modelling","tag-machine-learning","tag-modelling","tag-statistic"],"_links":{"self":[{"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/posts\/2285","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mintea.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2285"}],"version-history":[{"count":2,"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/posts\/2285\/revisions"}],"predecessor-version":[{"id":2303,"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/posts\/2285\/revisions\/2303"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mintea.blog\/index.php?rest_route=\/wp\/v2\/media\/2379"}],"wp:attachment":[{"href":"https:\/\/mintea.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2285"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mintea.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2285"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mintea.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2285"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"Cherry](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/cherry-picking.png\")
<\/p>\n\n
![\"Data](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/data-dredging.png\")
<\/p>\n\n
![\"Survivorship](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/survivorship-bias.png\")
<\/p>\n\n
![\"Cobra](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/cobra-effect.png\")
<\/p>\n\n
![\"False](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/false-causality.png\")
<\/p>\n\n
![\"Gerrymandering\"](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/gerrymandering.png\")
<\/p>\n\n
![\"Sampling](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/sampling-bias.png\")
<\/p>\n\n
![\"Gambler's](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/gamblers-fallacy.png\")
<\/p>\n\n
![\"Regression](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/regression-toward-the-mean.png\")
<\/p>\n\n
![\"Hawthorne](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/hawthorne-effect.png\")
<\/p>\n\n
![\"Simpson's](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/simpsons-paradox.png\")
<\/p>\n\n
![\"McNamara](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/mcnamara-fallacy.png\")
<\/p>\n\n
![\"Overfitting\"](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/overfitting.png\")
<\/p>\n\n
![\"Publication](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/publication-bias.png\")
<\/p>\n\n
![\"Danger](\"https:\/\/mintea.blog\/wp-content\/uploads\/2022\/09\/danger-of-summary-metrics.png\")
<\/p>\n\n