I used to do a series of lectures about the skills I found most lacking in the education of detectives. The lecture about evaluating the revealed wisdom that pours forth from the Internet was always fun to deliver.
One example that I used when I started doing these, was a site that identified the second gunman in the Kennedy assassination — there was Elvis holding a Thompson sub-machine gun on the grassy knoll. It was on the Internet so it must be true.
Here is the 13 point check list for evaluating information upon which I based the lecture.
Evaluation Check List
1. Recency. Do the data appear to be current on the subject or the most appropriate for the historical time period? Are sources dated, and maintained?
2. Relevancy. Is there a direct correlation to the subject? What is the tone of the information display? Is it popular, scholarly, or technical?
3. Authority. What is the reputation of the data, and the data-provider? Has this source of data been cited elsewhere? What is the reliability of the source? How can you document or qualify the source of the information?
4. Completeness. Are alternative data or views cited? Are references provided? Given what you know about the source, is there any evidence that the data is NOT ‘slanted’?
5. Accuracy. Does the source furnish background information about data sources and/or indepth data? Are the complex issues of data integrity and validity oversimplified? Are the terms adequately defined? Are there references or sources of quotes?
6. Clarity. Is the presentation of information really credible? Can bias of the information providers really be ruled out? Are there any logical fallacies in presentation of the data or assertions, or in other statements or publications on the page or by the source? Are key assumptions described, or are they hidden in the hope that the reader will be gullible?
7. Verifiability. Can the information be verified? Can you find corroboration? If not, why not?
8. Statistical validity. Can the key points or critical data be supported by standard statistical testing? With subjective information, one verifies by corroboration as it can be found. With numerical information, many questions arise. What statistical inference is needed in order to accept any implied inferences of the data displayed? Are there clear explanations for readers or viewers to qualify the implications of numerical “averages” or “percentages?”
9. Internal consistency. Do the data or commentary contain internal contradictions? Know what you can about the source, and scan for logical fallacies throughout the presentation.
10. External consistency. Do the data reflect any contradictions among the source documents? In the assertion of information or views, is there an acknowledgment or recognition of alternative views or sources? If not, and source documents are involved, one might suspect that the author had an “agenda”.
11. Context. Can fact be distinguished from opinion? Are sources taken, or used out of context?
12. Comparative quality. Are some data clearly inferior to other data? Which are the “best” data when you consider the above eleven tests ( i.e., most recent, most relevant, the most authoritative, the most complete, the most accurate and so forth). You should always be evaluating the information as you browse the Net. Check the little things that journalists watch for. A misspelled name, for example, could be a warning sign, even in an academic paper, that the author was careless in other areas as well. Do any statements seem exaggerated? If so, why has the author exaggerated? Is it a spur-of-themoment statement by email, or is that exaggeration more deliberate? Are you reading instant analysis, quick off the mark, or the results of a carefully crafted study, or a metaanalysis, which is a study of all previous studies on a subject? What do you think has been left out of the report? What an author omits may be just as important to a researcher as what an author includes. What’s left out could reveal much about the bias of the information you are reading. Take notes on such matters as you find sources, and possibly include footnotes in your paper, if that’s what you’re doing; it’ll provide evidence that you have critical thinking abilities! Don’t ignore bias as a valuable source of information, as well: Even an untrustworthy source is valuable for what it reveals about the personality of an author, especially if he or she is an actor in the events.
13. Problems. There’s another problem, growing in the 1990s, called by some the Crossfire Syndrome, after Crossfire, the CNN public affairs show and its tabloid television imitators. The Crossfire Syndrome drowns out the moderate voices in favor of polarization and polemics. Confrontation between polar opposites may make for good television, but it often paints a distorted view of reality. On the Net, and throughout North American media culture, the Crossfire Syndrome is acute. In flame wars, the shouters on both sides are left to use up bandwidth while the moderate voices back off. Political correctness of any sort, right or left, religious or secular, tends to distort material while creating interest, at the cost of ‘truth’. All of us are left with the task, with nearly any information we’re exposed to, of seeking out the facts behind the bias. It’s been said many times that there is no such thing as objectivity. The honest researcher tries to be fair. The best researcher is both prosecutor and defense attorney, searching out the facts on all sides (there are often more than two).
The advent of the 24-hour all-news network, along with a procession of imitators and the Web, have all come into being since the last major recession. The CNN Effect lives off of one continually asked question, ‘how bad can it get?’ as a way to maintain viewer interest. This severely distorts the information presented. No wonder so many people think we are headed for a depression even though the numbers are similar to 1975 and 1982. This is hardly the Great Depression, but one would never know it to listen the talking heads.