To improve your evaluation skills, develop three abilities:
- Maintain a skeptical mind-set.
- Learn which sources are most trustworthy.
- Learn to identify reporting errors and inconsistencies. If something does not look right, investigate its veracity.
In addition, the skeptical investigator must develop a structured means of evaluating the relevance and reliability of the collected data along with the ability to communicate this. In my opinion, this is the most demanding part of any investigation.
The following list of 13 evaluation criteria has developed over decades of practice by researchers and investigators worldwide. This list of evaluation criteria appears in one form or another throughout the literature on research and intelligence analysis. I use a Microsoft Excel spread sheet based upon them to record the evaluation of sources for most investigations. Doing this is time-consuming, but it is often necessary to maintain the integrity of the reported data and the conclusions that are drawn from it.
The Internet is renowned for harbouring unreliable information. The following evaluation matrix will work for you if you rigorously apply it.
- Recency.Do the data appear to be current on the subject or the most appropriate for the historical time period? Are sources dated, and maintained?
- Relevancy. Is there a direct correlation to the subject? What is the tone of the information display? Is it popular, scholarly, or technical?
- Authority. What is the reputation of the data, and the data-provider? Has this source of data been cited elsewhere? What is the reliability of the source? How can you document or qualify the source of the information?
- Completeness. Are alternative data or views cited? Are references provided? Given what you know about the source, is there any evidence that the data is NOT ‘slanted’?
- Accuracy. Does the source furnish background information about data sources and/or in-depth data? Are the complex issues of data integrity and validity oversimplified? Are the terms adequately defined? Are there references or sources of quotes?
- Clarity. Is the presentation of information really credible? Can bias of the information providers really be ruled out? Are there any logical fallacies in presentation of the data or assertions, or in other statements or publications on the page or by the source? Are key assumptions described, or are they hidden in the hope that the reader will be gullible?
- Verifiability. Can the information be verified? Can you find corroboration? If not, why not?
- Statistical validity. Can the key points or critical data be supported by standard statistical testing? With subjective information, one verifies by corroboration as it can be found. With numerical information, many questions arise. What statistical inference is needed in order to accept any implied inferences of the data displayed? Are there clear explanations for readers or viewers to qualify the implications of numerical “averages” or “percentages?”
- Internal consistency. Do the data or commentary contain internal contradictions? Know what you can about the source, and scan for logical fallacies throughout the presentation.
- External consistency. Do the data reflect any contradictions among the source documents? In the assertion of information or views, is there an acknowledgment or recognition of alternative views or sources? If not, and source documents are involved, one might suspect that the author had an “agenda”.
- Context. Can fact be distinguished from opinion? Are sources taken, or used out of context?
- Comparative quality. Are some data clearly inferior to other data? Which are the “best” data when you consider the above eleven tests ( i.e., most recent, most relevant, the most authoritative, the most complete, the most accurate and so forth). You should always be evaluating the information as you browse the Net. Check the little things that journalists watch for. A misspelled name, for example, could be a warning sign, even in an academic paper, that the author was careless in other areas as well. Do any statements seem exaggerated? If so, why has the author exaggerated? Is it a spur-of-themoment statement by email, or is that exaggeration more deliberate? Are you reading instant analysis, quick off the mark, or the results of a carefully crafted study, or a metaanalysis, which is a study of all previous studies on a subject? What do you think has been left out of the report? What an author omits may be just as important to a researcher as what an author includes. What’s left out could reveal much about the bias of the information you are reading. Take notes on such matters as you find sources, and possibly include footnotes in your paper, if that’s what you’re doing; it’ll provide evidence that you have critical thinking abilities! Don’t ignore bias as a valuable source of information, as well: Even an untrustworthy source is valuable for what it reveals about the personality of an author, especially if he or she is an actor in the events.
- Problems.There’s another problem, growing in the 1990s, called by some the Crossfire Syndrome, after Crossfire, the CNN public affairs show and its tabloid television imitators. The Crossfire Syndrome drowns out the moderate voices in favor of polarization and polemics. Confrontation between polar opposites may make for good television, but it often paints a distorted view of reality. On the Net, and throughout North American media culture, the Crossfire Syndrome is acute. In flame wars, the shouters on both sides are left to use up bandwidth while the moderate voices back off. Political correctness of any sort, right or left, religious or secular, tends to distort material while creating interest, at the cost of ‘truth’. All of us are left with the task, with nearly any information we’re exposed to, of seeking out the facts behind the bias. It’s been said many times that there is no such thing as objectivity. The honest researcher tries to be fair. The best researcher is both prosecutor and defense attorney, searching out the facts on all sides (there are often more than two).
The advent of the 24-hour all-news network, along with a procession of imitators and the Web. The CNN Effect lives off of one continually asked question, ‘how bad can it get?’ as a way to maintain viewer interest. This severely distorts the information presented.
If you want to test what you have learned here is an example.
You are researching how IT operations impact carbon dioxide emissions. In the data you uncover, you find the following journal article: Towards Green Communıcatıons. What would your evaluation of this article be?