Evaluating Research Credibility: 3 Misconceptions

Writing a background section or a review requires combing the vast online literature to identify the most relevant prior work. However, beyond relevance, sources must be credible, which is often assessed based on not only high-quality research design but also reputation in the field. The standings of the journal, the lead author, and the article itself are frequently the three main considerations.

Evaluating the credibility of a researcher and his or her research in this way seems straightforward: the journal's name and the lead author's name and affiliation are on the title page of the article, and the article's citation count is easily accessible online (such as via Google Scholar). Even if you are a young researcher, you likely already have a sense of which journals, senior researchers, and institutions are considered reputable in your field, and you know that higher citation numbers are considered best.

These quick gauges are particularly useful when you are hurrying to submit a manuscript or grant proposal or are new to a field. Judging the reliability of a researcher and his or her work based on these three features is also far easier than scrutinizing sample size, controls, logic, and other detailed aspects of study design.

However, there are some flaws in this approach:

1. A journal's reputation is not always an indicator of credibility.

Just because a journal is prominent in your field, highly selective, and/or peer reviewed does not guarantee that its published work is credible. Affiliation with a specific publisher, society, or institution is also not a definite indicator of reliability. Even non-predatory, “good” journals may misjudge credibility, as indicated by spoof paper acceptance and retraction notices; in fact, journals with high impact factors have been found to have higher retraction frequencies. Meanwhile, good-quality research may be published in a lesser-known journal, such as a regional journal, a specialized journal, or a new publication, due to the work's niche significance or the author's time or language constraints.

2. An author's reputation and his or her institutional affiliation are not always indicators of credibility.

An author's seniority, reputation, publication record, education, affiliation with a prominent research institution, and/or training by a prominent researcher may belie sloppiness or bias. This may especially be due to senior investigators' limited oversight over the day-to-day research in their laboratories. Meanwhile, a young researcher, a newcomer to the field, or a researcher from a small or lesser-known institution (subscription required) may present fresh, valid ideas.

3. An article's online citation count is not always an indicator of credibility.

Of course, citation numbers can be inflated or deflated by the two common misconceptions above.

Unfortunately, these limitations to relying on the renown of journals, authors, and articles to determine credibility may lead to both false positives and false negatives; in other words, you may ultimately cite questionable studies and omit credible ones. The former error may undermine the credibility of your own study if you are not explicitly critiquing that work and may also perpetuate inaccuracies in understanding and subsequent research, while the latter error could weaken your study by causing you to overlook pertinent findings that might either support or refute your work.

It may now seem that assessing an author's and an article's credibility is insurmountable. Although you can never be certain of validity, stepping beyond the above gauges may be helpful to bolster your confidence.

In particular, in addition to assessing research design, information outside of the paper itself may be useful, although this requires a bit more online research:

• Is the work still relevant? (Are there any retraction notices or errata listed for the article? In what context has this paper been cited?)
• Are the references in the paper credible and properly interpreted?
• Is the reference list comprehensive?
• How do the post-publication peer reviews, if any, evaluate the quality of the study?

Of course, you may not have the time or resources to research all of these aspects, but even delving into just one or two could ultimately enhance both your understanding of the literature and your own credibility.