The h-index ([2]) and impact factor of journals. In general, I try to stick to an IF > 2 though this is not canon. These can usually be found with a simple Google search.
The evidence hierarchy — to help me see the forest from the trees. If I drop too far below the apex, I can make 1=2.
The plural of anecdote is not evidence and correlation does not equal causation.
Publication bias — by searching for unpublished trials, displaying funnel plots, and using statistics like fail-safe N to investigate the possibility of suppressed research. A small fail-safe N or asymmetric funnel plot suggest bias.
Conflicts of interest, not just in declarations at the end of the article but by looking up the authors' online resumés, research histories and paid lectures. NIH funded studies are preferred but can still have serious issues. Money, ego and prestige are insidious. It is difficult to convince a man of something if his salary depends on him believing the opposite.
Retraction Watch publishes a list of scientists with the most retracted papers, either due to p-hacking, poor statistical methods or even actively fabricating data. You can access this list here: [3]
Journal lists:
The Abridged Index Medicus — a list of 114 journals that are generally gold standard. Another is the 2003 Brandon/Hill list which includes 141 journals, though it is no longer maintained.
Beall's list — a compilation of problematic journals, discussed comprehensively here: [4] It has not been updated in some time and there are limitations but still a phenomenal open-source candle in the dark. Be cautious of hijacked and vanity "journals". MDPI, Frontiers and Hindawi are some of the more frequent offenders.
CiteWatch — Wikipedia's homage to Beall; an excellent resource that is updated twice monthly.
Cabells' Predatory Reports — the successor to Beall's; a comprehensive multidisciplinary update. Unfortunately provided by a paid subscription service only available to institutions, not individual researchers - [5]
I usually start out by looking at diagrams/tables and carefully reading the captions because pictures are easier for my reptile brain to digest. Looking at p intervals and sample sizes give me some sense of an idea's sincerity. This alone lands me light years ahead of where I would have been just reading the abstract. It can be overwhelming at first, but gets easier with practice.
Bayesian analyses > frequentist inferences. The former is a deductive probability, the latter inductive and binary. Combined Bayesian + frequentist analyses are better than either individually, with the truth often living where they meet.
Overadjustment bias for conclusions that emerge or disappear only after correction for confounding variables. There could be a causal path. Cox proportional hazards models, in particular, are susceptible.
As an example: incorrect adjustment for blood pressure while studying the relationship between obesity and kidney failure. Obesity causes high blood pressure, which is its mechanism for destroying your kidneys. Correcting for hypertension obscures the mechanism and causes a Type II error. This method can also be inverted to cause Type I errors. Such mistakes induce bias instead of preventing it.
Cox models also try to force data into linearity and falter with J- or U-shaped correlations.
Distribution of p-values in meta-analyses to distinguish Monte Carlo type approaches from p-hacking.
For controversial or mainstream topics, I take a peak at the talk page and edit history to see if the issue I'm about to revise has been explored, especially if it's in a subject I don't usually edit.
If an article I want to read is behind a paywall, sometimes I try e-mailing the author a kind note to ask for a copy. This usually works, especially if I pack in a compliment or two. Researchers are like plants; they flourish with attention.
Images need to be CC BY or CC BY SA. For NC and ND licensed images you can upload them to NC Commons.
All heuristics are equal, but availability is more equal than others.
The One begets the Two. The Two begets the Three, and the Three begets the 10,000 things.
In a series of studies in 2005 and 2006, researchers at the University of Michigan found that when misinformed people, particularly political partisans, were exposed to corrected facts in news stories, they rarely changed their minds. In fact, they often became even more strongly set in their beliefs. Facts, they found, were not curing misinformation. Like an underpowered antibiotic, facts could actually make misinformation even stronger.
Arguing with an idiot is like playing chess with a pigeon. It's just going to knock the pieces over, shit on the board, and then strut around like it won.
People would rather believe a simple lie than the complex truth.
The popularity of a scale rarely equates to its validity.
True humility is not thinking less of yourself. It is thinking of your self less.
I never gave away anything without wishing I had kept it; nor kept it without wishing I had given it away.
When once a man is launched on an adventure as this, he must bid farewell to hopes and fears, otherwise death or deliverance will both come too late to save his honour and his reason!
In this world Ellwood, you must be oh so smart, or oh so pleasant. Well for years I was smart; I recommend pleasant. And you may quote me.
Frank Sinatra saved my life once. He said, "Okay, boys. That's enough."
If you want to go fast, go alone. If you want to go far, go together.
