View on GitHub


How the Scientific Establishment Generates Bad Science, and What We Can Do About It

Understanding prior work

Example 1: In the 1980s, a popular technique in education was targetting students’ learning styles. Educators believed that different students learned better if information were presented in an auditory, visual, or kinesthetic way. However, over the following decades, there was no evidence for learning styles. Eventually, several meta-studies came out which debunked the concept of learning styles. However, papers describing technologies to support different learning styles continue to come out, oblivious to such work, and generally repeating methodological errors in prior work.

Example 2: In MOOCs (the space where I operate) a significant number of papers – more than 10 percent – begin with one of two faulty assumptions:

Many such papers are simply incorrect. Many attempt to improve on drop-out by adding interventions late in the course. By that point, most of the alleged “drop-out” has happened (users never logging in, or only logging in for a few minutes). Many papers also introduce interventions which are either already used in state-of-the-art MOOCs, or less effective than techniques used in state-of-the-art MOOCs.

Peer reviewers and authors from general ed-tech conferences do not have enough background to catch such errors; there are no incentives for them to check prior work. For a while, a few people from Harvard played whack-a-mole with such work. This approach did not work – rebuttals were rarely read, and misconceptions continued to propagate from paper to paper.

These examples are the tip of the iceberg. Many MOOC papers unknowningly replicate prior results in education research, or incorrectly contradict based on well-known methodological errors.

For the most part, researchers who ignore prior work can:

In a competitive landscape, such researchers tend to win.

For your presentation: