Understanding prior work
Example 1: In the 1980s, a popular technique in education was targetting students’ learning styles. Educators believed that different students learned better if information were presented in an auditory, visual, or kinesthetic way. However, over the following decades, there was no evidence for learning styles. Eventually, several meta-studies came out which debunked the concept of learning styles. However, papers describing technologies to support different learning styles continue to come out, oblivious to such work, and generally repeating methodological errors in prior work.
Example 2: In MOOCs (the space where I operate) a significant number of papers – more than 10 percent – begin with one of two faulty assumptions:
- In the early days of MOOCs, there was a mischaracterization of the drop-out rate in MOOCs due to some poor press management.
- State-of-the-art in MOOCs is hour-long videos and multiple choice questions (which was often the state-of-the-practice, due to failure to scale best practices, but was never the state-of-the-art).
Many such papers are simply incorrect. Many attempt to improve on drop-out by adding interventions late in the course. By that point, most of the alleged “drop-out” has happened (users never logging in, or only logging in for a few minutes). Many papers also introduce interventions which are either already used in state-of-the-art MOOCs, or less effective than techniques used in state-of-the-art MOOCs.
Peer reviewers and authors from general ed-tech conferences do not have enough background to catch such errors; there are no incentives for them to check prior work. For a while, a few people from Harvard played whack-a-mole with such work. This approach did not work – rebuttals were rarely read, and misconceptions continued to propagate from paper to paper.
These examples are the tip of the iceberg. Many MOOC papers unknowningly replicate prior results in education research, or incorrectly contradict based on well-known methodological errors.
For the most part, researchers who ignore prior work can:
- Make greater claims about the impact of their own work
- Have more time for research.
In a competitive landscape, such researchers tend to win.
For your presentation:
- Summarize the problem
- Summarize the key parties and incentives involved
- Describe an approach to address the problem