About Anything

I was disappointed to see the results of this study, but not surprised.

The study, Effectiveness of Reading and Mathematics Software Products Findings From Two Student Cohorts, sponsored by the U.S. Department of Education, reports on student test scores of a second year of use of selected software programs aimed at 1st grade reading, 4th grade reading, 6th grade math, and algebra I. They looked at 10 software products and found that only one had a statistically significant effect. Given that there were 10 chances, that one should also be considered suspect.

Until we really understand the details of human learning we will not be able to build or evaluate effective teaching technology. These broad brushed studies provide such a coarse look at the overall process that we can conclude very little. The study itself ends with a list of caveats that include: “the study preclude direct comparisons of product effects”;  “Because districts and schools volunteered to implement particular products, their characteristics differ and these differences may relate to effectiveness”; “The study design does not rule out the possibility that a product the study finds to be ineffective could be effective if implemented by other districts or schools”.

So why did the Department of Education bother to do it?

It would be much better to spend resources on understanding the learning process with enough rigor to construct educational environments that improve it.

A research report in Science, Why Peer Discussion Improves Student Performance on In-Class Concept Questions (subscription required), reports that clickers used in the classroom (see clickers in the classroom for a review) can help students arrive at conceptual understanding on their own through the process of group discussion and debate.  According to the authors of the study, most instructors report that the percentage of correct answers almost always increases after peer discussion. They go on to say that it has been generally assumed that active engagement of students during discussion with peers, some of whom know the correct answer, leads to increased conceptual understanding, resulting in improved performance after PI.

In this study, students were asked to answer the question a second time after their peer discussion, but also to answer a different, conceptually related question. Neither the correct answer, nor the student responses were shown between questions. The authors conclude that “our results suggest that peer discussion can be effective for understanding difficult concepts even when no one in the group initially knows the correct answer.”

One of their students, quoted in the study says it pretty well: “Often when talking through the questions, the group can figure out the questions without originally knowing the answer, and the answer almost sticks better that way because we talked through it instead of just hearing the answer.”