The Article
In this article, Mary Kennedy, professor in the Department of Teacher Education, deals with the difficulty for education researchers of accurately gauging complex student learning as a result of policy changes. She analyzes and compares various assessment research by looking at such things as standardized tests, open-ended teacher interviews and observations. She examines and compares the various assessment tools.

Discussion
Education policy researchers face a difficult task in documenting student learning based on the implementation of new policies or changes to existing policies. It is extremely difficult, she says, to chart a clear path from policy change to student outcome. The key is that there is often no adequate measure of the outcomes researchers seek. There simply is no widely available and inexpensive (and generally agreed-upon) indicator or indicators of complex student learning. That puts researchers at a disadvantage when trying to gauge the influence of a particular policy or program. Instead, researchers have had to rely on more easily documented intermediate outcomes. Intermediate outcomes include such things as the impact a policy change has had on teachers’ attitudes and assuming that such an effect will lead to further changes in student outcomes. Kennedy breaks down these intermediate outcomes, known as approximations, into four levels, and shows that they vary widely in terms of how well they actually approximate an indicator of complex student learning. The first level is classroom observation and standardized tests. Standardized tests, she finds, have the advantage of being inexpensive, convenient, and perceived as valid indicators of student learning. Yet they also have been criticized as too narrow to gauge complex learning. Observations, on the other hand, are capable of documenting the intellectual complexity of the work students do in class. But like standardized tests, observations have shortcomings, including the fact that they cannot document the extent to which students have learned to handle complex problems. In addition, they are costly to conduct. The second level involves situated descriptions of teaching. In this case, researchers rely on teachers to obtain “as situated a description as possible of the teachers’ own teaching practice.” Common forms of situated descriptions are the teacher log and the vignette. Kennedy finds that this form of assessment also has pluses and minuses. The third level is the non-situated testimony about practice. This level relies on teacher questionnaires and interviews that ask about teaching practices but are not situated in specific teaching episodes. These types of questions elicit general responses such as “I always treat my students equally.” While used widely in survey research, Kennedy notes that non-situated testimony has only a “slight” relationship to complex student learning. The final level is testimony about effects of policies and programs. Kennedy finds this level even more removed than from non-situated testimony. In this level, teachers testify about whether a particular policy or program was helpful or whether it influenced teachers in some way. Although helpful in understanding teachers’ satisfaction or dissatisfaction with a policy or program, these approximations are “even more susceptible to self-serving biases than second- and third-level approximations are.” Kennedy concludes by drawing on the research to point out that the best first-level approximation available to researchers would be a classroom observation that focuses on the nature of intellectual work students do in class. Because of the expense, she suspects many researchers cannot afford this type of assessment. In such cases, she advises researchers that “it might be more fruitful to invest time in developing and field-testing questions that are closely situated, so that the distance between these questions and closer approximations is reduced as much as possible.”

Citation
Kennedy, M.M. (2000). Approximations to indicators of student outcomes. Education Evaluation and Policy Analysis, 21(4), 345-363.

<back to 2000 ed-Research Reports