The Article
Standards are an increasingly important issue in K-12 education. But how do educators establish standards? What methodologies can educators use to establish workable and effective educational standards? MSU Professor Mark Reckase presents a general framework for describing components of standard-setting procedures, develops some criteria for considering alternative standard setting methods, and describes a number of potential methods.

Discussion
Although it may seem simple, standard setting is actually a very complex process that involves a number of components, each of which is important. Reckase describes five components in standard-setting methodology: (1) an “authority” to set policy, (2) a content domain (mathematics, English, etc.) that is the focus of the standard, (3) a selection of persons (judges) to make judgments about desired levels of performance, (4) methodology for collecting judgments and estimating standards, and (5) some means for reporting the results of the process. Each component is critical to the process. For instance, without sound methodology for collecting information from judges about their recommendations for standards, the connection between policy, content, and scores cannot be defended. Because of the importance of this component, many alternative standard-setting methods have been developed and substantial work is being done to evaluate the quality of alternative methods. Reckase describes a number of methods such as the bookmark method, the anchor-based procedure, generalized examinee-centered method, and the multistage aggregation method. The various methods provide differing methodologies from asking judges to rate increasing amounts of performance information over rounds (multi-stage aggregation method) to having judges use a 7-point rating scale related to four performance categories of minimal, partially, proficient, and advanced (generalized-examinee centered method). All of them have potential, but most have been used only in limited research or pilot studies. Because standard setting can be implemented in a number of different ways, it is difficult to identify firm criteria for determining whether a standard-setting procedure will likely give sound results.  A very good standard-setting method can give poor results, for example, if the judges are not properly training, or if they do not have the necessary content background. Reckase provides readers with four criteria for a sound standard-setting methodology: (1) minimal level of distortion in converting judgments to a standard, (2) moderate to low cognitive complexity of the tasks judges are asked to perform, (3) acceptable standard errors of estimate for the scores, and (4) replicable process for conducting the standard setting study.

What It Means To You
Standards are increasingly important tool for accountability and are the subject of much attention by lawmakers, parents, administrators and teachers. But as Reckase’s article points out, setting standards is a lengthy and complex process that involves a number of crucial components in order for results to be a valid reflection of performance.

For More Information
Reckase, M. (2000). A survey and evaluation of recently developed procedures for setting standards on educational tests. Student performance standards on the National Assessment of Educational Progress: Affirmation and improvements. Washington, D.C.: National Assessment Governing Board.

<back to 2000 ed-Research Reports