Measured Progress Researchers Work to “Bridge the Gaps” in Educational Assessments

Reaching Students in the Gap was initiated by the New England Compact, a consortium led by education commissioners and deputy commissioners from Maine, New Hampshire, Rhode Island, and Vermont, and submitted by Rhode Island to the USDOE Enhanced Assessment Initiative competition. The project, which focuses on eighth-grade mathematics, is being facilitated by the Education Development Center. In addition to Measured Progress, other working partners include the Center for Applied Special Technology (CAST), the Technology Assessment Study Collaborative (inTASC), and The Education Alliance at Brown University.

• Goal 1: Define common criteria across the four Compact states for identifying students in the gap.
• Goal 2: Plan, conceptualize, and develop task module assessment strategies, and examine validity and reliability.
• Goal 3: Based on study findings, conceptualize and recommend core components of an assessment structure that would lessen the gaps in the assessment system and allow more students to effectively demonstrate what they know and can do.
• Goal 4: Disseminate products to others considering assessments for students in the gap. 

This project was a natural fit for Measured Progress, OIEA Director Sue Bechard said. Measured Progress develops and administers large-scale and/or alternate assessments for the New England Compact states. “Since we work with the states on various components of their assessments, we want to make sure that those tests are valid for all students,” Bechard said. “We believe that all students can learn and so we want to be sure that our tests fairly measure the learning of all students. When the opportunity to work on this project exploring learning gaps and identifying better ways to assess students in those gaps came along, we at Measured Progress were very eager to participate.” 

As its first task, the New England Compact identified two gaps. Gap 1 occurs at the proficiency cutoff point and includes students who demonstrate proficiency in the classroom but do not demonstrate proficiency on state assessments. For these students “performance on the assessment, even with approved accommodations, is lower than would be expected given the knowledge and skills they regularly exhibit in the classroom.” 

The second gap is at the lower end of the assessment performance continuum and affects students who perform below grade level in the classroom but do not qualify for the alternate assessment. For students in Gap 2, “the large-scale assessment, even with appropriate accommodations, is not a useful measure of their progress.” These students are “working to their capacity, are learning in school, and still are not performing at grade level as demonstrated by the assessment.” 

Researchers at Measured Progress conducted a series of six studies—focusing on students who took the New England Common Assessment Program and the Maine Educational Assessment grade eight mathematics tests—to provide greater depth of information on students in the two gaps. These included analyses of data collected at the time of the test, as well as the following additional surveys and interviews:

• A survey of test administrators to help define the context of the gap and gather teacher and principal reactions to the test in general
• A survey students filled out after completing tests, asking about their motivation, the difficulty of the tests, and their classroom experiences, thus providing insight into factors associated with test performance
• A categorization of gap groups based on student test scores and teacher ratings of student classroom performance level; by comparing test scores with teacher ratings, researchers were able to identify students in the gap.
• Demographic profiles of students in the gap based on information collected in test booklets, as well as state demographic data, leading to more detailed descriptions of students in the gap
• An online survey of teachers regarding characteristics of math programs, and teachers’ reactions to tests and test results; this survey identified areas of need for helping students in the gap.
• Face-to-face teacher interviews about gap students’ classroom experiences and test performance, enabling researchers to develop criteria for identifying students in the gap 

Measured Progress researchers felt “privileged to work with four very thoughtful states that are committed to exploring this issue in depth and to find out which students were not being appropriately assessed,” Bechard said. “Our work addressed the questions: 

• Who are the students in the gaps?
• Of all students who are not proficient, how can states identify those who are in the assessment gaps?
• What are the attributes of students in the gaps, and how do these students perform? 

“We strove to find out more about the students in these assessment gaps so different assessment approaches could be considered based on their needs,” she added. 

To meet its second goal, the New England Compact developed a prototype assessment in eighth-grade mathematics. This assessment's design is such that if students succeed at a complex question, they will move on to another complex question on a different topic. However, if they do not succeed on one complex question, they are then led to a series of simpler questions, each of which addresses a different skill needed to master the complex question. In this way, teachers can determine exactly which skills students have mastered and which require further instruction. 

Based on the findings of these studies, the New England Compact has established preliminary recommendations for an assessment system supporting students in the gap, using varying degrees of accommodations and scaffolds, to be developed as an extension of the current assessment system. 

During the project, Bechard and Ken Godin, also of Measured Progress, collaborated on two research papers to identify students in the gap. 

Finding the real assessment gaps: A process for states to identify gaps in their assessment systems addresses the issue of helping states to identify gap students from among all of the students who are not proficient. This is an “exploratory study to develop a process to identify students in a state assessment system for whom neither the general nor alternate assessments provide valid and/or meaningful information on their true abilities.” 

Who are students in the gaps, what are their attributes, and how do they perform? examines “the characteristics, achievement, and use of accommodations of students in two assessment gaps to answer the question: What are the attributes of students in the gaps, and how do these students perform?” 

The New England Compact’s work was so successful that the USDOE recently announced that a portion of the project’s findings will be posted on the USDOE Web site. Researchers have also been in discussions with the USDOE about the project’s implications for assessments based on modified achievement standards for two percent of the U.S. student population. “The work we have done,” Bechard said, “is very appropriate for the conversation on how these assessments will be implemented.” Additionally, Bechard and other project participants will present the results of their research at three upcoming conferences.

“We are very excited by the results of this project,” Bechard said. “We feel that our research has established a foundation and laid the groundwork for moving forward with a better understanding of the entire population of students taking assessments.”