Frequently Asked Questions

Q: Why do we need large-scale assessment programs?
Q: How does Measured Progress approach the design of customized large-scale assessment programs?
Q: What are some of the significant benefits Measured Progress offers its clients?
Q: What are typical methods for developing and implementing large-scale assessments?
Q: Which assessment methods does Measured Progress offer?
Q: How are assessment materials produced?
Q: How does Measured Progress handle the logistics of large-scale assessment programs?
Q: How are test answers scored?
Q: How are assessment results generated and used?
Q: How does Measured Progress ensure quality control?

In response to state and federal education reform acts and accountability requirements, most recently, the No Child Left Behind Act of 2001 (NCLB), national agencies, states, and large districts across the country have implemented far-reaching systems that reform educational standards and include large-scale assessments to monitor student achievement of those standards. Basic reform strategies emphasize the need for high academic standards—describing what all children should know and be able to do—and high-quality assessments aligned with those standards. States are required to develop and implement assessment programs that correspond to curriculum standards and assess students in designated grade levels and subjects. Under NCLB, students in grades 3 through 8 plus 1 high-school grade must be assessed annually in reading and math, beginning in 2005-2006. Mandated science assessments at 3 grade levels will begin in 2007-2008.

Measured Progress combines both traditional and innovative measures in the design of large-scale assessment programs, most of which include multiple-choice, short-answer, open-response questions and writing tasks for primary, middle, and high school students. Some of the more innovative programs also use performance tasks and portfolio evaluations. By designing programs that include various types of measures—all determined by our clients’ assessment goals—we broaden the vision of what is possible in large-scale testing programs. Customization is the hallmark of each of our programs. Our clients establish their assessment programs’ specific curriculum frameworks and performance standards. Local educators in client states are usually involved in the processes of test development, standard setting, and interpretation of results.

Measured Progress is unique in its approach to contract management. We form partnerships with our clients and focus on their needs while also lending the benefit of our years of experience. While the client ultimately determines the direction, priorities, and purposes of any assessment, Measured Progress brings to each program

• expert and experienced staff and technical consultants;
• creativity in assessment design;
• custom development of assessment programs;
• flexibility and responsiveness to clients’ needs;
• alternate means of assessing performance for students whose significant disabilities prevent their participation in a general assessment, even with accommodations;
• thorough understanding of the relationship between educational policy and psychometrics; and
• various levels of professional development to help educators integrate standards with classroom instruction and assessment.

Virtually all of our state assessment contracts include item and test development in collaboration with content advisory committees. During the formative and early implementation phases of a program, Measured Progress works extensively with state departments of education to provide critical information and support to educators, and offers assessment expertise to key decision makers in the state through consultation and workshops. Every element of an assessment program is the result of collaboration among Measured Progress personnel, state administrators, and local educators. We also collaborate on the development of alternate assessments, exploring ways to assess achievement of the state's standards for students with significant disabilities.
For most clients, Measured Progress not only develops and produces assessments, we also assist with field testing, administration, distribution, receiving, scoring, data analysis, and reporting. Measured Progress facilitates numerous professional development workshops with educators addressing classroom assessment strategies and the use of assessment results. Our program-management teams provide reliable assistance to policy makers, educators, and parents during the development, administration, and reporting of assessment programs. For some clients, Measured Progress establishes and maintains service centers to provide assistance relating to a particular program. Center staff answers calls from superintendents, school administrators, teachers, parents, and the general public.

To meet the unique goals of each assessment program, Measured Progress designs and creates assessment instruments, including a variety of questions.

Traditional multiple-choice questions: Traditional multiple-choice (selected-response) questions are highly efficient mechanisms for assessing the breadth of a student’s knowledge.

“Enhanced” multiple-choice questions: Enhanced multiple-choice (selected-response) questions require not only knowledge of basic facts, but also the ability to apply higher-order thinking skills to make sense of those facts.

Short-answer questions: Short-answer questions require responses ranging from a few words to several sentences. They allow the assessing of higher-order thinking skills, but, unlike multiple-choice questions, are relatively immune to random guessing.

Open-response questions: Open-response (constructed-response) questions invite students to demonstrate not only their knowledge of facts and comprehension of a subject, but how they can apply their knowledge. These questions require students to construct a detailed or descriptive response.

Scenario-based questions: Scenario-based questions present an introductory piece that may include figures, tables, maps, graphs, and the like. Several questions relate to each scenario, and may be of various item types. Scenario-based questions permit assessment of integrated knowledge in a richer context.

Videotape-based questions: Some English language arts assessments include a session during which students view a videotape and then respond to questions based on what they have seen and heard.

Writing assessment: Writing assessment methods fall into two major categories: indirect and direct. The indirect approach typically uses multiple-choice questions and revision exercises to assess discrete skills such as spelling, capitalization, punctuation, grammar and word usage (writing conventions). In a direct writing assessment, students are asked to write one or more essays, which are then evaluated by a range of scoring techniques, such as primary trait, analytic, holistic, and annotated holistic methods. Most direct writing essays are written “on demand,” in immediate response to a prompt.

Portfolios: Portfolio assessments measure students’ skills and achievement via different types of measures, including long-term tasks or projects. Each student's portfolio could include a combination of classroom and on-demand work.

Curriculum-embedded assessment: Curriculum-embedded assessment provides an activity or instructional piece that teachers use in their classrooms, adapting as necessary to fit the needs of their students and their own teaching styles. An assessment unit can span several hours or several weeks. The assessment component remains kept secure until administered in the classroom.

Performance tasks: Performance tasks contain both small group and individual student work. Tasks usually require students to use resource materials or equipment (e.g., maps, photographs, art supplies, lab equipment, measurement devices). Measured Progress has developed and administered performance tasks in the areas of mathematics, science, social studies, arts and humanities, practical living, and vocational studies.

Curriculum and assessment specialists develop questions according to contract specifications and requirements, working with local committees to create new items for each program, and drawing from a large pool of tried and reliable items, if needed. Our computerized Item Referencing System (I-Ref) expedites the development process. This first implementation of quality control sets the stage for a standard of high quality throughout the publication process. Included in this process are editorial review, management of the database of test items, test construction and production, desktop publishing, print buying, and technical specification management. Along with the test booklets, the range of produced materials includes interpretive guides, released item sets, special test forms (foreign translations, Braille, large print), and ancillary test materials (manuals, scannable documents).

Measured Progress’s distribution, return, and scanning capacity continues to grow. Our Newington, New Hampshire, warehouse facility allows for proper materials staging, a variety of collation activities, packing, and shipping, and retrieval operations, which often include extensive verification activities and sorting of materials.

After student answer booklets are received and checked in, they are optically scanned to capture all multiple-choice answers, as well as the written student responses to open-response questions and writing prompts. Our high-capacity scanning systems use the latest technology with a variety of data-capture techniques. Specific quality control procedures developed for each project assure data integrity. Scoring operations for non-multiple-choice responses take place at various sites across the country and involve thousands of individuals, all of whom meet or exceed our exacting qualifications. Finally, data processors and psychometricians complete the data analyses and generate reports.

Measured Progress analyzes assessment items and results, and produces student, school, district, and state reports.

Test Item Analyses

Classical item statistics are produced for all test items, both field test and operational. Analyses include verification that scoring keys and rubrics are correct, determination of item difficulty and discrimination, and computation of item non-response rates (items skipped and items not reached).

Standard Setting

Programs that report the results of their assessments by proficiency levels include standard-setting studies, usually during the first year of a program, to determine the threshold total test scores separating the proficiency levels. Measured Progress has conducted standard-setting studies, and studies on reliability, validity, and comparability of the assessments, for several clients. All standard-setting procedures incorporate the expertise and specifications of policy makers and educators in each client state.

Reports

Assessment results are reported in ways that are clear and meaningful to people who deal with students (parents and teachers) and to those whose decisions most directly affect students (administrators, policy makers, and the public). Measured Progress offers customized workshops and services to help educators explore questions related to standards-based education, scoring student work, and interpretation of assessment reports in order to develop the insights and skills to make the transition to a standards-based instructional system.

Quality-control procedures and redundant security precautions are embedded in our customized assessment programs. We design assessment instruments to assure reliability of results. Test items are selected based on their match to objectives and psychometric quality. Test forms are shrink-wrapped to prevent materials from being lost or separated before administration. Manuals and certification forms are used to ensure uniformity of administration procedures across schools. Equating procedures assure comparability of assessment results across years for many programs.