How Do You Set Up an Alternate Assessment Scoring Center?

The process of setting up a scoring center for alternate assessment portfolios can be a daunting task. In most cases, the state will have hired a contractor to manage most of the process. Before the process begins, however, the state department of education, along with other key stakeholders, needs to make several decisions to ensure that scoring progresses according to their expectations. Along with creating the scoring rules for that state’s standards, these decisions include: when scoring will occur and who will work on the scoring project. 

The state department of education makes two separate decisions regarding scheduling: the date scoring will occur and the hours of operation. First, officials set a deadline for score reporting in the schools and districts. Based on this deadline, the schedule is built backwards (i.e., data analysis, clean-up, scoring, etc.). To meet NCLB requirements, the trend is toward earlier and earlier reporting dates. With many states wanting data back by July 1, or even prior to the end of the school year, spring scoring has become more common. 

Closely related to scoring dates are the hours of operation. Longer hours of operation mean fewer days are required for scoring. For example, scoring may occur during typical 8:00-5:00 days or, alternatively, during 4:00-9:00 shifts on Monday through Friday and from 8:00-5:00 on Saturdays. For recruiting purposes, decisions regarding scheduling and about minimum scorer qualifications must be made simultaneously. 

Several types of personnel are required for the efficient running of a scoring center. These include the people directly responsible for scoring and those who support them. The scoring director, table leaders, and, most importantly, the scorers (also called readers) are directly involved in scoring student portfolios. Additional personnel, such as the center manager and assistants, manage logistics, to ensure that those who are involved in scoring can concentrate on that activity. 

The principal role of the scoring director is to ensure that all portfolios are scored consistently. The scoring director must provide intensive, quality training initially, as well as ongoing monitoring of scoring consistency throughout the entire scoring session. When selecting a scoring director, it is important to ask the following questions.

• Is the candidate familiar with the assessment to be scored? So that scoring is completed in accordance with the way the teachers in that particular state were trained to collect information, ideally the scoring director would have been involved in the development of and training for the assessment itself.
• Is the candidate able to multi-task? While monitoring the overall rate and accuracy of the entire center, the scoring director must also identify and realign particular tables or individuals who are beginning to drift from the scoring rules.
• Is the candidate able to communicate clearly? The scoring director must be able to provide precise instructions to the table leaders and scorers.
• What type of working atmosphere will the scoring director promote? It is important that the scoring director promote a collegial, positive atmosphere for scoring professionals. 

Depending on the number of portfolios to be scored and the timeframe required to complete the process, an extra layer of support may be necessary in the form of table leaders. Table leaders have three main responsibilities: 1.) to answer scorers’ questions, as appropriate; 2.) to be the liaison between the scoring director and the scorers; and 3.) to monitor the quality and quantity of work done by the scorers at each table. 

One of the most important decisions a state department of education must make involves the qualifications required to be a scorer. Several critical questions need to be considered:

• How many scorers are needed?
• How many hours a week will scorers work?
• Is it important that the scorers be current or former teachers?
• Is it necessary for scorers to have experience with the assessment?
• Does the scoring process have a secondary purpose of providing professional development to current teachers? Future teachers?

Two typical groups of potential scorers to consider are current or former special education professionals and pre-service educators. Both options have advantages and disadvantages. While the first group of scorers may have a higher level of credibility with teachers and administrators, their available time is much more restricted.  In addition, they need to be monitored closely to ensure that their own experiences do not unduly influence how they score. Using pre-service educators will lead to a larger and more available applicant pool; however, the resulting scores may be less acceptable to the schools and districts. The most important criterion for becoming a scorer must be the ability to score consistently. The best way to establish whether an individual has this ability may not be based on credentials and prior experience but on actual performance scoring sample portfolios.