Evaluating Special Education Teachers’ Classroom Performance: Rater Reliability at the Rubric Item Level:

Published on Nov 1, 2018in Teacher Education and Special Education
· DOI :10.1177/0888406417718250
Janelle E. Lawson2
Estimated H-index: 2
(SFSU: San Francisco State University),
Rebecca A. Cruz5
Estimated H-index: 5
(University of California, Berkeley)
Classroom observations are an integral component of teacher evaluation systems, but little is known about who is best qualified to observe and evaluate special educators, who have a specialized skillset, and whether observation instruments adequately reflect their instructional practices. In this study, 19 special education teachers in California and Idaho each contributed three video-recorded classroom lessons. Using rubric items designed to reflect efficacious instructional practices for teaching students with disabilities, school administrators and peers scored the teachers’ lessons. Rater reliability and sources of error variance were examined using generalizability theory. Findings indicate that peers were more reliable raters than school administrators, who did not have expertise in special education, and the school administrators’ ratings varied at the rubric items level. Implications for classroom observation systems are discussed.
📖 Papers frequently viewed together
1 Citations
1 Citations
8 Citations
#1Peter F. Halpin (NYU: New York University)H-Index: 14
#2Michael J. Kieffer (NYU: New York University)H-Index: 31
The authors outline the application of latent class analysis (LCA) to classroom observational instruments. LCA offers diagnostic information about teachers’ instructional strengths and weaknesses, along with estimates of measurement error for individual teachers, while remaining relatively straightforward to implement and interpret. It is discussed how the methodology can support formative feedback to educators and facilitate research into the associations between instructional practices and stu...
32 CitationsSource
#1Ellen Goldring (Vandy: Vanderbilt University)H-Index: 43
#2Jason A. Grissom (Vandy: Vanderbilt University)H-Index: 28
Last. Patrick Schuermann (Vandy: Vanderbilt University)H-Index: 7
view all 7 authors...
Increasingly, states and districts are combining student growth measures with rigorous, rubric-aligned teacher observations in constructing teacher evaluation measures. Although the student growth or value-added components of these measures have received much research and policy attention, the results of this study suggest that the data generated by high-quality observation systems have potential to inform principals’ use of data for human capital decisions. Interview and survey data from six sc...
105 CitationsSource
#1Carrie Semmelroth (BSU: Boise State University)H-Index: 5
#2Evelyn S. Johnson (BSU: Boise State University)H-Index: 12
This study used generalizability theory to measure reliability on the Recognizing Effective Special Education Teachers (RESET) observation tool designed to evaluate special education teacher effectiveness. At the time of this study, the RESET tool included three evidence-based instructional practices (direct, explicit instruction; whole-group instruction; and discrete trial teaching) as the basis for special education teacher evaluation. Five raters participated in two sessions to evaluate speci...
18 CitationsSource
#1Nathan D. Jones (BU: Boston University)H-Index: 10
#2Mary T. Brownell (UF: University of Florida)H-Index: 30
A primary focus of recent educational policymaking has been on improving measures of teaching effectiveness, attributable in part to recent federal policies such as the Race to the Top initiative and Investing in Innovation. To date, neither researchers nor practitioners have arrived at a consensus on the best methods for evaluating special educators. While value-added scores are likely not suitable for the majority of special education teachers, observation systems appear to be more promising b...
51 CitationsSource
#1Jason A. Grissom (Vandy: Vanderbilt University)H-Index: 28
#2Susanna Loeb (Stanford University)H-Index: 74
Last. Benjamin K. Master (Stanford University)H-Index: 4
view all 3 authors...
Scholars have long argued that principals should be instructional leaders, but few studies have empirically linked specific instructional leadership behaviors to school performance. This study examines the associations between leadership behaviors and student achievement gains using a unique data source: in-person, full-day observations of approximately 100 urban principals collected over 3 school years. We find that principals’ time spent broadly on instructional functions does not predict stud...
186 CitationsSource
#1Ann Sledge (University of Texas at Austin)H-Index: 1
#2Barbara L. Pazey (University of Texas at Austin)H-Index: 11
While teacher quality is recognized as a critical component in school reform and the pursuit of new teacher evaluation systems has gained national attention, the question of whether proposed teacher assessment models recognize and account for the unique roles and responsibilities of special education teachers has gone largely unnoticed. The purpose of this article is to (a) provide a review of current efforts to reform practices in teacher assessment, (b) describe recommendations for emerging te...
23 CitationsSource
#1Courtney A. BellH-Index: 16
#2Drew H. Gitomer (RU: Rutgers University)H-Index: 23
Last. Yi QiH-Index: 6
view all 6 authors...
This article develops a validity argument approach for use on observation protocols currently used to assess teacher quality for high-stakes personnel and professional development decisions. After defining the teaching quality domain, we articulate an interpretive argument for observation protocols. To illustrate the types of evidence that might compose a validity argument, we draw on data from a validity study of the Classroom Assessment Scoring System for secondary classrooms. Based on data fr...
126 CitationsSource
#1Joseph P. Allen (UVA: University of Virginia)H-Index: 69
#2Robert C. Pianta (UVA: University of Virginia)H-Index: 139
Last. Janetta Lun (UMD: University of Maryland, College Park)H-Index: 10
view all 5 authors...
Improving teaching quality is widely recognized as critical to addressing deficiencies in secondary school education, yet the field has struggled to identify rigorously evaluated teacher-development approaches that can produce reliable gains in student achievement. A randomized controlled trial of My Teaching Partner–Secondary—a Web-mediated approach focused on improving teacher-student interactions in the classroom—examined the efficacy of the approach in improving teacher quality and student a...
352 CitationsSource
#1Jean CardinetH-Index: 7
#2Sandra JohnsonH-Index: 1
Last. Gianreto Pini (UoB: University of Bristol)H-Index: 1
view all 3 authors...
Introduction. 1. What is Generalizability Theory? 2. Generalizability Theory: Concepts and Principles. 3. Using EduG: The Generalizability Theory Software. 4. Applications to the Behavioral and Social Sciences. 5. Practice Exercises. 6. Current Developments and Future Possibilities. Appendixes.
90 Citations
#1Chris Chiu (LSAC: Law School Admission Council)H-Index: 1
#2Edward W. Wolfe (MSU: Michigan State University)H-Index: 25
In generalizability analyses, unstable, and potentially invalid, variance component estimates may result from using only a limited portion of available data. However, missing observations are common in operational performance assessment settings because of the nature of the assessment design. This article describes a procedure for overcoming the computational and technological limitations in analyzing data with missing observations by extracting data from a sparsely .lled data matrix into analyz...
29 CitationsSource
Cited By1
#1Janelle E. Rodl (SFSU: San Francisco State University)H-Index: 3
#2Rebecca A. Cruz (University of California, Berkeley)H-Index: 5
Last. Gregory A. Knollman (TU: Towson University)H-Index: 2
view all 3 authors...
Abstract This article examines Q methodology as an empirical approach for use in teacher evaluation research, specifically research examining evaluative measures as applied to teachers of students with disabilities. Q is a qualiquantalogical method that requires participants to order or scale items relative to each other in characterizing a person or behavior. Participant viewpoints are intercorrelated and factor analyzed to determine common viewpoints regarding the salience of items representin...
1 CitationsSource