Matching Items (2)
Filtering by
- All Subjects: Educational evaluation
- Creators: Ewbank, Ann
Description
ABSTRACT Education policymakers at the national level have initiated reforms in K-12 education for that past several years that have focused on teacher quality and teacher evaluation. More recently, reforms have included legislation that focuses on administrator quality as well. Included in far-reaching recent legislation in Arizona is a requirement that administrators be evaluated on a standards-based evaluation system that is linked to student outcomes. The end result is an annual summative measure of administrator effectiveness that impacts job retention. Because of this, Arizona administrators have become concerned about rapidly becoming proficient in the new evaluation systems. Administrators rarely have the explicit professional development opportunities they need to collaborate on a shared understanding of these new evaluation systems. This action research study focused on a group of eight administrators in a small urban district grappling with a new, complex, and high-stakes administrator evaluation that is a component of an all-encompassing Teacher Incentive Fund Grant. An existing professional learning time was engaged to assist administrators in lessening their concerns and increasing their understanding and use of the evaluation instrument. Activities were designed to engage the administrators in dynamic, contextualized learning. Participants interacted in a group to interpret the meaning of the evaluation instrument share practical knowledge and support each other's acquisition understanding. Data were gathered with mixed methods. Administrators were given pre-and post-surveys prior to and immediately after this six-week innovation. Formal and informal interviews were conduct throughout the innovation. Additionally, detailed records in the form of meeting records and a researcher journal were kept. Qualitative and quantitative data were triangulated to validate findings. Results identified concerns and understanding of administrators as they attempted to come to a shared understanding of the new evaluation instrument. As a result of learning together, their concerns about the use of the instrument lessened. Other concerns however, remained or increased. Administrators found the process of the Administrator Learning Community valuable and felt their understanding and use of the instrument had increased. Intense concerns about the competing priorities and initiatives led to the administrators to consider a reevaluation of the competing initiatives. Implications from this study can be used to help other administrators and professional development facilitators grappling with common concerns.
ContributorsEsmont, Leah W (Author) / Wetzel, Keith (Thesis advisor) / Ewbank, Ann (Thesis advisor) / McNeil, David (Committee member) / Arizona State University (Publisher)
Created2014
Description
ABSTRACT
This study examines validity evidence of a state policy-directed teacher evaluation system implemented in Arizona during school year 2012-2013. The purpose was to evaluate the warrant for making high stakes, consequential judgments of teacher competence based on value-added (VAM) estimates of instructional impact and observations of professional practice (PP). The research also explores educator influence (voice) in evaluation design and the role information brokers have in local decision making. Findings are situated in an evidentiary and policy context at both the LEA and state policy levels.
The study employs a single-phase, concurrent, mixed-methods research design triangulating multiple sources of qualitative and quantitative evidence onto a single (unified) validation construct: Teacher Instructional Quality. It focuses on assessing the characteristics of metrics used to construct quantitative ratings of instructional competence and the alignment of stakeholder perspectives to facets implicit in the evaluation framework. Validity examinations include assembly of criterion, content, reliability, consequential and construct articulation evidences. Perceptual perspectives were obtained from teachers, principals, district leadership, and state policy decision makers. Data for this study came from a large suburban public school district in metropolitan Phoenix, Arizona.
Study findings suggest that the evaluation framework is insufficient for supporting high stakes, consequential inferences of teacher instructional quality. This is based, in part on the following: (1) Weak associations between VAM and PP metrics; (2) Unstable VAM measures across time and between tested content areas; (3) Less than adequate scale reliabilities; (4) Lack of coherence between theorized and empirical PP factor structures; (5) Omission/underrepresentation of important instructional attributes/effects; (6) Stakeholder concerns over rater consistency, bias, and the inability of test scores to adequately represent instructional competence; (7) Negative sentiments regarding the system's ability to improve instructional competence and/or student learning; (8) Concerns regarding unintended consequences including increased stress, lower morale, harm to professional identity, and restricted learning opportunities; and (9) The general lack of empowerment and educator exclusion from the decision making process. Study findings also highlight the value of information brokers in policy decision making and the importance of having access to unbiased empirical information during the design and implementation phases of important change initiatives.
This study examines validity evidence of a state policy-directed teacher evaluation system implemented in Arizona during school year 2012-2013. The purpose was to evaluate the warrant for making high stakes, consequential judgments of teacher competence based on value-added (VAM) estimates of instructional impact and observations of professional practice (PP). The research also explores educator influence (voice) in evaluation design and the role information brokers have in local decision making. Findings are situated in an evidentiary and policy context at both the LEA and state policy levels.
The study employs a single-phase, concurrent, mixed-methods research design triangulating multiple sources of qualitative and quantitative evidence onto a single (unified) validation construct: Teacher Instructional Quality. It focuses on assessing the characteristics of metrics used to construct quantitative ratings of instructional competence and the alignment of stakeholder perspectives to facets implicit in the evaluation framework. Validity examinations include assembly of criterion, content, reliability, consequential and construct articulation evidences. Perceptual perspectives were obtained from teachers, principals, district leadership, and state policy decision makers. Data for this study came from a large suburban public school district in metropolitan Phoenix, Arizona.
Study findings suggest that the evaluation framework is insufficient for supporting high stakes, consequential inferences of teacher instructional quality. This is based, in part on the following: (1) Weak associations between VAM and PP metrics; (2) Unstable VAM measures across time and between tested content areas; (3) Less than adequate scale reliabilities; (4) Lack of coherence between theorized and empirical PP factor structures; (5) Omission/underrepresentation of important instructional attributes/effects; (6) Stakeholder concerns over rater consistency, bias, and the inability of test scores to adequately represent instructional competence; (7) Negative sentiments regarding the system's ability to improve instructional competence and/or student learning; (8) Concerns regarding unintended consequences including increased stress, lower morale, harm to professional identity, and restricted learning opportunities; and (9) The general lack of empowerment and educator exclusion from the decision making process. Study findings also highlight the value of information brokers in policy decision making and the importance of having access to unbiased empirical information during the design and implementation phases of important change initiatives.
ContributorsSloat, Edward F. (Author) / Wetzel, Keith (Thesis advisor) / Amrein-Beardsley, Audrey (Thesis advisor) / Ewbank, Ann (Committee member) / Shough, Lori (Committee member) / Arizona State University (Publisher)
Created2015