Matching Items (7)
Filtering by
- All Subjects: Teachers
- All Subjects: Evaluation
- Creators: Amrein-Beardsley, Audrey
- Creators: Zambo, Debby
- Status: Published
Description
This study examined the intended and unintended consequences associated with the Education Value-Added Assessment System (EVAAS) as perceived and experienced by teachers in the Houston Independent School District (HISD). To evaluate teacher effectiveness, HISD is using EVAAS for high-stakes consequences more than any other district or state in the country. A large-scale electronic survey was used to investigate the model's reliability and validity; to determine whether teachers used the EVAAS data in formative ways as intended; to gather teachers' opinions on EVAAS's claimed benefits and statements; and to understand the unintended consequences that occurred as a result of EVAAS use in HISD. Mixed methods data collection and analyses were used to present the findings in user-friendly ways, particularly when using the words and experiences of the teachers themselves. Results revealed that the reliability of the EVAAS model produced split and inconsistent results among teacher participants, and teachers indicated that students biased the EVAAS results. The majority of teachers did not report similar EVAAS and principal observation scores, reducing the criterion-related validity of both measures of teacher quality. Teachers revealed discrepancies in the distribution of EVAAS reports, the awareness of trainings offered, and among principals' understanding of EVAAS across the district. This resulted in an underwhelming number of teachers who reportedly used EVAAS data for formative purposes. Teachers disagreed with EVAAS marketing claims, implying the majority did not believe EVAAS worked as intended and promoted. Additionally, many unintended consequences associated with the high-stakes use of EVAAS emerged through teachers' responses, which revealed among others that teachers felt heightened pressure and competition, which reduced morale and collaboration, and encouraged cheating or teaching to the test in attempt to raise EVAAS scores. This study is one of the first to investigate how the EVAAS model works in practice and provides a glimpse of whether value-added models might produce desired outcomes and encourage best teacher practices. This is information of which policymakers, researchers, and districts should be aware and consider when implementing the EVAAS, or any value-added model for teacher evaluation, as many of the reported issues are not specific to the EVAAS model.
ContributorsCollins, Clarin (Author) / Amrein-Beardsley, Audrey (Thesis advisor) / Berliner, David C. (Committee member) / Fischman, Gustavo E (Committee member) / Arizona State University (Publisher)
Created2012
Description
Public demands for accountability and educational change are at an all-time high. No Child Left Behind set the stage for public accountability of educators and the recently created Race to the Top grant raised the stakes of public school accountability even more with the creation of national standards and assessments as well as public accountability of individual teacher performance based on student test scores. This high-stakes context has placed pressure on local schools to change their instructional practices rapidly to ensure students are learning what they need to in order to perform well on looming Partnership for Assessment of Readiness for College and Careers (PARCC) exams. The purpose of this mixed methods action research study was to explore a shared leadership model and discover the impact of a change facilitation team using the Concerns Based Adoption Model tools on the speed and quality of innovation diffusion at a Title One elementary school. The nine-member change facilitation team received support for 20 weeks in the form of professional development and ongoing team coaching as a means to empower teacher-leaders to more effectively take on the challenges of change. Eight of those members participated in this research. This approach draws on the research on change, learning organizations, and coaching. Quantitative results from the Change Facilitator Stages of Concern Questionnaire were triangulated with qualitative data from interviews, field notes, and Innovation Configuration Maps. Results show the impact on instructional innovation when teacher-leadership is leveraged to support change. Further, there is an important role for change coaches when leading change initiatives. Implications from this study can be used to support other site leaders grappling with instructional innovation and calls for additional research.
ContributorsCruz, Jennifer (Author) / Zambo, Debby (Thesis advisor) / Foulger, Teresa (Committee member) / Tseunis, Paula (Committee member) / Arizona State University (Publisher)
Created2014
Description
ABSTRACT
This study examines validity evidence of a state policy-directed teacher evaluation system implemented in Arizona during school year 2012-2013. The purpose was to evaluate the warrant for making high stakes, consequential judgments of teacher competence based on value-added (VAM) estimates of instructional impact and observations of professional practice (PP). The research also explores educator influence (voice) in evaluation design and the role information brokers have in local decision making. Findings are situated in an evidentiary and policy context at both the LEA and state policy levels.
The study employs a single-phase, concurrent, mixed-methods research design triangulating multiple sources of qualitative and quantitative evidence onto a single (unified) validation construct: Teacher Instructional Quality. It focuses on assessing the characteristics of metrics used to construct quantitative ratings of instructional competence and the alignment of stakeholder perspectives to facets implicit in the evaluation framework. Validity examinations include assembly of criterion, content, reliability, consequential and construct articulation evidences. Perceptual perspectives were obtained from teachers, principals, district leadership, and state policy decision makers. Data for this study came from a large suburban public school district in metropolitan Phoenix, Arizona.
Study findings suggest that the evaluation framework is insufficient for supporting high stakes, consequential inferences of teacher instructional quality. This is based, in part on the following: (1) Weak associations between VAM and PP metrics; (2) Unstable VAM measures across time and between tested content areas; (3) Less than adequate scale reliabilities; (4) Lack of coherence between theorized and empirical PP factor structures; (5) Omission/underrepresentation of important instructional attributes/effects; (6) Stakeholder concerns over rater consistency, bias, and the inability of test scores to adequately represent instructional competence; (7) Negative sentiments regarding the system's ability to improve instructional competence and/or student learning; (8) Concerns regarding unintended consequences including increased stress, lower morale, harm to professional identity, and restricted learning opportunities; and (9) The general lack of empowerment and educator exclusion from the decision making process. Study findings also highlight the value of information brokers in policy decision making and the importance of having access to unbiased empirical information during the design and implementation phases of important change initiatives.
This study examines validity evidence of a state policy-directed teacher evaluation system implemented in Arizona during school year 2012-2013. The purpose was to evaluate the warrant for making high stakes, consequential judgments of teacher competence based on value-added (VAM) estimates of instructional impact and observations of professional practice (PP). The research also explores educator influence (voice) in evaluation design and the role information brokers have in local decision making. Findings are situated in an evidentiary and policy context at both the LEA and state policy levels.
The study employs a single-phase, concurrent, mixed-methods research design triangulating multiple sources of qualitative and quantitative evidence onto a single (unified) validation construct: Teacher Instructional Quality. It focuses on assessing the characteristics of metrics used to construct quantitative ratings of instructional competence and the alignment of stakeholder perspectives to facets implicit in the evaluation framework. Validity examinations include assembly of criterion, content, reliability, consequential and construct articulation evidences. Perceptual perspectives were obtained from teachers, principals, district leadership, and state policy decision makers. Data for this study came from a large suburban public school district in metropolitan Phoenix, Arizona.
Study findings suggest that the evaluation framework is insufficient for supporting high stakes, consequential inferences of teacher instructional quality. This is based, in part on the following: (1) Weak associations between VAM and PP metrics; (2) Unstable VAM measures across time and between tested content areas; (3) Less than adequate scale reliabilities; (4) Lack of coherence between theorized and empirical PP factor structures; (5) Omission/underrepresentation of important instructional attributes/effects; (6) Stakeholder concerns over rater consistency, bias, and the inability of test scores to adequately represent instructional competence; (7) Negative sentiments regarding the system's ability to improve instructional competence and/or student learning; (8) Concerns regarding unintended consequences including increased stress, lower morale, harm to professional identity, and restricted learning opportunities; and (9) The general lack of empowerment and educator exclusion from the decision making process. Study findings also highlight the value of information brokers in policy decision making and the importance of having access to unbiased empirical information during the design and implementation phases of important change initiatives.
ContributorsSloat, Edward F. (Author) / Wetzel, Keith (Thesis advisor) / Amrein-Beardsley, Audrey (Thesis advisor) / Ewbank, Ann (Committee member) / Shough, Lori (Committee member) / Arizona State University (Publisher)
Created2015
Description
Due to the push down of academics, today's elementary students are being asked to learn more concepts and sit for longer periods of time. Sitting slows thinking, whereas movement wakes up the brain. Using movement to learn is embodied cognition, or learning through both the body and the brain. Movement should be part of instruction for young students; however teachers are often not sure how to incorporate movement in their lesson plans. The Japanese practice of lesson study may help because it embeds teachers' new learning in their classrooms while intimately connecting it to the learning of their students, and it links with the cyclical, constructed theory of learning provided by Vygotsky Space. If teachers incorporate movement in their lessons, children have the potential to become more engaged and learn. This action research study was designed to understand if two first grade, two second grade, and one third grade teacher at a Title One elementary school in the Southwestern United States could learn how to use movement more during instruction through lesson study. This innovation took place for 14-weeks during which 12 lessons using movement were developed and taught. Data were collected prior to the study and during each portion of the cyclical process including, while teachers learned, during lessons using movement, and when lessons were discussed and changed. The data sources were pre and post teacher surveys, student surveys, observation protocols, lesson plans, transcripts of lesson study meetings, and researcher notes. To reduce bias a triangulated mixed methods design was used. Results indicate that through lesson study teachers were able to learn about movement, try it, observe the results, and adjust it to fit their teaching style and their students' needs. Data showed increased student engagement in lessons that incorporated movement as evidenced in the students' words, bodies, and learning. After participating in the study, the teachers realized they personally use movement to learn, and teachers' efficacy regarding their ability to plan movement in their lessons increased. Additionally, they started purposefully planning movement across their curriculum. Based on the results, further cycles
ContributorsLyding, Linnea (Author) / Zambo, Debby (Thesis advisor) / Hansen, Cory (Committee member) / Isai, Shelley (Committee member) / Arizona State University (Publisher)
Created2012
Description
Policymakers at the national level have recently initiated K-12 education reforms focused on teacher quality and teacher evaluation. Far-reaching legislation was subsequently enacted in the state of Arizona requiring schools to adopt standards-based teacher evaluation systems and link them to student outcomes. The end product is to result in annual summative measures of teacher effectiveness. Because of this, Arizona school administrators have become concerned about rapidly becoming experts in high-stakes teacher evaluation. Principals rarely have time to come together to talk about teacher evaluation, and consider the reliability of their evaluations and how to use teacher evaluation to help teachers improve their practice. This action research study focused on a group of nine administrators in a small urban district grappling with a more complex and high-stakes teacher evaluation system. An existing community of practice was engaged to help administrators become more effective, fair, and consistent evaluators. Activities were designed to engage the group in dynamic, contextualized learning. Participants interacted in small groups to interpret the meaning of newly adopted evaluation instruments and professional teaching standards, share practical knowledge, and compare teacher evaluation experiences in classrooms. Data were gathered with mixed methods. Prior to, and immediately after engaging in this 20-week innovation, principals and district administrators were given two surveys and interviewed about teacher evaluation. Additionally, a detailed record of this project was kept in the form of meeting records and a research journal. Quantitative and qualitative data were triangulated to validate findings. Results identified concerns and understandings of administrators as they attempted to come to a shared consensus regarding teacher evaluation, increase inter-rater reliability, and use teacher evaluation to improve professional practice. As a result of working and learning together administrators lowered their concerns about inter-rater reliability. Other concerns, however, remained and grew. Administrators found the process of gaining a common understanding of teacher evaluation to be complex and far more time-consuming than anticipated. Intense concerns about alignment of the evaluation system with other reforms led these administrators to consider modifications in their evaluation system. Implications from this study can be used to help other administrators grappling with common concerns.
ContributorsCanelake, Chris (Author) / Zambo, Debby (Thesis advisor) / Hansen, Cory (Thesis advisor) / Ashby, Tacy (Committee member) / Arizona State University (Publisher)
Created2012
Description
There is a serious need for early childhood intervention practices for children who are living at or below the poverty line. Since 1965 Head Start has provided a federally funded, free preschool program for children in this population. The City of Phoenix Head Start program consists of nine delegate agencies, seven of which reside in school districts. These agencies are currently not conducting local longitudinal evaluations of their preschool graduates. The purpose of this study was to recommend initial steps the City of Phoenix grantee and the delegate agencies can take to begin a longitudinal evaluation process of their Head Start programs. Seven City of Phoenix Head Start agency directors were interviewed. These interviews provided information about the attitudes of the directors when considering longitudinal evaluations and how Head Start already evaluates their programs through internal assessments. The researcher also took notes on the Third Grade Follow-Up to the Head Start Executive Summary in order to make recommendations to the City of Phoenix Head Start programs about the best practices for longitudinal student evaluations.
ContributorsArmfield, Jessica Ann (Author) / Duggan, Mary Anne (Thesis director) / Dumka, Larry (Committee member) / Barrett, The Honors College (Contributor) / T. Denny Sanford School of Social and Family Dynamics (Contributor) / School of International Letters and Cultures (Contributor)
Created2014-05
Description
Teacher evaluation policies have recently shifted in the United States. For the first time in history, many states, districts, and administrators are now required to evaluate teachers by methods that are up to 50% based on their "value-added," as demonstrated at the classroom-level by growth on student achievement data over time. Other related instruments and methods, such as classroom observations and rubrics, have also become common practices in teacher evaluation systems. Such methods are consistent with the neoliberal discourse that has dominated the social and political sphere for the past three decades. Employing a discourse analytic approach that called upon a governmentality framework, the author used a complementary approach to understand how contemporary teacher evaluation polices, practices, and instruments work to discursively (re)define teachers and teacher quality in terms of their market value.
For the first part of the analysis, the author collected and analyzed documents and field notes related to the teacher evaluation system at one urban middle school. The analysis included official policy documents, official White House speeches and press releases, evaluation system promotional materials, evaluator training materials, and the like. For the second part of the analysis, she interviewed teachers and their evaluators at the local middle school in order to understand how the participants had embodied the market-based discourse to define themselves as teachers and qualify their practice, quality, and worth accordingly.
The findings of the study suggest that teacher evaluation policies, practices, and instruments make possible a variety of techniques, such as numericization, hierarchical surveillance, normalizing judgments, and audit, in order to first make teachers objects of knowledge and then act upon that knowledge to manage teachers' conduct. The author also found that teachers and their evaluators have taken up this discourse in order to think about and act upon themselves as responsibilized subjects. Ultimately, the author argues that while much of the attention related to teacher evaluations has focused on the instruments used to measure the construct of teacher quality, that teacher evaluation instruments work in a mutually constitutive ways to discursively shape the construct of teacher quality.
For the first part of the analysis, the author collected and analyzed documents and field notes related to the teacher evaluation system at one urban middle school. The analysis included official policy documents, official White House speeches and press releases, evaluation system promotional materials, evaluator training materials, and the like. For the second part of the analysis, she interviewed teachers and their evaluators at the local middle school in order to understand how the participants had embodied the market-based discourse to define themselves as teachers and qualify their practice, quality, and worth accordingly.
The findings of the study suggest that teacher evaluation policies, practices, and instruments make possible a variety of techniques, such as numericization, hierarchical surveillance, normalizing judgments, and audit, in order to first make teachers objects of knowledge and then act upon that knowledge to manage teachers' conduct. The author also found that teachers and their evaluators have taken up this discourse in order to think about and act upon themselves as responsibilized subjects. Ultimately, the author argues that while much of the attention related to teacher evaluations has focused on the instruments used to measure the construct of teacher quality, that teacher evaluation instruments work in a mutually constitutive ways to discursively shape the construct of teacher quality.
ContributorsHolloway-Libell, Jessica (Author) / Amrein-Beardsley, Audrey (Thesis advisor) / Anderson, Kate T. (Thesis advisor) / Berliner, David C. (Committee member) / Arizona State University (Publisher)
Created2014