Matching Items (5)
Filtering by
- All Subjects: Educational evaluation
- Creators: Amrein-Beardsley, Audrey
- Creators: Dahlstrom, Margo
- Creators: Kelley, Michael
- Resource Type: Text
Description
This study examined the intended and unintended consequences associated with the Education Value-Added Assessment System (EVAAS) as perceived and experienced by teachers in the Houston Independent School District (HISD). To evaluate teacher effectiveness, HISD is using EVAAS for high-stakes consequences more than any other district or state in the country. A large-scale electronic survey was used to investigate the model's reliability and validity; to determine whether teachers used the EVAAS data in formative ways as intended; to gather teachers' opinions on EVAAS's claimed benefits and statements; and to understand the unintended consequences that occurred as a result of EVAAS use in HISD. Mixed methods data collection and analyses were used to present the findings in user-friendly ways, particularly when using the words and experiences of the teachers themselves. Results revealed that the reliability of the EVAAS model produced split and inconsistent results among teacher participants, and teachers indicated that students biased the EVAAS results. The majority of teachers did not report similar EVAAS and principal observation scores, reducing the criterion-related validity of both measures of teacher quality. Teachers revealed discrepancies in the distribution of EVAAS reports, the awareness of trainings offered, and among principals' understanding of EVAAS across the district. This resulted in an underwhelming number of teachers who reportedly used EVAAS data for formative purposes. Teachers disagreed with EVAAS marketing claims, implying the majority did not believe EVAAS worked as intended and promoted. Additionally, many unintended consequences associated with the high-stakes use of EVAAS emerged through teachers' responses, which revealed among others that teachers felt heightened pressure and competition, which reduced morale and collaboration, and encouraged cheating or teaching to the test in attempt to raise EVAAS scores. This study is one of the first to investigate how the EVAAS model works in practice and provides a glimpse of whether value-added models might produce desired outcomes and encourage best teacher practices. This is information of which policymakers, researchers, and districts should be aware and consider when implementing the EVAAS, or any value-added model for teacher evaluation, as many of the reported issues are not specific to the EVAAS model.
ContributorsCollins, Clarin (Author) / Amrein-Beardsley, Audrey (Thesis advisor) / Berliner, David C. (Committee member) / Fischman, Gustavo E (Committee member) / Arizona State University (Publisher)
Created2012
Description
ABSTRACT
This study examines validity evidence of a state policy-directed teacher evaluation system implemented in Arizona during school year 2012-2013. The purpose was to evaluate the warrant for making high stakes, consequential judgments of teacher competence based on value-added (VAM) estimates of instructional impact and observations of professional practice (PP). The research also explores educator influence (voice) in evaluation design and the role information brokers have in local decision making. Findings are situated in an evidentiary and policy context at both the LEA and state policy levels.
The study employs a single-phase, concurrent, mixed-methods research design triangulating multiple sources of qualitative and quantitative evidence onto a single (unified) validation construct: Teacher Instructional Quality. It focuses on assessing the characteristics of metrics used to construct quantitative ratings of instructional competence and the alignment of stakeholder perspectives to facets implicit in the evaluation framework. Validity examinations include assembly of criterion, content, reliability, consequential and construct articulation evidences. Perceptual perspectives were obtained from teachers, principals, district leadership, and state policy decision makers. Data for this study came from a large suburban public school district in metropolitan Phoenix, Arizona.
Study findings suggest that the evaluation framework is insufficient for supporting high stakes, consequential inferences of teacher instructional quality. This is based, in part on the following: (1) Weak associations between VAM and PP metrics; (2) Unstable VAM measures across time and between tested content areas; (3) Less than adequate scale reliabilities; (4) Lack of coherence between theorized and empirical PP factor structures; (5) Omission/underrepresentation of important instructional attributes/effects; (6) Stakeholder concerns over rater consistency, bias, and the inability of test scores to adequately represent instructional competence; (7) Negative sentiments regarding the system's ability to improve instructional competence and/or student learning; (8) Concerns regarding unintended consequences including increased stress, lower morale, harm to professional identity, and restricted learning opportunities; and (9) The general lack of empowerment and educator exclusion from the decision making process. Study findings also highlight the value of information brokers in policy decision making and the importance of having access to unbiased empirical information during the design and implementation phases of important change initiatives.
This study examines validity evidence of a state policy-directed teacher evaluation system implemented in Arizona during school year 2012-2013. The purpose was to evaluate the warrant for making high stakes, consequential judgments of teacher competence based on value-added (VAM) estimates of instructional impact and observations of professional practice (PP). The research also explores educator influence (voice) in evaluation design and the role information brokers have in local decision making. Findings are situated in an evidentiary and policy context at both the LEA and state policy levels.
The study employs a single-phase, concurrent, mixed-methods research design triangulating multiple sources of qualitative and quantitative evidence onto a single (unified) validation construct: Teacher Instructional Quality. It focuses on assessing the characteristics of metrics used to construct quantitative ratings of instructional competence and the alignment of stakeholder perspectives to facets implicit in the evaluation framework. Validity examinations include assembly of criterion, content, reliability, consequential and construct articulation evidences. Perceptual perspectives were obtained from teachers, principals, district leadership, and state policy decision makers. Data for this study came from a large suburban public school district in metropolitan Phoenix, Arizona.
Study findings suggest that the evaluation framework is insufficient for supporting high stakes, consequential inferences of teacher instructional quality. This is based, in part on the following: (1) Weak associations between VAM and PP metrics; (2) Unstable VAM measures across time and between tested content areas; (3) Less than adequate scale reliabilities; (4) Lack of coherence between theorized and empirical PP factor structures; (5) Omission/underrepresentation of important instructional attributes/effects; (6) Stakeholder concerns over rater consistency, bias, and the inability of test scores to adequately represent instructional competence; (7) Negative sentiments regarding the system's ability to improve instructional competence and/or student learning; (8) Concerns regarding unintended consequences including increased stress, lower morale, harm to professional identity, and restricted learning opportunities; and (9) The general lack of empowerment and educator exclusion from the decision making process. Study findings also highlight the value of information brokers in policy decision making and the importance of having access to unbiased empirical information during the design and implementation phases of important change initiatives.
ContributorsSloat, Edward F. (Author) / Wetzel, Keith (Thesis advisor) / Amrein-Beardsley, Audrey (Thesis advisor) / Ewbank, Ann (Committee member) / Shough, Lori (Committee member) / Arizona State University (Publisher)
Created2015
Description
This action research dissertation study was undertaken to establish the foundation of a comprehensive evaluation component for the Turn-It-Around (TIA) workshop intervention program at Arizona State University (ASU), and was delivered in the form of a program development consultation. The study's intent was to enhance the ASU Counseling Service's departmental capacity to evaluate one of its important clinical services. The outcomes of this study included multiple assessments of TIA's evaluability and the fidelity of its implementation to its program design. The study products include a well-articulated program theory comprised of program goals, learning objectives, a detailed description of program activities, a logic model, and theoretical construct checklist documents articulating the behavioral science theory underlying the TIA intervention. In addition, instruments tailored to the Turn-It-Around intervention that are suitable for assessing program outcomes were developed and are implementation ready. TIA's clinical stakeholders were interviewed following the generation and delivery of the products and instruments mentioned above to determine whether they found the study's processes and products to be worthwhile and useful. In general, the clinicians reported that they were very satisfied with the benefits and outcomes of the program development consultation. As an action research dissertation, this study generated useful and usable collateral materials in the form of reports, documents, and models. These products are now at the disposal of TIA's institutional stakeholders for use in day-to-day business activities such as training new facilitators and liaisons, and giving presentations that describe the usefulness of TIA as an intervention. Beyond the documents generated to form a program evaluation infrastructure for Turn-It-Around, the processes involved in crafting the documents served to engage relevant stakeholders in a cycle of action research that enriched and solidified their understandings of TIA and furnished them with insight into their counterparts' thinking about the intervention and its potential to benefit the college students they are responsible for helping. Consistent with the intent of action research, the processes involved in accomplishing the objectives of this study surfaced new topics and questions that will be useful in subsequent cycles of program improvement.
ContributorsLacey, Sheila (Author) / Lacey, Sheila D (Thesis advisor) / Clark, Christopher (Committee member) / Kelley, Michael (Committee member) / Krasnow, Aaron (Committee member) / Arizona State University (Publisher)
Created2012
Description
Over the past few years, the issue of childhood trauma in the United States has become significant. A growing number of children are experiencing abuse, neglect, or some other form of maltreatment each year. Considering the stressful home lives of maltreated children, the one sure sanctuary is school. However, this idea requires teachers to be actively involved in identifying and caring for the children who need it most. Traumatic childhood experiences leave lasting scars on its victims, so it is helpful if teachers learn how to identify and support children who have lived through them. It is unfortunate that teachers will most likely encounter children throughout their career who have experienced horrendous things, but it is a reality. With this being said, teachers need to develop an understanding of what traumatized children live with, and learn how to address these issues with skilled sensitivity. Schools are not just a place where children learn how to read and write; they build the foundation for a successful life. This project was designed to provide teachers with a necessary resource for helping children who have suffered traumatic experiences. The methodology of this project began with interviews with organizations specializing in working with traumatized children such as Arizonans for Children, Free Arts for Abused Children, The Sojourner Center, and UMOM. The next step was a review of the current literature on the subject of childhood trauma. The findings have all been compiled into one, convenient document for teacher use and distribution. Upon completion of this document, an interactive video presentation will be made available through an online education website, so that distribution will be made simpler. Hopefully, teachers will share the information with people in their networks and create a chain reaction. The goal is to make it available to as many teachers as possible, so that more children will receive the support they need.
ContributorsHanrahan, Katelyn Ann (Author) / Dahlstrom, Margo (Thesis director) / Kelley, Michael (Committee member) / Division of Teacher Preparation (Contributor) / Sanford School of Social and Family Dynamics (Contributor) / Barrett, The Honors College (Contributor)
Created2016-05
Description
A look at the benefits of the integration of music in the classroom. This thesis focuses on how music supports brain development and how that affects the ways children learn the classroom. It also highlights how current teachers feel about integrating music in the classroom and the best practices used for integrating music. Lastly, this thesis contains strategies on how to integrate music in the classroom using the Common Core standards as well as personal compositions written using Common Core standards.
ContributorsAnger, Jack Vottero (Author) / Dahlstrom, Margo (Thesis director) / Stahlman, Rebecca (Committee member) / Mann, Michael (Committee member) / Barrett, The Honors College (Contributor) / Division of Teacher Preparation (Contributor)
Created2013-05