Matching Items (22)
Description
This study examined the intended and unintended consequences associated with the Education Value-Added Assessment System (EVAAS) as perceived and experienced by teachers in the Houston Independent School District (HISD). To evaluate teacher effectiveness, HISD is using EVAAS for high-stakes consequences more than any other district or state in the country. A large-scale electronic survey was used to investigate the model's reliability and validity; to determine whether teachers used the EVAAS data in formative ways as intended; to gather teachers' opinions on EVAAS's claimed benefits and statements; and to understand the unintended consequences that occurred as a result of EVAAS use in HISD. Mixed methods data collection and analyses were used to present the findings in user-friendly ways, particularly when using the words and experiences of the teachers themselves. Results revealed that the reliability of the EVAAS model produced split and inconsistent results among teacher participants, and teachers indicated that students biased the EVAAS results. The majority of teachers did not report similar EVAAS and principal observation scores, reducing the criterion-related validity of both measures of teacher quality. Teachers revealed discrepancies in the distribution of EVAAS reports, the awareness of trainings offered, and among principals' understanding of EVAAS across the district. This resulted in an underwhelming number of teachers who reportedly used EVAAS data for formative purposes. Teachers disagreed with EVAAS marketing claims, implying the majority did not believe EVAAS worked as intended and promoted. Additionally, many unintended consequences associated with the high-stakes use of EVAAS emerged through teachers' responses, which revealed among others that teachers felt heightened pressure and competition, which reduced morale and collaboration, and encouraged cheating or teaching to the test in attempt to raise EVAAS scores. This study is one of the first to investigate how the EVAAS model works in practice and provides a glimpse of whether value-added models might produce desired outcomes and encourage best teacher practices. This is information of which policymakers, researchers, and districts should be aware and consider when implementing the EVAAS, or any value-added model for teacher evaluation, as many of the reported issues are not specific to the EVAAS model.
ContributorsCollins, Clarin (Author) / Amrein-Beardsley, Audrey (Thesis advisor) / Berliner, David C. (Committee member) / Fischman, Gustavo E (Committee member) / Arizona State University (Publisher)
Created2012
Description
ABSTRACT
This study examines validity evidence of a state policy-directed teacher evaluation system implemented in Arizona during school year 2012-2013. The purpose was to evaluate the warrant for making high stakes, consequential judgments of teacher competence based on value-added (VAM) estimates of instructional impact and observations of professional practice (PP). The research also explores educator influence (voice) in evaluation design and the role information brokers have in local decision making. Findings are situated in an evidentiary and policy context at both the LEA and state policy levels.
The study employs a single-phase, concurrent, mixed-methods research design triangulating multiple sources of qualitative and quantitative evidence onto a single (unified) validation construct: Teacher Instructional Quality. It focuses on assessing the characteristics of metrics used to construct quantitative ratings of instructional competence and the alignment of stakeholder perspectives to facets implicit in the evaluation framework. Validity examinations include assembly of criterion, content, reliability, consequential and construct articulation evidences. Perceptual perspectives were obtained from teachers, principals, district leadership, and state policy decision makers. Data for this study came from a large suburban public school district in metropolitan Phoenix, Arizona.
Study findings suggest that the evaluation framework is insufficient for supporting high stakes, consequential inferences of teacher instructional quality. This is based, in part on the following: (1) Weak associations between VAM and PP metrics; (2) Unstable VAM measures across time and between tested content areas; (3) Less than adequate scale reliabilities; (4) Lack of coherence between theorized and empirical PP factor structures; (5) Omission/underrepresentation of important instructional attributes/effects; (6) Stakeholder concerns over rater consistency, bias, and the inability of test scores to adequately represent instructional competence; (7) Negative sentiments regarding the system's ability to improve instructional competence and/or student learning; (8) Concerns regarding unintended consequences including increased stress, lower morale, harm to professional identity, and restricted learning opportunities; and (9) The general lack of empowerment and educator exclusion from the decision making process. Study findings also highlight the value of information brokers in policy decision making and the importance of having access to unbiased empirical information during the design and implementation phases of important change initiatives.
This study examines validity evidence of a state policy-directed teacher evaluation system implemented in Arizona during school year 2012-2013. The purpose was to evaluate the warrant for making high stakes, consequential judgments of teacher competence based on value-added (VAM) estimates of instructional impact and observations of professional practice (PP). The research also explores educator influence (voice) in evaluation design and the role information brokers have in local decision making. Findings are situated in an evidentiary and policy context at both the LEA and state policy levels.
The study employs a single-phase, concurrent, mixed-methods research design triangulating multiple sources of qualitative and quantitative evidence onto a single (unified) validation construct: Teacher Instructional Quality. It focuses on assessing the characteristics of metrics used to construct quantitative ratings of instructional competence and the alignment of stakeholder perspectives to facets implicit in the evaluation framework. Validity examinations include assembly of criterion, content, reliability, consequential and construct articulation evidences. Perceptual perspectives were obtained from teachers, principals, district leadership, and state policy decision makers. Data for this study came from a large suburban public school district in metropolitan Phoenix, Arizona.
Study findings suggest that the evaluation framework is insufficient for supporting high stakes, consequential inferences of teacher instructional quality. This is based, in part on the following: (1) Weak associations between VAM and PP metrics; (2) Unstable VAM measures across time and between tested content areas; (3) Less than adequate scale reliabilities; (4) Lack of coherence between theorized and empirical PP factor structures; (5) Omission/underrepresentation of important instructional attributes/effects; (6) Stakeholder concerns over rater consistency, bias, and the inability of test scores to adequately represent instructional competence; (7) Negative sentiments regarding the system's ability to improve instructional competence and/or student learning; (8) Concerns regarding unintended consequences including increased stress, lower morale, harm to professional identity, and restricted learning opportunities; and (9) The general lack of empowerment and educator exclusion from the decision making process. Study findings also highlight the value of information brokers in policy decision making and the importance of having access to unbiased empirical information during the design and implementation phases of important change initiatives.
ContributorsSloat, Edward F. (Author) / Wetzel, Keith (Thesis advisor) / Amrein-Beardsley, Audrey (Thesis advisor) / Ewbank, Ann (Committee member) / Shough, Lori (Committee member) / Arizona State University (Publisher)
Created2015
Description
This mixed-methods study explored perceptions of new adjuncts on various trainings with regards to satisfying their professional and aspirational needs. Three trainings were offered in fall 2018 quarter as optional professional development: workshop, and two roundtable sessions. These trainings assisted adjuncts with their teaching skills, educational technology and pedagogy. Guidance was provided from experienced adjuncts and staff.
Surveys and interviews with adjuncts, along with a focus group with staff were the sources of data for this study. A repeated measures Analysis of Covariance (ANCOVA) model was utilized. Analysis of data showed that there was a positive and statistical significance of change in perceptions of adjuncts who participated in all trainings towards fulfilling their needs, as compared to those who did not participate in any trainings. Adjuncts perceived an improvement in their professional growth based on Herzberg’s motivation-hygiene theory and the trainings also fulfilled their higher-level growth needs based on Maslow’s hierarchical needs theory. A large practical significance was also found which measures the practical impact of such trainings at local communities of practice.
Surveys and interviews with adjuncts, along with a focus group with staff were the sources of data for this study. A repeated measures Analysis of Covariance (ANCOVA) model was utilized. Analysis of data showed that there was a positive and statistical significance of change in perceptions of adjuncts who participated in all trainings towards fulfilling their needs, as compared to those who did not participate in any trainings. Adjuncts perceived an improvement in their professional growth based on Herzberg’s motivation-hygiene theory and the trainings also fulfilled their higher-level growth needs based on Maslow’s hierarchical needs theory. A large practical significance was also found which measures the practical impact of such trainings at local communities of practice.
ContributorsSreekaram, Siddhartha (Author) / Marsh, Josephine (Thesis advisor) / Amrein-Beardsley, Audrey (Committee member) / Kim, Jeongeun (Committee member) / Arizona State University (Publisher)
Created2019
Description
In 2010, the Arizona Legislature established a performance-based diploma initiative known as Move On When Ready (MOWR). The policy relies on an education model designed to evaluate students' college and career readiness by measuring their academic ability to succeed in the first credit-bearing course in community college. Move On When Ready is a structurally oriented, qualification system that attempts to attain a relatively narrow goal: increase the number of students able to successfully perform at a college-level academic standard. By relying on a set of benchmarked assessments to measure success and failure, MOWR propagates a categorical binary. The binary establishes explicit performance criteria on a set of examinations students are required to meet in order to earn a high school qualification that, by design, certifies whether students are ready or not ready for college.
This study sought to reveal how students’ perceptions of the policy and schooling in general affect their understanding of the concept of college readiness and the college readiness binary and to identify factors that help formulate those perceptions. This interpretivist, qualitative study relied on analysis of multiple face-to-face interviews with students to better understand how they think and act within the context of Move On When Ready, paying particular attention to students from historically vulnerable minority subgroups (e.g., the Latina (a)/Hispanic sub-population) enrolled in two schools deploying the MOWR strategy.
Findings suggest that interviewed students understand little about MOWR's design, intent or implications for their future educational trajectories. Moreover, what they believe is generally misinformed, regardless of aspiration, socio-cultural background, or academic standing. School-based sources of messaging (e.g., teachers and administrators) supply the bulk of information to students about MOWR. However, in these two schools, the flow of information is constricted. In addition, the information conveyed is either distorted by message mediators or misinterpreted by the students. The data reveal that formal and informal mediators of policy messages influence students’ engagement with the policy and affect students’ capacity to play an active role in determining the policy’s effect on their educational outcomes.
This study sought to reveal how students’ perceptions of the policy and schooling in general affect their understanding of the concept of college readiness and the college readiness binary and to identify factors that help formulate those perceptions. This interpretivist, qualitative study relied on analysis of multiple face-to-face interviews with students to better understand how they think and act within the context of Move On When Ready, paying particular attention to students from historically vulnerable minority subgroups (e.g., the Latina (a)/Hispanic sub-population) enrolled in two schools deploying the MOWR strategy.
Findings suggest that interviewed students understand little about MOWR's design, intent or implications for their future educational trajectories. Moreover, what they believe is generally misinformed, regardless of aspiration, socio-cultural background, or academic standing. School-based sources of messaging (e.g., teachers and administrators) supply the bulk of information to students about MOWR. However, in these two schools, the flow of information is constricted. In addition, the information conveyed is either distorted by message mediators or misinterpreted by the students. The data reveal that formal and informal mediators of policy messages influence students’ engagement with the policy and affect students’ capacity to play an active role in determining the policy’s effect on their educational outcomes.
ContributorsSilver, Michael Greg (Author) / Berliner, David C. (Thesis advisor) / Fischman, Gustavo (Committee member) / Amrein-Beardsley, Audrey (Committee member) / Arizona State University (Publisher)
Created2015
Description
Public schools across the country are increasingly dealing with children who enter schools speaking a language other than English and Arizona is not the exception. As a result, schools across the country have to adequately ensure this populations’ academic achievement, which is directly impacted by English proficiency and ELLs (English Language Learners) program placement. However, restrictive language policies such as Proposition 203, the four-hour English Language Development (ELD) block, and the exclusion of ELLs from Dual Language Programs (DLPs) in Arizona are not effectively preparing linguistic minority and ethnic student populations for academic achievement and competitiveness in a global economy.
For the first part of the analysis, the author examined bilingual education and DLPs policies, access, and practices impacting Latina/o communities by utilizing a case study methodology framework to present the phenomenon of DLPs in a state that by law only supports English only education. The author discussed the case study research design to answer the research questions: (1) Which public k-12 schools are implementing Dual Language Programs (DLPs) in the state of AZ? (2) What are the DLPs’ characteristics? (3) Where are the schools located? (4) What are the stakeholder participants’ perceptions of DLPs and the context in which these DLPs navigate? The author also describe the context of the study, the participants, data, and the data collection process, as well as the analytical techniques she used to make sense of the data and draw findings.
The findings suggest that bilingual education programs in the form of DLPs are being implemented in the state of Arizona despite the English only law of Proposition 203, English for the Children. The growing demand for DLPs is increasing the implementation of such programs, however, language minority students that are classified as ELL are excluded from being part of such programs. Moreover, the findings of the study suggest that although bilingual education is being implemented in Arizona through DLPs, language minority education policy is being negatively influenced by Interest Convergence tenets and Racist Nativist ideology in which the interest of the dominant culture are further advanced to the detriment of minority groups’ interest.
For the first part of the analysis, the author examined bilingual education and DLPs policies, access, and practices impacting Latina/o communities by utilizing a case study methodology framework to present the phenomenon of DLPs in a state that by law only supports English only education. The author discussed the case study research design to answer the research questions: (1) Which public k-12 schools are implementing Dual Language Programs (DLPs) in the state of AZ? (2) What are the DLPs’ characteristics? (3) Where are the schools located? (4) What are the stakeholder participants’ perceptions of DLPs and the context in which these DLPs navigate? The author also describe the context of the study, the participants, data, and the data collection process, as well as the analytical techniques she used to make sense of the data and draw findings.
The findings suggest that bilingual education programs in the form of DLPs are being implemented in the state of Arizona despite the English only law of Proposition 203, English for the Children. The growing demand for DLPs is increasing the implementation of such programs, however, language minority students that are classified as ELL are excluded from being part of such programs. Moreover, the findings of the study suggest that although bilingual education is being implemented in Arizona through DLPs, language minority education policy is being negatively influenced by Interest Convergence tenets and Racist Nativist ideology in which the interest of the dominant culture are further advanced to the detriment of minority groups’ interest.
ContributorsGomez Gonzalez, Laura M (Author) / Jimenez-Silva, Margarita (Thesis advisor) / Amrein-Beardsley, Audrey (Thesis advisor) / Combs, Mary C (Committee member) / Arizona State University (Publisher)
Created2016
Description
The transition from high school to college can cause an undue amount of attrition for fully qualified, college-intending first-generation students. Although the students may have overcome multiple obstacles to be accepted to a college and arrive at the transition over summer, it can feel overwhelming to complete the flood of tasks without access to a supportive network to guide and interpret the intricate steps. Many programs focus on college preparation and access to college but do not devote attention to the delicate transition from access to enrollment during the summer months. The term opportunity melt for students who confirm their enrollment and do not enroll in any institution of higher education in the fall semester. This study identified the influence of strategic peer mentor support during the summer months for Chico State students who applied, were admitted, and accepted their college admission. This action research intervention applies key concepts of academic capital theory and follows up on previous cycles of action research in the California State University system to identify barriers for those who intended to enroll but decided not to attend any Cal State or other institution of higher education in the Fall semester after high school graduation.
ContributorsRyan, Shawn (Author) / Dorn, Sherman (Thesis advisor) / Kim, Jeongeun (Committee member) / Weston, Jeffrey (Committee member) / Arizona State University (Publisher)
Created2022
Description
In this mixed-methods study, I sought to design and develop a test delivery method to reduce linguistic bias in English-based mathematics tests. Guided by translanguaging, a recent linguistic theory recognizing the complexity of multilingualism, I designed a computer-based test delivery method allowing test-takers to toggle between English and their self-identified dominant language. This three-part study asks and answers research questions from all phases of the novel test delivery design. In the first phase, I conducted cognitive interviews with 11 Mandarin Chinese dominant speakers and 11 Spanish speaking dominant undergraduate students while taking a well-regarded calculus conceptual exam, the Precalculus Concept Assessment (PCA). In the second phase, I designed and developed the linguistically adaptive test (LAT) version of the PCA using the Concerto test delivery platform. In the third phase, I conducted a within-subjects random-assignment study of the efficacy the LAT. I also conducted in-depth interviews with a subset of the test-takers.
Nine items on the PCA revealed linguistic issues during the cognitive interviews demonstrating the need to improve the linguistic bias on the test items. Additionally, the newly developed LAT demonstrated evidence of reliability and validity. However, the large-scale efficacy study showed that the LAT did not appear to make a significant difference in scores for dominant speakers of Spanish or dominant speakers of Mandarin Chinese. This finding held true for overall test scores as well as at the item level indicating that the LAT test delivery system does not appear to reduce linguistic bias in testing.
Additionally, in-depth interviews revealed that many students felt that the linguistically adaptive test was either the same or essentially the same as the non-LAT version of the test. Some participants felt that the toggle button was not necessary if they could understand the mathematics item well enough. As one participant noted, “It's math, It's math. It doesn't matter if it's in English or in Spanish.” This dissertation concludes with a discussion about the implications for test developers and suggestions for future direction of study.
ContributorsClose, Kevin (Author) / Zheng, Yi (Thesis advisor) / Amrein-Beardsley, Audrey (Thesis advisor) / Anderson, Kate (Committee member) / Arizona State University (Publisher)
Created2021
Description
The proliferation of fake news on social media has become a concern for manycountries due to its adverse effects on various areas, such as the economy, politics, health,
and society. In light of the growing use of social media in Saudi Arabia, numerous media
outlets actively utilize social media platforms to collect and disseminate news and
information. As a result, Saudi journalists have faced various challenges, including the
spread of fake news. Therefore, this study explores how Saudi journalists define and
verify fake news published on social media and the challenges they face. Furthermore,
this study explores journalists’ role perceptions in society concerning spreading fake
news and how they can promote media literacy to the audience.
This study employed in-depth qualitative interviews with 14 journalists from
various Saudi printing and online newspapers. The thematic analysis of the interviews
showed that Saudi journalists define fake news in several ways, encompassing three
essential elements: source, content, and timing. In addition, the study found that
journalists primarily use traditional verification practices to verify fake news published
on social media, followed by new verification practices. The findings showed that Saudi
journalists face challenges at all levels of the hierarchy of influence model. Moreover, the
findings identify three different roles journalists perceive in society regarding fake news
published on social media: disseminators, populist mobilizers, and interpreters. Lastly,
the study found that journalists lack media literacy knowledge but are willing to
cooperate with other government institutions to promote and distribute media literacy
among the public.
ContributorsBasfar, Majed (Author) / Thornton, Leslie-Jean (Thesis advisor) / Silcock, B. William (Committee member) / Roschke, Kristy (Committee member) / Kim, Jeongeun (Committee member) / Arizona State University (Publisher)
Created2023
Description
The purpose of this study is to explore how internationalization is formed and operationalized in the Intensive English Programs (IEPs) at three Michigan higher education institutions. Drawing from Latour’s (2005) actor-network theory, this study examined the human and non-human actors involved in constructions of internationalization, which was defined as relational processes (programs and policies) that define and deliver international, intercultural, or global elements into the purpose, function and delivery of postsecondary education (Altbach, 2007; Knight, 2003). As an entry point into the study, I focused on the director of the programs and their mission statements, a written articulation of beliefs, as suggested by Childress (2007; 2009).
To explore these potential networks, I utilized Comparative Case Study (Bartlett and Vavrus, 2016), which allowed for more unbounded cases; Actor-Network Theory (Latour, 1999; Latour, 2005) which allowed for agency among non-human actors that also coexist, transform, translate or modify meaning; and relational network analysis methods (Herz et al. 2014; Heath et al. 2009; Clarke 2005), which helped to explore and make sense of complex relational data. This was in the effort to construct an understanding of the “processual, built activities, performed by the actants out of which they are composed” (Crawford, 2004, p. 1). I mapped actors within each site who were performing their local and contingent processes of internationalization.
The results indicate the formation of complex and far reaching webs of actors and activities that accomplish a form of internationalization that is highly localized. While each program under investigation responded to similar pressures, such as funding shortfalls via student enrollment declines, the responses and networks that were created from these constraints were wildly different. Indeed, the study found these programs engaged in international activities that enrolled various external actors, from campus departments to local community groups. In engaging in relational connections that moved beyond their primary instructional purpose, English language instruction and cultural acclimatization, the IEPs in this study were able to 1) contribute to the internationalization of university departmental curricula, 2) serve their communities in dynamic and impactful ways and 3) develop their own sense of internationalization in a university setting.
To explore these potential networks, I utilized Comparative Case Study (Bartlett and Vavrus, 2016), which allowed for more unbounded cases; Actor-Network Theory (Latour, 1999; Latour, 2005) which allowed for agency among non-human actors that also coexist, transform, translate or modify meaning; and relational network analysis methods (Herz et al. 2014; Heath et al. 2009; Clarke 2005), which helped to explore and make sense of complex relational data. This was in the effort to construct an understanding of the “processual, built activities, performed by the actants out of which they are composed” (Crawford, 2004, p. 1). I mapped actors within each site who were performing their local and contingent processes of internationalization.
The results indicate the formation of complex and far reaching webs of actors and activities that accomplish a form of internationalization that is highly localized. While each program under investigation responded to similar pressures, such as funding shortfalls via student enrollment declines, the responses and networks that were created from these constraints were wildly different. Indeed, the study found these programs engaged in international activities that enrolled various external actors, from campus departments to local community groups. In engaging in relational connections that moved beyond their primary instructional purpose, English language instruction and cultural acclimatization, the IEPs in this study were able to 1) contribute to the internationalization of university departmental curricula, 2) serve their communities in dynamic and impactful ways and 3) develop their own sense of internationalization in a university setting.
ContributorsClark, Adam Thomas (Author) / Koro, Mirka (Thesis advisor) / Kim, Jeongeun (Committee member) / Carlson, David L. (Committee member) / Okhremtchouk, Irina S (Committee member) / Vellenga, Heidi (Committee member) / Arizona State University (Publisher)
Created2020
Description
This study applied the Social Cognitive Theory (SCT) to explore the sources of self-efficacy and professional development activities that are most predictive of PreK-6 music teachers’ efficacious beliefs. This study also compared teacher efficacy levels across different groups. The target population for this study was PreK-6 music teachers in the state of Arizona. The survey was disseminated through the National Association for Music Education (NAfME), the Arizona chapters of the American Orff-Schulwerk Association (AOSA), the Organization of American Kodály Educators (OAKE), and snowball sampling via a Facebook message. Of the 660 teachers invited to participate, 92 (13.94%) voluntarily completed the survey. Results from simultaneous multiple regression analyses indicated that teacher efficacy for instructional strategies was best predicted by their mastery experience, followed by vicarious experience, while mastery experience was the strongest predictor of teacher efficacy for student engagement. Additionally, the acquisition of method certification and watching teaching resources via YouTube were significant predictors of teacher efficacy for instructional strategies, while observation hours per year was the only predictor of teacher efficacy for student engagement. Results from factorial between-subjects ANOVAs indicated that teaching experience had a significant main effect on teacher efficacy for instructional strategies and student engagement. However, neither main teaching areas nor the combined effects of main teaching areas and teaching experience had a significant effect on teacher efficacy for instructional strategies and student engagement. Results from independent-samples t-test analyses showed that school types had a significant effect on teacher efficacy for student engagement, while no differences were found between school types regarding teacher efficacy for instructional strategies. The analysis of open-ended comments identified themes related to factors that strengthen or weaken participant teacher efficacy, the impact of the COVID-19 pandemic on teacher efficacy, the types of professional development activities that they engaged during the year, the most effective professional development activities for enhancing teacher efficacy. Findings of this study have theoretical and practical implications for school principals, school administrators, policy makers, music teacher educators, and music teachers to promote and support music teachers’ self-efficacy.
ContributorsCha, Dong-Ju (Author) / Amrein-Beardsley, Audrey (Thesis advisor) / Stauffer, Sandra (Thesis advisor) / Fiorentino, Matthew (Committee member) / Schmidt, Margaret (Committee member) / Arizona State University (Publisher)
Created2023