Matching Items (42)

390-Thumbnail Image.png

'Use it or Lose It' Professional Judgment: Educational Evaluation and Bayesian Reasoning

Description

This paper presents a Bayesian framework for evaluative classification. Current education policy debates center on arguments about whether and how to use student test score data in school and personnel

This paper presents a Bayesian framework for evaluative classification. Current education policy debates center on arguments about whether and how to use student test score data in school and personnel evaluation. Proponents of such use argue that refusing to use data violates both the public’s need to hold schools accountable when they use taxpayer dollars and students’ right to educational opportunities. Opponents of formulaic use of test-score data argue that most standardized test data is susceptible to fatal technical flaws, is a partial picture of student achievement, and leads to behavior that corrupts the measures.

A Bayesian perspective on summative ordinal classification is a possible framework for combining quantitative outcome data for students with the qualitative types of evaluation that critics of high-stakes testing advocate. This paper describes the key characteristics of a Bayesian perspective on classification, describes a method to translate a naïve Bayesian classifier into a point-based system for evaluation, and draws conclusions from the comparison on the construction of algorithmic (including point-based) systems that could capture the political and practical benefits of a Bayesian approach. The most important practical conclusion is that point-based systems with fixed components and weights cannot capture the dynamic and political benefits of a reciprocal relationship between professional judgment and quantitative student outcome data.

Contributors

Agent

Created

Date Created
  • 2009

151988-Thumbnail Image.png

Empowering apprentice teachers: tracking instructional practices with MyiLOGS

Description

Growing popularity of alternatively certifying teachers has created challenges for teacher preparation programs. Many non-traditional routes into classroom include no full-time mentor teacher. Absence of a mentor teacher in the

Growing popularity of alternatively certifying teachers has created challenges for teacher preparation programs. Many non-traditional routes into classroom include no full-time mentor teacher. Absence of a mentor teacher in the classroom leaves teachers with a deficit. This study follows ten teachers on the intern certificate enrolled in both an alternative certification teacher preparation program and the Teach for America organization as they pursue a master's degree in education and state teaching certification from a large southwestern university. The five randomly chosen for the treatment group and the control group contained 1 male and 4 female teachers, some of whom teach at public schools and others at charter schools. All were secondary education language arts teachers ranging in age from 22- 29. The treatment used in this study is a job-embedded, professional development, software tool designed to help teachers track their classroom practices called MyiLOGS. The purpose of this action research project was to study the effect using MyiLOGS had on six of the nine areas evaluated by a modified version of the Teacher Advancement Program evaluation rubric, alignment with Opportunity To Learn constructs, and the tool's influence on the efficacy of these first year teachers. The data generated from this study indicate that the MyiLOGS tool did have a positive effect on the teachers' TAP evaluation performances. Also, the MyiLOGS tool had a large impact on the teachers' instruction as measured by the constructs of Opportunity to Learn and their teaching self-efficacy. Implications suggested the tool was an asset to these teachers because they tracked their data, became more reflective, and self-sufficient.

Contributors

Agent

Created

Date Created
  • 2013

156658-Thumbnail Image.png

Implications of Learning Outcomes of In-Person and Virtual Field-Based Geoscience Instruction at Grand Canyon National Park

Description

Education through field exploration is fundamental in geoscience. But not all students enjoy equal access to field-based learning because of time, cost, distance, ability, and safety constraints. At the same

Education through field exploration is fundamental in geoscience. But not all students enjoy equal access to field-based learning because of time, cost, distance, ability, and safety constraints. At the same time, technological advances afford ever more immersive, rich, and student-centered virtual field experiences. Virtual field trips may be the only practical options for most students to explore pedagogically rich but inaccessible places. A mixed-methods research project was conducted on an introductory and an advanced geology class to explore the implications of learning outcomes of in-person and virtual field-based instruction at Grand Canyon National Park. The study incorporated the Great Unconformity in the Grand Canyon, a 1.2 billion year break in the rock record; the Trail of Time, an interpretive walking timeline; and two immersive, interactive virtual field trips (iVFTs). The in-person field trip (ipFT) groups collectively explored the canyon and took an instructor-guided inquiry hike along the interpretive Trail of Time from rim level, while iVFT students individually explored the canyon and took a guided-inquiry virtual tour of Grand Canyon geology from river level. High-resolution 360° spherical images anchor the iVFTs and serve as a framework for programmed overlays that enable interactivity and allow the iVFT to provide feedback in response to student actions. Students in both modalities received pre- and post-trip Positive and Negative Affect Schedules (PANAS). The iVFT students recorded pre- to post-trip increases in positive affect (PA) scores and decreases in negative (NA) affect scores, representing an affective state conducive to learning. Pre- to post-trip mean scores on concept sketches used to assess visualization and geological knowledge increased for both classes and modalities. However, the iVFT pre- to post-trip increases were three times greater (statistically significant) than the ipFT gains. Both iVFT and ipFT students scored 92-98% on guided-inquiry worksheets completed during the trips, signifying both met learning outcomes. Virtual field trips do not trump traditional in-person field work, but they can meet and/or exceed similar learning objectives and may replace an inaccessible or impractical in-person field trip.

Contributors

Agent

Created

Date Created
  • 2018

158195-Thumbnail Image.png

Training Future Entrepreneurs – Developing and Assessing Sustainability Competencies in Entrepreneurship Education

Description

Employee-owned businesses, benefit corporations, social enterprises, and other sustainability entrepreneurship innovations are responding to challenges such as climate change, economic inequalities, and unethical business behavior. Academic programs to date, however,

Employee-owned businesses, benefit corporations, social enterprises, and other sustainability entrepreneurship innovations are responding to challenges such as climate change, economic inequalities, and unethical business behavior. Academic programs to date, however, often fall short in sufficiently equipping students with competencies in sustainability entrepreneurship – from a coherent set of learning objectives, through effective and engaging pedagogies, to rigorous assessment of learning outcomes. This dissertation contributes to bridging these gaps. The first study proposes a process-oriented and literature-based framework of sustainability entrepreneurship competencies. It offers a general vision for students, faculty, and entrepreneurs, as well as for the design of curricula, courses, and assessments. The second study presents an exploration into the nature of sustainability entrepreneurship courses, with a focus on teaching and learning processes. Using pioneering courses at Arizona State University, the study analyzes and compares the links between learning objectives, pedagogies, and learning outcomes. Based on document analysis and semi-structured interviews with course instructors, the study identifies cognitive apprenticeship from input processing to experimentation, constructive alignment from learning objectives to assessments, and curriculum-level coordination across courses as key success factors of sustainability entrepreneurship education. The result of this study can inform instructors and researchers in applying and further substantiating effective educational models for future entrepreneurs. The third study addresses the key question of competence assessment: what are reliable tools for assessing students’ competence in sustainability entrepreneurship? This study developed and tested a novel tool for assessing students’ competence in sustainability entrepreneurship through in-vivo simulated professional situations. The tool was in different settings and evaluated against a set of criteria derived from the literature. To inform educators in business and management programs, this study discusses and concludes under which conditions this assessment tool seems most effective, as well as improvement for future applications of the tool.

Contributors

Agent

Created

Date Created
  • 2020

150772-Thumbnail Image.png

Implementing a sustainable program evaluation component at a large university counseling center

Description

This action research dissertation study was undertaken to establish the foundation of a comprehensive evaluation component for the Turn-It-Around (TIA) workshop intervention program at Arizona State University (ASU), and was

This action research dissertation study was undertaken to establish the foundation of a comprehensive evaluation component for the Turn-It-Around (TIA) workshop intervention program at Arizona State University (ASU), and was delivered in the form of a program development consultation. The study's intent was to enhance the ASU Counseling Service's departmental capacity to evaluate one of its important clinical services. The outcomes of this study included multiple assessments of TIA's evaluability and the fidelity of its implementation to its program design. The study products include a well-articulated program theory comprised of program goals, learning objectives, a detailed description of program activities, a logic model, and theoretical construct checklist documents articulating the behavioral science theory underlying the TIA intervention. In addition, instruments tailored to the Turn-It-Around intervention that are suitable for assessing program outcomes were developed and are implementation ready. TIA's clinical stakeholders were interviewed following the generation and delivery of the products and instruments mentioned above to determine whether they found the study's processes and products to be worthwhile and useful. In general, the clinicians reported that they were very satisfied with the benefits and outcomes of the program development consultation. As an action research dissertation, this study generated useful and usable collateral materials in the form of reports, documents, and models. These products are now at the disposal of TIA's institutional stakeholders for use in day-to-day business activities such as training new facilitators and liaisons, and giving presentations that describe the usefulness of TIA as an intervention. Beyond the documents generated to form a program evaluation infrastructure for Turn-It-Around, the processes involved in crafting the documents served to engage relevant stakeholders in a cycle of action research that enriched and solidified their understandings of TIA and furnished them with insight into their counterparts' thinking about the intervention and its potential to benefit the college students they are responsible for helping. Consistent with the intent of action research, the processes involved in accomplishing the objectives of this study surfaced new topics and questions that will be useful in subsequent cycles of program improvement.  

Contributors

Agent

Created

Date Created
  • 2012

150855-Thumbnail Image.png

Implementing a K-12 train the trainer professional development model through the school improvement grant

Description

Effective professional development has been shown to improve instruction and increase student academic achievement. The Train the Trainer professional development model is often chosen by the state Department of Education

Effective professional development has been shown to improve instruction and increase student academic achievement. The Train the Trainer professional development model is often chosen by the state Department of Education for its efficiency and cost effectiveness of delivering training to schools and districts widely distributed throughout the state. This is a study of the Train the Trainer component of an innovative K12 professional development model designed to meet the needs of the state's lowest performing schools that served some of the state's most marginalized students. Pursuing a Vygotzkian social constructivist framework, the model was developed and informed by its stakeholders, providing training that was collaborative, job-embedded, ongoing, and continuously adapted to meet the needs of the School Improvement Grant participants. Schools in the multi-case study were awarded the federal ARRA School Improvement Grant in 2010. Focus questions include: What influence does the Train the Trainer component have on classroom instruction specifically as it relates to formative assessment? and To what extent does the trainer support the implementation of the Train the Trainer professional development at the classroom level? The action research study took place from August 2011 to February 2012 and used a mixed-methods research design.

Contributors

Agent

Created

Date Created
  • 2012

150989-Thumbnail Image.png

Curriculum improvement in education for sustainable development: measuring learning outcomes in an introductory urban planning course

Description

Education for sustainable development (ESD) is an academic goal for many courses in higher learning. ESD encompasses a specific range of learning outcomes, competencies, skills and literacies that include and

Education for sustainable development (ESD) is an academic goal for many courses in higher learning. ESD encompasses a specific range of learning outcomes, competencies, skills and literacies that include and exceed the acquisition of content knowledge. Methods and case studies for measuring learning outcomes in ESD is absent from the literature. This case study of an undergraduate course in urban sustainability examines the processes, curriculum, pedagogies, and methods to explore whether or not learning outcomes in education for sustainable development are being reached. Observations of the course, and the statistical analysis of student surveys from course evaluations, are explored to help identify the relationships between learning outcomes in ESD and the processes of learning and teaching in the case study. Recommendations are made for applying the lessons of the case study to other courses, and for continuing further research in this area.

Contributors

Agent

Created

Date Created
  • 2012

151477-Thumbnail Image.png

Houston, we have a problem: studying the SAS Education Value-Added Assessment System (EVAAS) from teachers' perspectives in the Houston Independent School District (HISD)

Description

This study examined the intended and unintended consequences associated with the Education Value-Added Assessment System (EVAAS) as perceived and experienced by teachers in the Houston Independent School District (HISD). To

This study examined the intended and unintended consequences associated with the Education Value-Added Assessment System (EVAAS) as perceived and experienced by teachers in the Houston Independent School District (HISD). To evaluate teacher effectiveness, HISD is using EVAAS for high-stakes consequences more than any other district or state in the country. A large-scale electronic survey was used to investigate the model's reliability and validity; to determine whether teachers used the EVAAS data in formative ways as intended; to gather teachers' opinions on EVAAS's claimed benefits and statements; and to understand the unintended consequences that occurred as a result of EVAAS use in HISD. Mixed methods data collection and analyses were used to present the findings in user-friendly ways, particularly when using the words and experiences of the teachers themselves. Results revealed that the reliability of the EVAAS model produced split and inconsistent results among teacher participants, and teachers indicated that students biased the EVAAS results. The majority of teachers did not report similar EVAAS and principal observation scores, reducing the criterion-related validity of both measures of teacher quality. Teachers revealed discrepancies in the distribution of EVAAS reports, the awareness of trainings offered, and among principals' understanding of EVAAS across the district. This resulted in an underwhelming number of teachers who reportedly used EVAAS data for formative purposes. Teachers disagreed with EVAAS marketing claims, implying the majority did not believe EVAAS worked as intended and promoted. Additionally, many unintended consequences associated with the high-stakes use of EVAAS emerged through teachers' responses, which revealed among others that teachers felt heightened pressure and competition, which reduced morale and collaboration, and encouraged cheating or teaching to the test in attempt to raise EVAAS scores. This study is one of the first to investigate how the EVAAS model works in practice and provides a glimpse of whether value-added models might produce desired outcomes and encourage best teacher practices. This is information of which policymakers, researchers, and districts should be aware and consider when implementing the EVAAS, or any value-added model for teacher evaluation, as many of the reported issues are not specific to the EVAAS model.

Contributors

Agent

Created

Date Created
  • 2012

150519-Thumbnail Image.png

Implementing a standards-based teacher evaluation system: learning experiences for administrators in an urban school district

Description

Policymakers at the national level have recently initiated K-12 education reforms focused on teacher quality and teacher evaluation. Far-reaching legislation was subsequently enacted in the state of Arizona requiring schools

Policymakers at the national level have recently initiated K-12 education reforms focused on teacher quality and teacher evaluation. Far-reaching legislation was subsequently enacted in the state of Arizona requiring schools to adopt standards-based teacher evaluation systems and link them to student outcomes. The end product is to result in annual summative measures of teacher effectiveness. Because of this, Arizona school administrators have become concerned about rapidly becoming experts in high-stakes teacher evaluation. Principals rarely have time to come together to talk about teacher evaluation, and consider the reliability of their evaluations and how to use teacher evaluation to help teachers improve their practice. This action research study focused on a group of nine administrators in a small urban district grappling with a more complex and high-stakes teacher evaluation system. An existing community of practice was engaged to help administrators become more effective, fair, and consistent evaluators. Activities were designed to engage the group in dynamic, contextualized learning. Participants interacted in small groups to interpret the meaning of newly adopted evaluation instruments and professional teaching standards, share practical knowledge, and compare teacher evaluation experiences in classrooms. Data were gathered with mixed methods. Prior to, and immediately after engaging in this 20-week innovation, principals and district administrators were given two surveys and interviewed about teacher evaluation. Additionally, a detailed record of this project was kept in the form of meeting records and a research journal. Quantitative and qualitative data were triangulated to validate findings. Results identified concerns and understandings of administrators as they attempted to come to a shared consensus regarding teacher evaluation, increase inter-rater reliability, and use teacher evaluation to improve professional practice. As a result of working and learning together administrators lowered their concerns about inter-rater reliability. Other concerns, however, remained and grew. Administrators found the process of gaining a common understanding of teacher evaluation to be complex and far more time-consuming than anticipated. Intense concerns about alignment of the evaluation system with other reforms led these administrators to consider modifications in their evaluation system. Implications from this study can be used to help other administrators grappling with common concerns.

Contributors

Agent

Created

Date Created
  • 2012

153078-Thumbnail Image.png

Investigating a teacher evaluation system: school administrator and teacher perceptions of the system's standards of effectiveness

Description

Increasing public criticism of traditional teacher evaluation systems based largely on classroom observations has spurred an unprecedented shift in the debate surrounding educational accountability policies, specifically about the purposes for

Increasing public criticism of traditional teacher evaluation systems based largely on classroom observations has spurred an unprecedented shift in the debate surrounding educational accountability policies, specifically about the purposes for and measures used to evaluate teachers. In response to growing public demand and associated federal mandates, states have been prompted to design and implement teacher evaluation systems that use increasingly available, statistically complex models (i.e., value-added) intended to isolate and measure the effects of individual teachers on student academic growth over time. The purpose of this study was to examine the perceptions of school administrators and teachers within one of the largest school districts in the state of Arizona with regards to the design and implementation of a federally-supported, state policy-directed teacher evaluation system based on professional practice and value-added measures. While much research has been conducted on teacher evaluation, few studies have examined teacher evaluation systems in context to better understand the standards of effectiveness used by school administrators and teachers to measure system effectiveness. The perceptions of school administrators and teachers, considering their lived experiences as the subjects of the nation's new and improved teacher evaluation systems in context, must be better understood if state and federal policymakers are to also better recognize and understand the consequences (intended and unintended) associated with the design and implementation of these systems in practice.

Contributors

Agent

Created

Date Created
  • 2014