Matching Items (2)
Filtering by
- All Subjects: Evaluation
- Creators: Ewbank, Ann
Description
Virtual Patient Simulations (VPS) are web-based exercises involving simulated patients in virtual environments. This study investigates the utility of VPS for increasing medical student clinical reasoning skills, collaboration, and engagement. Many studies indicate that VPS provide medical students with essential practice in clinical decision making before they encounter real life patients. The utility of a recursive, inductive VPS for increasing clinical decision-making skills, collaboration, or engagement is unknown. Following a design-based methodology, VPS were implemented in two phases with two different cohorts of first year medical students: spring and fall of 2013. Participants were 108 medical students and six of their clinical faculty tutors. Students collaborated in teams of three to complete a series of virtual patient cases, submitting a ballpark diagnosis at the conclusion of each session. Student participants subsequently completed an electronic, 28-item Exit Survey. Finally, students participated in a randomized controlled trial comparing traditional (tutor-led) and VPS case instruction methods. This sequence of activities rendered quantitative and qualitative data that were triangulated during data analysis to increase the validity of findings. After practicing through four VPS cases, student triad teams selected accurate ballpark diagnosis 92 percent of the time. Pre-post test results revealed that PPT was significantly more effective than VPS after 20 minutes of instruction. PPT instruction resulted in significantly higher learning gains, but both modalities supported significant learning gains in clinical reasoning. Students collaborated well and held rich clinical discussions; the central phenomenon that emerged was "synthesizing evidence inductively to make clinical decisions." Using an inductive process, student teams collaborated to analyze patient data, and in nearly all instances successfully solved the case, while remaining cognitively engaged. This is the first design-based study regarding virtual patient simulation, reporting iterative phases of implementation and design improvement, culminating in local theories (petite generalizations) about VPS design. A thick, rich description of environment, process, and findings may benefit other researchers and institutions in designing and implementing effective VPS.
ContributorsMcCoy, Lise (Author) / Wetzel, Keith (Thesis advisor) / Ewbank, Ann (Thesis advisor) / Simon, Harvey (Committee member) / Arizona State University (Publisher)
Created2014
Description
ABSTRACT
This study examines validity evidence of a state policy-directed teacher evaluation system implemented in Arizona during school year 2012-2013. The purpose was to evaluate the warrant for making high stakes, consequential judgments of teacher competence based on value-added (VAM) estimates of instructional impact and observations of professional practice (PP). The research also explores educator influence (voice) in evaluation design and the role information brokers have in local decision making. Findings are situated in an evidentiary and policy context at both the LEA and state policy levels.
The study employs a single-phase, concurrent, mixed-methods research design triangulating multiple sources of qualitative and quantitative evidence onto a single (unified) validation construct: Teacher Instructional Quality. It focuses on assessing the characteristics of metrics used to construct quantitative ratings of instructional competence and the alignment of stakeholder perspectives to facets implicit in the evaluation framework. Validity examinations include assembly of criterion, content, reliability, consequential and construct articulation evidences. Perceptual perspectives were obtained from teachers, principals, district leadership, and state policy decision makers. Data for this study came from a large suburban public school district in metropolitan Phoenix, Arizona.
Study findings suggest that the evaluation framework is insufficient for supporting high stakes, consequential inferences of teacher instructional quality. This is based, in part on the following: (1) Weak associations between VAM and PP metrics; (2) Unstable VAM measures across time and between tested content areas; (3) Less than adequate scale reliabilities; (4) Lack of coherence between theorized and empirical PP factor structures; (5) Omission/underrepresentation of important instructional attributes/effects; (6) Stakeholder concerns over rater consistency, bias, and the inability of test scores to adequately represent instructional competence; (7) Negative sentiments regarding the system's ability to improve instructional competence and/or student learning; (8) Concerns regarding unintended consequences including increased stress, lower morale, harm to professional identity, and restricted learning opportunities; and (9) The general lack of empowerment and educator exclusion from the decision making process. Study findings also highlight the value of information brokers in policy decision making and the importance of having access to unbiased empirical information during the design and implementation phases of important change initiatives.
This study examines validity evidence of a state policy-directed teacher evaluation system implemented in Arizona during school year 2012-2013. The purpose was to evaluate the warrant for making high stakes, consequential judgments of teacher competence based on value-added (VAM) estimates of instructional impact and observations of professional practice (PP). The research also explores educator influence (voice) in evaluation design and the role information brokers have in local decision making. Findings are situated in an evidentiary and policy context at both the LEA and state policy levels.
The study employs a single-phase, concurrent, mixed-methods research design triangulating multiple sources of qualitative and quantitative evidence onto a single (unified) validation construct: Teacher Instructional Quality. It focuses on assessing the characteristics of metrics used to construct quantitative ratings of instructional competence and the alignment of stakeholder perspectives to facets implicit in the evaluation framework. Validity examinations include assembly of criterion, content, reliability, consequential and construct articulation evidences. Perceptual perspectives were obtained from teachers, principals, district leadership, and state policy decision makers. Data for this study came from a large suburban public school district in metropolitan Phoenix, Arizona.
Study findings suggest that the evaluation framework is insufficient for supporting high stakes, consequential inferences of teacher instructional quality. This is based, in part on the following: (1) Weak associations between VAM and PP metrics; (2) Unstable VAM measures across time and between tested content areas; (3) Less than adequate scale reliabilities; (4) Lack of coherence between theorized and empirical PP factor structures; (5) Omission/underrepresentation of important instructional attributes/effects; (6) Stakeholder concerns over rater consistency, bias, and the inability of test scores to adequately represent instructional competence; (7) Negative sentiments regarding the system's ability to improve instructional competence and/or student learning; (8) Concerns regarding unintended consequences including increased stress, lower morale, harm to professional identity, and restricted learning opportunities; and (9) The general lack of empowerment and educator exclusion from the decision making process. Study findings also highlight the value of information brokers in policy decision making and the importance of having access to unbiased empirical information during the design and implementation phases of important change initiatives.
ContributorsSloat, Edward F. (Author) / Wetzel, Keith (Thesis advisor) / Amrein-Beardsley, Audrey (Thesis advisor) / Ewbank, Ann (Committee member) / Shough, Lori (Committee member) / Arizona State University (Publisher)
Created2015