Matching Items (6)
Filtering by
- All Subjects: Educational tests & measurements
- Creators: Green, Samuel
Description
This simulation study compared the utility of various discrepancy measures within a posterior predictive model checking (PPMC) framework for detecting different types of data-model misfit in multidimensional Bayesian network (BN) models. The investigated conditions were motivated by an applied research program utilizing an operational complex performance assessment within a digital-simulation educational context grounded in theories of cognition and learning. BN models were manipulated along two factors: latent variable dependency structure and number of latent classes. Distributions of posterior predicted p-values (PPP-values) served as the primary outcome measure and were summarized in graphical presentations, by median values across replications, and by proportions of replications in which the PPP-values were extreme. An effect size measure for PPMC was introduced as a supplemental numerical summary to the PPP-value. Consistent with previous PPMC research, all investigated fit functions tended to perform conservatively, but Standardized Generalized Dimensionality Discrepancy Measure (SGDDM), Yen's Q3, and Hierarchy Consistency Index (HCI) only mildly so. Adequate power to detect at least some types of misfit was demonstrated by SGDDM, Q3, HCI, Item Consistency Index (ICI), and to a lesser extent Deviance, while proportion correct (PC), a chi-square-type item-fit measure, Ranked Probability Score (RPS), and Good's Logarithmic Scale (GLS) were powerless across all investigated factors. Bivariate SGDDM and Q3 were found to provide powerful and detailed feedback for all investigated types of misfit.
ContributorsCrawford, Aaron (Author) / Levy, Roy (Thesis advisor) / Green, Samuel (Committee member) / Thompson, Marilyn (Committee member) / Arizona State University (Publisher)
Created2014
Description
Institutions of higher education often tout that they are developing students to become lifelong learners. Evaluative efforts in this area have been presumably hindered by the lack of a uniform conceptualization of lifelong learning. Lifelong learning has been defined from institutional, economic, socio-cultural, and pedagogical perspectives, among others. This study presents the existing operational definitions and theories of lifelong learning in the context of higher education and synthesizes them to propose a unified model of college students' orientation toward lifelong learning. The model theorizes that orientation toward lifelong learning is a latent construct which manifests as students' likelihood to engage in four types of learning activities: formal work-related activities, informal work-related activities, formal personal interest activities, and informal personal interest activities. The Postsecondary Orientation toward Lifelong Learning scale (POLL) was developed and the validity of the resulting score interpretations was examined. The instrument was used to compare potential differences in orientation toward lifelong learning between freshmen and seniors. Exploratory factor analyses of the responses of 138 undergraduate college students in the pilot study data provided tentative support for the factor structure within each type of learning activity. Guttman's <λ>λ2 estimates of the learning activity subscales ranged from .78 to .85. Follow-up confirmatory factor analysis using structural equation modeling did not corroborate support for the hypothesized four-factor model using the main student sample data of 405 undergraduate students. Several alternative reflective factor structures were explored. A two-factor model representing factors for Instructing/Presenting and Reading learning activities produced marginal model-data fit and warrants further investigation. The summed POLL total scores had a relatively strong positive correlation with global interest in learning (.58), moderate positive correlations with civic engagement and participation (.38) and life satisfaction (.29), and a small positive correlation with social desirability (.15). The results of the main study do not provide support for the malleability of postsecondary students' orientation toward lifelong learning, as measured by the summed POLL scores. The difference between freshmen and seniors' average total POLL scores was not statistically significant and was negligible in size.
ContributorsArcuria, Phil (Author) / Thompson, Marilyn (Thesis advisor) / Green, Samuel (Committee member) / Levy, Roy (Committee member) / Arizona State University (Publisher)
Created2011
Modeling motivation: examining the structural validity of the Sport Motivation Scale-6 among runners
Description
Two models of motivation are prevalent in the literature on sport and exercise participation (Deci & Ryan, 1991; Vallerand, 1997, 2000). Both models are grounded in self-determination theory (Deci & Ryan, 1985; Ryan & Deci, 2000) and consider the relationship between intrinsic, extrinsic, and amotivation in explaining behavior choice and outcomes. Both models articulate the relationship between need satisfaction (i.e., autonomy, competence, relatedness; Deci & Ryan, 1985, 2000; Ryan & Deci, 2000) and various cognitive, affective, and behavioral outcomes as a function of self-determined motivation. Despite these comprehensive models, inconsistencies remain between the theories and their practical applications. The purpose of my study was to examine alternative theoretical models of intrinsic, extrinsic, and amotivation using the Sport Motivation Scale-6 (SMS-6; Mallett et al., 2007) to more thoroughly study the structure of motivation and the practical utility of using such a scale to measure motivation among runners. Confirmatory factor analysis was used to evaluate eight alternative models. After finding unsatisfactory fit of these models, exploratory factor analysis was conducted post hoc to further examine the measurement structure of motivation. A three-factor structure of general motivation, external accolades, and isolation/solitude explained motivation best, although high cross-loadings of items suggest the structure of this construct still lacks clarity. Future directions to modify item content and re-examine structure as well as limitations of this study are discussed.
ContributorsKube, Erin (Author) / Thompson, Marilyn (Thesis advisor) / Tracey, Terence (Thesis advisor) / Green, Samuel (Committee member) / Arizona State University (Publisher)
Created2012
Description
The existing minima for sample size and test length recommendations for DIMTEST (750 examinees and 25 items) are tied to features of the procedure that are no longer in use. The current version of DIMTEST uses a bootstrapping procedure to remove bias from the test statistic and is packaged with a conditional covariance-based procedure called ATFIND for partitioning test items. Key factors such as sample size, test length, test structure, the correlation between dimensions, and strength of dependence were manipulated in a Monte Carlo study to assess the effectiveness of the current version of DIMTEST with fewer examinees and items. In addition, the DETECT program was also used to partition test items; a second feature of this study also compared the structure of test partitions obtained with ATFIND and DETECT in a number of ways. With some exceptions, the performance of DIMTEST was quite conservative in unidimensional conditions. The performance of DIMTEST in multidimensional conditions depended on each of the manipulated factors, and did suggest that the minima of sample size and test length can be made lower for some conditions. In terms of partitioning test items in unidimensional conditions, DETECT tended to produce longer assessment subtests than ATFIND in turn yielding different test partitions. In multidimensional conditions, test partitions became more similar and were more accurate with increased sample size, for factorially simple data, greater strength of dependence, and a decreased correlation between dimensions. Recommendations for sample size and test length minima are provided along with suggestions for future research.
ContributorsFay, Derek (Author) / Levy, Roy (Thesis advisor) / Green, Samuel (Committee member) / Gorin, Joanna (Committee member) / Arizona State University (Publisher)
Created2012
Description
Investigation of measurement invariance (MI) commonly assumes correct specification of dimensionality across multiple groups. Although research shows that violation of the dimensionality assumption can cause bias in model parameter estimation for single-group analyses, little research on this issue has been conducted for multiple-group analyses. This study explored the effects of mismatch in dimensionality between data and analysis models with multiple-group analyses at the population and sample levels. Datasets were generated using a bifactor model with different factor structures and were analyzed with bifactor and single-factor models to assess misspecification effects on assessments of MI and latent mean differences. As baseline models, the bifactor models fit data well and had minimal bias in latent mean estimation. However, the low convergence rates of fitting bifactor models to data with complex structures and small sample sizes caused concern. On the other hand, effects of fitting the misspecified single-factor models on the assessments of MI and latent means differed by the bifactor structures underlying data. For data following one general factor and one group factor affecting a small set of indicators, the effects of ignoring the group factor in analysis models on the tests of MI and latent mean differences were mild. In contrast, for data following one general factor and several group factors, oversimplifications of analysis models can lead to inaccurate conclusions regarding MI assessment and latent mean estimation.
ContributorsXu, Yuning (Author) / Green, Samuel (Thesis advisor) / Levy, Roy (Committee member) / Thompson, Marilyn (Committee member) / Arizona State University (Publisher)
Created2018
Description
The use of exams for classification purposes has become prevalent across many fields including professional assessment for employment screening and standards based testing in educational settings. Classification exams assign individuals to performance groups based on the comparison of their observed test scores to a pre-selected criterion (e.g. masters vs. nonmasters in dichotomous classification scenarios). The successful use of exams for classification purposes assumes at least minimal levels of accuracy of these classifications. Classification accuracy is an index that reflects the rate of correct classification of individuals into the same category which contains their true ability score. Traditional methods estimate classification accuracy via methods which assume that true scores follow a four-parameter beta-binomial distribution. Recent research suggests that Item Response Theory may be a preferable alternative framework for estimating examinees' true scores and may return more accurate classifications based on these scores. Researchers hypothesized that test length, the location of the cut score, the distribution of items, and the distribution of examinee ability would impact the recovery of accurate estimates of classification accuracy. The current simulation study manipulated these factors to assess their potential influence on classification accuracy. Observed classification as masters vs. nonmasters, true classification accuracy, estimated classification accuracy, BIAS, and RMSE were analyzed. In addition, Analysis of Variance tests were conducted to determine whether an interrelationship existed between levels of the four manipulated factors. Results showed small values of estimated classification accuracy and increased BIAS in accuracy estimates with few items, mismatched distributions of item difficulty and examinee ability, and extreme cut scores. A significant four-way interaction between manipulated variables was observed. In additional to interpretations of these findings and explanation of potential causes for the recovered values, recommendations that inform practice and avenues of future research are provided.
ContributorsKunze, Katie (Author) / Gorin, Joanna (Thesis advisor) / Levy, Roy (Thesis advisor) / Green, Samuel (Committee member) / Arizona State University (Publisher)
Created2013