Building Useful Factors and Scales to Aid in the Assessm ent of - PowerPoint PPT Presentation

Higher Education Research I nstitute at UCLA Building Useful Factors and Scales to Aid in the Assessm ent of Learning Gains and Other Student Outcom es Linda DeAngelo Jessica Sharkness Higher Education Research Institute University of California, Los Angeles Friday, November 14, 2008 Friday, November 14, 2008 CAIR 2008 CAIR 2008 33rd Annual Conference 33rd Annual Conference Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA Higher Education Research I nstitute CI RP Funded Cooperative I nstitutional Research Research Program Freshman YFCY CSS Survey •National Institutes of Health •National Science Foundation •Templeton Foundation Faculty Survey Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA This presentation � General factor analysis overview � Example of creating and refining a factor � Use of factor score in comparing institutions � Use of factor score in examining student experiences and outcomes � Future directions for research at CIRP Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA W hat is Factor Analysis? � Mathematical procedure to analyze interrelationships (correlations) among a set of variables � Can explain the interrelationships in terms of a reduced number of variables – factors � Factors : hypothetical (latent) variables that influence scores on one or more observed variables � Factors represent the “reason” why variables are highly correlated Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA Tw o Kinds of Factor Analysis Exploratory Factor Analysis (EFA) � � Explore the underlying structure of a set of observed variables without imposing a preconceived structure on the outcome Confirm atory Factor Analysis (CFA) � � Allows the researcher to test whether a hypothesized relationship between observed variables and their underlying latent construct(s) exists. The relationship is postulated a priori and then tested statistically. Both analyses tell us whether the responses to a � set of survey questions are organized into clusters, but have different functions Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA Exploratory Factor Analysis Exam ple: Cross-Racial I nteractions ( YFCY) Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA Correlation Matrix 1 2 3 4 5 6 7 8 9 10 1 Dined or shared a meal 1 2 Discussed race/ethnic relations outside class 0.61 1 3 Had guarded, cautious interactions 0.25 0.40 1 4 Shared personal feelings and problems 0.65 0.63 0.29 1 5 Had tense, somewhat hostile interactions 0.18 0.31 0.59 0.26 1 6 Had intellectual discussions outside of class 0.63 0.66 0.27 0.72 0.25 1 7 Felt insulted or threatened because of race/ethnicity 0.12 0.25 0.50 0.16 0.62 0.16 1 8 Studied or prepared for class 0.56 0.52 0.28 0.61 0.27 0.65 0.17 1 9 Socialized or partied 0.60 0.49 0.19 0.60 0.18 0.57 0.11 0.55 1 10 Attended events by other racial/ethnic groups 0.45 0.50 0.28 0.46 0.26 0.47 0.26 0.47 0.48 1 Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA Exploratory Factor Analysis � Three stages: � (1) choose an extraction method � (2) decide the num ber of factors � (3) choose a rotation method Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA Extraction Two common extraction techniques: � Com ponent (In SPSS: Principal Components � Analysis, PCA ) � A data reduction method � Utilizes all of the variance in a set of variables � Most common “True” Factor analysis (In SPSS: Principal Axis � Factoring, PAF ) � Also a data reduction method, but assumes that the variables co-vary in some way � Uses only the shared variance (correlations) of a set of variables to compute the factor solution Some researchers prefer one method, some prefer � the other. Many researchers believe that Principal Components � Analysis is not appropriate for exploratory factor analysis Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA Num ber of Factors: How to decide? � Choose a set of variables � Run a factor analysis using extraction method chosen � Here: Principal Axis Factoring � Examine Scree Plot � Plots Eigenvalues of all possible factors Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA Scree Plot 1 . Look for the natural bend, or break point w here the curve flattens out 2 . The num ber of data points above the break is the num ber of factors to retain Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA Rotation Rotation simplifies and clarifies the underlying � data structure Two common rotation methods: � � Varim ax – orthogonal rotation that assumes uncorrelated factors � Produces cleaner and more easily interpreted results � May not be appropriate for “messy” data of the real world � Prom ax – Oblique rotation method that allows factors to correlate � Produces slightly more complex output to interpret � May more accurately resemble the “real world” If factors are truly uncorrelated, both rotations � will produce nearly identical results Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA Output from both rotational m ethods PAF, Varimax PAF, Promax a Pattern Matrix a Rotated Factor Matrix Factor Factor 1 2 1 2 Had intellectual Had intellectual discussions outside of discussions outside of .825 .141 .848 -.030 class class Shared personal feelings Shared personal feelings .816 .151 .837 -.018 and problems and problems Dined or shared a meal Dined or shared a meal .815 -.079 .783 .086 Socialized or partied Socialized or partied .724 .066 .757 -.087 Studied or prepared for Studied or prepared for .734 .023 .723 .170 class class Had meaningful and Had meaningful and honest discussions about honest discussions about .702 .145 .716 .283 race/ethnic relations race/ethnic relations outside of class outside of class Attended events Attended events sponsored by other sponsored by other .548 .138 .564 .245 racial/ethnic groups racial/ethnic groups Had tense, somewhat Had tense, somewhat -.014 .841 .151 .822 hostile interactions hostile interactions Felt insulted or threatened Felt insulted or threatened because of race/ethnicity -.075 .761 because of race/ethnicity .076 .731 Had guarded, cautious Had guarded, cautious .103 .665 .231 .673 interactions interactions Extraction Method: Principal Axis Factoring. Extraction Method: Principal Axis Factoring. Rotation Method: Promax with Kaiser Normalization. Rotation Method: Varimax with Kaiser Normalization. a. a. Rotation converged in 3 iterations. Rotation converged in 3 iterations. Home of the CIRP The nation’s oldest and largest study of higher education

Higher Education Research I nstitute at UCLA Evaluating the fit of item s in a factor � Cronbach’s Alpha � Commonalities Reliability Statistics Communalities Item-Total Statistics Cronbach's Initial Extraction Alpha N of Items Dined or shared a meal .562 .621 Scale Corrected Cronbach's .901 7 Had meaningful and Scale Mean if Variance if Item-Total Alpha if Item honest discussions about Item Deleted Item Deleted Correlation Deleted .569 .593 race/ethnic relations Dined or shared a meal 17.64 37.177 .735 .884 outside of class Had meaningful and Had guarded, cautious honest discussions about .427 .506 18.30 37.476 .711 .886 interactions race/ethnic relations Shared personal feelings outside of class .630 .689 and problems Shared personal feelings 18.04 36.518 .775 .879 Had tense, somewhat and problems .504 .699 hostile interactions Had intellectual Had intellectual discussions outside of 18.05 36.493 .783 .878 discussions outside of class .647 .700 class Studied or prepared for 18.04 36.900 .701 .888 Felt insulted or threatened class because of race/ethnicity .421 .541 Socialized or partied 17.83 37.885 .684 .889 Attended events Studied or prepared for sponsored by other 18.67 39.581 .574 .901 .514 .552 class racial/ethnic groups Socialized or partied .492 .529 Attended events sponsored by other .366 .378 racial/ethnic groups Extraction Method: Principal Axis Factoring. Home of the CIRP The nation’s oldest and largest study of higher education