The Applicatjon and Promise of Hierarchical Linear Modeling (HLM) in - PowerPoint PPT Presentation

The Applicatjon and Promise of Hierarchical Linear Modeling (HLM) in Studying First-Year Student Programs Chad S. Briggs, Kathie Lorentz & Eric Davis Educatjon & Outreach University Housing Southern Illinois University Carbondale

Southern Illinois University Carbondale • Large doctoral research public university located in the southern tjp of Illinois • Six hours from Chicago • Rural community • Large number of students from the Chicago land area

Enrollment • On–Campus Enrollment at the tjme of study – 19,124 • On-Campus Residence Hall Enrollment at the tjme of study – 4,314 students

Primary Purpose and Applicatjons for Hierarchical Linear Modeling (HLM) • HLM allows us to assess and model the variable efgects of context or environment • Example Housing and FYE Applicatjons – Students nested within • Classrooms (e.g., freshman seminars) • Programs (e.g., LLCs or Peer Mentoring) • Residence halls and fmoors • Universitjes (cross-instjtutjonal research)

Additjonal Applicatjons for HLM • Item Analysis – Items nested within respondents • Growth Modeling or Longitudinal Research – Observatjons over tjme nested within students • Cross-Classifjcatjon – Students nested within more than one group (e.g., fmoors and classrooms, or difgerent living environments across tjme) • Meta-Analysis – Coeffjcients nested within studies

Failing to Capture the Social Ecology of First-Year Experience Programs • Partjcipatjon in FYE programs typically takes place within a group context AND this context ofuen infmuences individual outcomes – In fact, Living-Learning Communitjes (LLCs) are designed to capitalize on the resources and dynamics of group membership to yield desired outcomes (e.g., GPA and persistence) • Yet, FYE and Housing evaluatjon efgorts rarely use HLM to model the infmuence of these “context efgects”

Literature Review • Located Housing and First-Year Experience (FYE) artjcles that used HLM in their analysis – 4 major databases were searched • EBSCO, ERIC JSTOR and MUSE – Keywords for HLM, Housing and First-Year Experience programs were cross-referenced in each database • Results – Just 5 artjcles* were found * Please contact presenters for references

Traditjonal Methods of Modeling Context (or Group-Level) Efgects • Disaggregatjon of Group Characteristjcs – Group characteristjcs are assigned to everyone in a group – Violates assumptjon of independence • Aggregatjon of Individual Characteristjcs – Mean individual characteristjcs assigned to group – Loss of sample size, within-group variability and power • Consequence – Biased estjmates of efgect  inaccurate/misleading results

Hierarchical Linear Modeling (HLM) • HLM allows us to obtain unbiased estjmates of efgect for group context variables • Hierarchical – Indicates that Level 1 (or student-level) coeffjcients become outcomes at Level 2 (group- level)

Regression Refresher Y-axis (DV) Random Slope error Intercept X-axis (IV) Predicted    Y B B ( X ) r outcome for i o 1 i i Student “i”

Linear Model Terminology • Where, – Y ij = Outcome for person i in group j – β 0j = Intercept for group j • Value of Y when X = mean of group j X ij  X .. • If X is centered around the grand mean ( ), then the intercept equals the value of Y for a person with an X equal to the average X across all groups – β 1j = Slope for group j • Change in Y associated with a 1 unit change in X X  X – = group-mean centered value of Level 1 variable for person i in . j ij group j – r ij = random error term (predicted Y ij – observed Y ij ) for person i in group j • r ij ~ N(0,σ 2 ), or • r ij is assumed normally distributed with a mean of “0” and a constant variance equal to sigma-squared.

Regression with Multjple Groups • Two Group Case    Y B B ( X ) r i 1 01 11 i 1 i 1    Y B B ( X ) r i 2 02 12 i 2 i 2 • J Group Case       Y ( X X ) r . j ij 0 j 1 j ij ij

Two-Level Model • Level 1 Equatjon       Y ( X X ) r . j ij 0 j 1 j ij ij • Level 2 Equatjons       W u 0 j 00 01 j 0 j       W u 1 j 10 11 j 1 j

Mixed Two-Level Model Grand Main Efgects of W j and X ij Mean        Y ( W ) ( X X ) ij 00 01 j 10 i j . j Interactjon Efgect of W j and X ij    W ( Xi X ) . j 11 j j Random Error Terms for Intercept, Slope and Student       ( X X ) r . j 0 j 1 j i ij j

Two-Level Model Terminology • Generally,  – = Grand Intercept (mean of Y across all groups) 00  – = avg. difgerence between Grand Intercept and 01 Intercept for group j given W j  – = Grand slope (slope across all groups) 10  – = avg. difgerence between Grand Slope and slope 11 for group j given W j and X ij – µ 0j = random deviatjon of group means about the grand intercept – µ 1j = random deviatjon of group slopes about the grand slope – W j = Level 2 predictor for group j

Variance-Covariance Components • Var(r ij ) = σ 2 • Within-group variability • Var(µ 0j ) = τ 00 • Variability about grand mean • Var(µ 1j ) = τ 11 • Variability about grand slope • Cov(µ 0j , µ 1j ) = τ 01 • Covariance of slopes and intercepts

EXAMPLE ANALYSIS

Background on Housing Hierarchy • University Housing Halls/Floors – Brush Towers • 32 Floors • 2 Halls – Thompson Point • 33 Floors • 11 Halls – University Park • 56 Floors • 11 Halls • Totals – 121 Floors – 24 Halls

Living-Learning Community (LLC) Program at SIUC in 2005 • LLC Program Components – Academic/Special Emphasis Floors (AEFs) • First Implemented in 1996 • 12 Academic/Special Emphasis Floors – Freshman Interest Groups (FIGs) • First implemented in 2001 • 17 FIGs ofgered • Some FIGs were nested on AEFs

Data Collectjon • All example data were collected via university records • Sampling – LLC Students (n = 421) • All FIG students (n = 223) • Random sample of AEF students (n = 147) • All FIG students nested on AEFs (n = 51) – Random sample of Comparison students (n = 237)

2005 Cohort Sample Demographics Percent African Percent Percent Mean American Group N Female White ACT LLC 421 38% 64% 25% 22.44 Comparison 237 46% 57% 36% 21.21 Total 658 41% 62% 31% 22.00

Hierarchical 2-Level Dataset • Level 1 – 657 Students • Level 2 – 93 Floors • 1 to 31 students (mean = 7) populated each fmoor • Low n-sizes per fmoor are not ideal, but HLM makes it possible to estjmate coeffjcients with some accuracy via Bayes estjmatjon.

Variables Included in Example Analysis • Outcome (Y) – First-Semester GPA (Fall 2005) • Student-Level Variables (X’s) – Student ACT score – Student LLC Program Partjcipatjon (LLC student = 1, Other = 0) • Floor-Level Variables (W’s) – MEAN_ACT (average fmoor ACT score) – LLC Partjcipatjon Rate (Percent LLC students on fmoor)

Research Questjons Addressed in Example Analysis • How much do residence hall fmoors vary in terms of fjrst- semester GPA? • Do fmoors with high MEAN ACT scores also have high fjrst-semester GPAs? • Does the strength of the relatjonship between the student-level variables (e.g., student ACT) and GPA vary across fmoors? • Are ACT efgects greater at the student- or fmoor-level? • Does partjcipatjon in an LLC (and partjcipatjon rate per fmoor) infmuence fjrst-semester GPA afuer controlling for student- and fmoor-level ACT? • Are there any student-by-environment interactjons?

Model Building • HLM involves fjve model building steps: 1. One-Way ANOVA with random efgects 2. Means-as-Outcomes 3. One-Way ANCOVA with random efgects 4. Random Intercepts-and-Slopes 5. Intercepts-and-Slopes-as-Outcomes

One-Way ANOVA with Random Efgects LEVEL 1 MODEL F05GPA ij = β 0j + r ij Level-1 Slope is set equal to 0. LEVEL 2 MODEL β 0j = γ 00 + u 0j MIXED MODEL F05GPA ij = γ 00 + u 0j + r ij

ANOVA Results and Auxiliary Statjstjcs • Point Estjmate for Grand Mean  – = 2.59*** 00 • Variance Components – σ 2 = .93 – τ 00 = .05* • Because τ 00 is signifjcant, HLM is appropriate • Auxiliary Statjstjcs – Plausible Range of Floor Means  00 • 95%CI = +- 1.96(τ 00 ) ½ = 2.13 to 3.05 – Intraclass Correlatjon Coeffjcient (ICC) • ICC = .06 (6% of the var. in fjrst-semester GPA is between fmoors) – Reliability (of sample means) • λ =.24

Means-as-Outcomes LEVEL 1 MODEL F05GPA ij = β 0j + r ij LEVEL 2 MODEL Level-2 Predictor Added β 0j = γ 00 + γ 01 (MEAN_ACT j ) + u 0j to Intercept Model MIXED MODEL F05GPA ij = γ 00 + γ 01 ∗ MEAN_ACT j + u 0j + r ij

Means-as-Outcomes Results and Auxiliary Statjstjcs • Fixed Coeffjcients  – = 0.92*** (efgect of fmoor-level ACT) 01 • Variance Components – σ 2 = .93 – τ 00 = .04 (ns, p = .11) • Auxiliary Statjstjcs – Proportjon Reductjon in Variance (PRV) • PRV = .28 (MEAN ACT accounted for 28% of between-group var.) – Conditjonal ICC • ICC = .04 (Remaining unexplained variance between fmoors = 4%) – Conditjonal Reliability • λ = .20 (reliability with which we can discriminate among fmoors with identjcal MEAN ACT values)