PISA DATA 2012: Calculating variance by ESCS

[Jenny Michelle]

Hi there! I was wondering if anyone could help me figure out why I am getting the wrong results!

I am using the PISA dataset of 2012 to calculate the variance explained by ESCS.

I followed the steps in the PISA DATA ANALYSES Manuel 2nd edition for SAS:

1. SAS® syntax for preparing a data file for the multilevel analysis (Box 16.3 at p.238-239).

2. SAS® syntax for running multilevel models with the PROC_MIXED_PV macro (Box 16.6 at p.242).

3. Calculating the results:
-> I used following equation:

1-(T2/T1)

T2 = is the intercept/between-school variance in model 2 and;
T1 = is the intercept/between-school variance in the empty model


First I tried using SPSS, than SAS, but either way I am not getting the same results as in the examples given by PISA.
For example: According to PISA, in Norway, 7% of variance in math performance is explained by ESCS. According to my calculations that is wrong, as much as 20% is explained by ESCS, and 25% by ESCS and MU_ESCS.

Interestingly when I run a one-way ANOVA, than the R-square displays (almost) the same number as in the examples by PISA.

I only have a few more days left before submitting my BA Paper, and I really need to calculate the variance for countries that are not included in the examples on PISA's website.
Can someone please please help me figure out why I am not getting the right number.

[Steve Samuels]

Hello, Jenny! Unfortunately this is a forum dedicated to the the statistical package Stata (www.stata.com). We do occasionally discuss comparisons to other packages, but you need serious expertise in SAS or SPSS and multilvel analysis, and familiarity with the PISA publication you reference. I don't know where to direct you, maybe the SAS forum at TalkStats:
http://www.talkstats.com/forumdispla...tical-Software

You might offer to pay someone for help, so that they could obtain the SAS reference overnight from Amazon. They will certainly want to see the exact code you ran and the results. It may be something "simple". Still, as someone who has advised many students, I suggest that you contact your advisor about the situation now.

Good luck & Best wishes
Steve

[wbuchanan]

There are a few different things that could be affecting your results. Did you specify the sampling design/weighting correctly? If so, did you then handle the plausible values correctly (e.g., treated the plausible values as thought they were imputed missing values)? If you're not working with the individual level data you may have difficulty since the results set will have already incorporated the sampling design and the variance in scores between and within imputations before it got to you and analyzing the aggregates would probably leave you with smaller variances than what you've found published elsewhere.

[Jenny Michelle]

Thanks, wbuchanan. I think the plausibles values are being calculated correct. I am investigating if the off result has something to do with the data level I am working at.