Timing of variable standardization

[Philip Gigliotti]

Hello,

I have data on standardized test scores from all school districts in a given state. I would like to standardize these scores by year to a normal distribution, because the scale of the test scores changes in the last two years of the sample. Also, I know that some of the school districts will be excluded from my regressions because of missing data on a number of variables.

Should I perform the standardization on the full set of test score measures, or should I drop the observations that stata will exclude for missing observations, and standardize only the observations in my regression sample?

Thanks.

(cross posted on stack exchange: https://stats.stackexchange.com/ques...analysis-sampl )

[Roman Mostazir]

First of all, deleting observations will not contribute anything towards your standardization. And second but most importantly, you should not delete any observation just because they are missing. This will likely to bias your estimates later in your regression models. Bear in mind, depending on what model you will be developing, a missing value in an observation will not always be excluded from analysis. In OLS regression they will be excluded but in Maxmimum Likelihood based regression models i.e. for clustered/panel data, the model may retain the observation. Therefore, my suggestion is do not delete any observation.

[Philip Gigliotti]

I'm not making a choice to drop the observations. They are dropped by stata because I don't have data for all of the districts on all of the variables in my models.

It changes my estimates meaningfully if I standardize the entire group of available observations, versus just the observations contained in my final regression models.

My inclination is to standardize based on all available data, because those values have meaning, as in the districts place in the distribution of schools in their state. But I'm wondering if there is a reason it would be more correct to only standardize the sample that will be used in the regression model.

[Phil Bromiley]

You'll generally get more helpful answers by following the FAQ on asking questions including providing Stata code in code delimiters, readable Stata output, and sample data using dataex.

You seldom if ever need to worry about creating variables with normal distributions. Standardization does not do that anyway. You may assume the errors are normal when you interpret statistical significance, but that is the error not the variables themselves.

That you get different results when you standardize by all data versus data usable in your regression means that the dropped observations differ in distribution from the included observations in your regression. That implies you'll probably need a heckman/sample selection model.

In standardizing, you're essentially trying to make the different years have the same scales. I suspect is is more important to keep the same units in the sample in each year than anything else. Assuming the data don't have errors, I'd tend to standardize on the full set of districts in every year, and then worry about the missing data in the heckman estimation. However, I'm not terribly confident in this suggestion.

[Philip Gigliotti]

I ended up going with a standardization on only the regression sample. My results with the standardization on the full sample were more highly significant and favorable to my hypothesis, so I felt I had to go with the more conservative option. I think it's important that the descriptive statistics table shows a standardized variable with a mean 0 and standard deviation of one and similar sample size to the other variables if you're going to call it standardized, otherwise there will be raised eyebrows.

I talked to a friend who is a few years older and faculty in my field at another University. He immediately and unequivocally said that the variable should be standardized on the regression sample. His reasoning was that "the interpretation of the coefficients will be off if you are not standardized on the regression sample." (He is sort of terse and I didn't want to push him for elaboration).

[Jesse Wursten]

Must be a field specific thing - I don't think I've standardised a single variable in the last couple of years. I may have harmonised a few though...

[Philip Gigliotti]

Must be a field specific thing - I don't think I've standardised a single variable in the last couple of years. I may have harmonised a few though...

It's better not to standardize. In this case it was unavoidable due to a scale change in the variable. In years 6 and 7, test scores were graded on a 100 point scale instead of a 200 point scale. The year fixed effect soaks up some of that divergence, but it's still problematic because the observations in those years have a completely different distribution. To give all years the same distribution, it was necessary to standardize by year.

[Jesse Wursten]

What's wrong with just dividing the test scores by two? I feel like I'm missing something.

[Philip Gigliotti]

What's wrong with just dividing the test scores by two? I feel like I'm missing something.

That's one possible transformation, and I did try it. The problem is that it doesn't completely mitigate the different distribution in the 6th and 7th year. After multiplying those years by two, the scores in those years were still systematically lower than the other years. That's really problematic in a panel data fixed effects model, because even if a district improved relative to other districts or their past performance, the model will only show a within group decline in performance for all groups in those years. The only way to ensure consistent distributions in those last two years was to standardize all the years to the same distribution.

[Jesse Wursten]

I have to admit I know absolutely nothing about this sort of research, but couldn't it simply be that they simply scored worse in those years? If it's a panel-wide effect, then time dummies would filter that out (assuming you want that to be filtered out).

[Philip Gigliotti]

I think it's very unlikely that every district in the sample scored approximately 10 points below their group mean score in that year. Much more likely that the scale change was accompanied by some change in the test that led to systematically lower scores. While year fixed effects can counteract some of the effect, I'm not sure theoretically how well they deal with a change in the distribution of the variable, and I'm not comfortable relying on them when there is a standardization method that is more commonly used that by definition address the problems with the distributions. Even if every district did actually score lower that year for performance reasons, then the more interesting finding is how their standing changed in the distribution of scores relative to other districts. That's the interpretation that the standardization allows.

[Philip Gigliotti]

In addition, my identification strategies require district specific linear time trends. I think having a spurious downward trend in the last two years could be problematic, in that it would bias the time trend downward, when there might have been an upward trend.

[Philip Gigliotti]

I did some more digging on the changes in these two years. Rather than a different scale, a different formula was used to compute these measures in year 6 and 7. From the NYS accountability website:

2012-13 and 2013-14 elementary/middle-level ELA/math: Student scores on the tests are converted to six performance levels: Level 1 On Track, Level 1 Not On Track, Level 2 On Track, Level 2 Not On Track, Level 3, and Level 4. A PI is calculated using the levels and the following equation: ([2(Count at Level 1 On Track) + Count at Level 2 Not On Track + 2(Count at Level 2 On Track) + 2(Count at Level 3) + 2(Count at Level 4)] ÷ [Count of Tested Students]) × 100

2011-12 and Prior and 2014-15 and 2015-16 elementary/middle-level ELA/math: Student scores on the tests are converted to four performance levels, from Level 1 to Level 4. A PI is calculated using the levels and the following equation: ([(Count at Level 2) + 2(Count at Level 3) + 2(Count at Level 4)] ÷ [Count of Tested Students]) × 100

This leads to different distribution in the 2012-13 and 2013-14 years. Here are summary statistics:

summarize mathprof elaprof if Year > 2012

Variable Obs Mean Std. Dev. Min Max

mathprof 1,255 106.6956 25.50982 33 180
elaprof 1,255 104.8876 22.86684 37 169

summarize mathprof elaprof if Year < 2012

Variable Obs Mean Std. Dev. Min Max

mathprof 3,681 175.6322 16.065 96 200
elaprof 3,667 167.6657 15.80418 91 197

With this glitch, multiplying by 2 would not be an appropriate transformation, and I don't think there is an appropriate simple arithmetic transformation that could address this issue.

[Jesse Wursten]

Makes sense. Although you could wonder if standardisation does fix the issue, because the scores almost measure different things. I literally have no idea though.

[Philip Gigliotti]

Makes sense. Although you could wonder if standardisation does fix the issue, because the scores almost measure different things. I literally have no idea though.

I think the standardized variable produces a clear interpretation. It measures the performance of the same group of students on an evaluation of the same material, and conveys each school districts performance on those evaluations relative to each other. Within group change over that time tells whether the group improved relative to other schools in the state over the period in the sample.