Meta-regression with WLS: Wrong sign due to weighting

[Marvin Schuett]

Dear Statalist,

I'm asking for advice to solve a problem with my meta-regression model based on a WLS regression:
I'm using a dataset of 538 observations from 80 hedonic pricing studies that all report the price discount that houses experience due to the proximity to a negative environmental externality.
In my meta-regression model, I try to explain the variance of effect sizes across studies through 20 study characteristics that are, among others, the location in which the study was conducted, the type of negative externality, quality indicators for the original hedonic regression and most importantly the mean distance of the houses in each respective study. I expect that, when the average house in a given study is located more distant, then the reported effect size should be smaller, c.p. However, my regression results point significantly (p<0.01) to the opposite, indicating that, being closer to a negative externality is beneficial for the house prices, which is certainly counterintuitive.

I'm using

reg y x1 x2 ... x19 x_distance [aw=precision_sq], cluster(ID_Study)

where precision_sq is the squared precision of each estimate i.e. the inverse of each estimates squared standard error, as advised by
Stanley TD, Doucouliagos H (2012) Meta-regression analysis in economics and business. Routledge (p.69)
to give comparatively more weight to more precise estimates and to correct for heteroscedasticity.

I looked for outlying or influential observations, checked for multicollinearity, thought about omitted variables and used different definitions of the distance variable with no problem detected and robust "wrong" sign and significance.
However, when considering simple partial correlations, using WLS with a different weight (sample size) or using OLS instead, the coefficient on the distance variable takes the expected sign and is significant. Additionally, several other coefficients change sign and significance as well.

Does anyone have an idea why using WLS with precision squared as weight results in remarkably different results at least for some variables? As weighting by precision is the standard approach, it might be hard to abandon this approach.

I highly aprreciate your efforts and thank you in advance.

Kind regards

Marvin Schütt

[Phil Bromiley]

Welcome to Statalist. You didn't get a quick answer. You'll increase your chances of a useful answer by following the FAQ on asking questions - provide Stata code in code delimiters (what you actually ran), readable Stata output, and sample data using dataex. The level of assistance on Stata list depends a bit on whether someone active does what you're doing - I'm not sure we have a meta analysis user active on the list.

In general, weighting can dramatically change the results - suppose that x1 and x2 had a negative correlation in half of the data and positive in the other half of the data. Depending on which half was weighted how, you could get positive, zero, or negative estimates on correlation.

[Marvin Schuett]

Dear Phil,

thank you a lot for your reply. I appreciate your advise to use dataex and provide Stata output and this would have been my preferred approach of course. Unfortunately, I do not have the permission to share the data or the output at this point, not even a small sample and I fear that it is not possible to create a fake dataset or to use some of Stata's datasets to mimic my problem at hand.
I acknowledge of course, that this renders it very difficult to come up with good suggestions tailored to my problem apart from your already helpful comment that weighing can lead to virtually any change. I had hoped to have tackled an issue that is frequently encountered by meta-analysts but as you mention it: the likelihood to have a meta-analyst active on the list might be not be substantial.
Thank you again.

Best,
Marvin

[Annie Ng]

Dear Marvin,
I have the same problem as you did. The signs of some coefficients in WLS and OLS are completely different although they are both significant. Have you figured out how and why these occurred? Which model did you use?
Thank you.
Best.
Annie.

[Marvin Schuett]

Dear Annie,
thank you for your reply. In a way, I am glad that I am not the only one having this sort of problem.
Luckily, I sorted it out for my case. Sorry for not posting my solution earlier, I simply forgot about my post:

I figured out that my precision variable was very unevenly distributed, i.e. some observations were extremely precise while others were very imprecise. As a result, the WLS FE estimation yielded very different results compared to OLS and WLS RE as some observations received a lot of weight while others were practically conisdered as unimportant. Especially for the WLS RE results, the difference seems intuitive, as the additonal between-study variance is added to the weighting paramter and "smoothes" it.
As for solutions, Feld, L. P., & Heckemeyer, J. H. (2011). FDI and taxation: A meta‐study. Journal of economic surveys, 25(2), 233-272. offer to discard "precision outliers". In my case, this was not suitable as there was no clear cutoff-point and I clearly wanted to avoid dropping a lot of observations. I finally logarithmized the weighting parameter which resulted in results close to OLS and WLS RE. I am not sure if this approach can be generalized to similar situations, as I didn't find any article covering this.
I hope, that you can solve your problem.

Best
Marvin