An empirical question of multilevel models; and how to buid a graph like this one in Stata

[Damian Tojeiro]

Dear Statalist,

I am using Stata 15.1 and I am estimating a three level model (time nested in firms nested in regions). My question is about how to interpret an empty model (with random intercepts for firms and regions) reporting that regional random intercept is not necessary (LR test confirm that random effect for region is not significant), but when including firms as well as regional predictors, the regional ones are highly significant.
Does this mean that the conclusion from the empty model is that I should not include any regional characteristic? Or does this mean that there is not regional random effect, and thus I should use xtlogit (my dependent is a dummy variable) and include regional predictors? With regard to the latest question: I imagine that standard errors should be corrected using cluster(region), but xtlogit does not allow this option (only xtreg), what to do then? I am very confused with respect to include or not regional predictors since they are very significant, even though the empty model says that the region (as a level) is not significant.

The other question is how to replicate in Stata a graph like the one below. As you can see, the "y" variable (aggregated trust) is captured in three time periods for different regions, and it is ordered (in the horizontal axis, even though it is not shown) by the variable "x" which is inequality in 1980. This show a negative association between regions with higher inequality showing lower trust. I give you a dataex example of my data below, for if you can help me with this, it will be much appreciated.

* Example generated by -dataex-. To install: ssc install dataex
clear
input int year float(regionid x y) byte sample
2008 16  6.308918  .20467836 1
2011 16  6.308918   .1985294 1
2014 16  6.308918  .23333333 1
2008  7  .5649685  .19266056 1
2011  7  .5649685   .1632653 1
2014  7  .5649685  .16216215 1
2008 10  4.013527  .15384616 1
2011 10  4.013527  .18487395 1
2014 10  4.013527   .1625616 1
2008  1  4.319562  .13846155 1
2011  1  4.319562   .0886076 1
2014  1  4.319562  .13114753 1
2008 11  .1517857   .0967742 1
2011 11  .1517857   .0882353 1
2014 11  .1517857 .037037037 1
2008  4 .14693877  .04166667 1
2011  4 .14693877          0 1
2014  4 .14693877          0 1
2008  8  1.183995  .13913043 1
2011  8  1.183995  .14678898 1
2014  8  1.183995  .11881188 1
2008  9 16.033205  .25537634 1
2011  9 16.033205  .24547803 1
2014  9 16.033205   .2235294 1
2008  2  1.286149  .20689656 1
2011  2  1.286149   .1612903 1
2014  2  1.286149  .17857143 1
2008 13 18.833364    .154717 1
2011 13 18.833364   .1923077 1
2014 13 18.833364  .11904762 1
2008 15  .5725027  .25454545 1
2011 15  .5725027  .27272728 1
2014 15  .5725027  .14545454 1
2008 12 1.7787274       .192 1
2011 12 1.7787274  .11278196 1
2014 12 1.7787274   .0952381 1
2008  3  .5671447  .10638298 1
2011  3  .5671447        .18 1
2014  3  .5671447   .2162162 1
2008 17         0   .1904762 1
2011 17         0       .125 1
2014 17         0         .2 1
2008  6 1.0164384  .11538462 1
2011  6 1.0164384  .17391305 1
2014  6 1.0164384          0 1
2008 14  .8678223  .19148937 1
2011 14  .8678223   .2244898 1
2014 14  .8678223   .3170732 1
2008  5  .8506494  .06060606 1
2011  5  .8506494          0 1
2014  5  .8506494  .04761905 1
end

[Clyde Schechter]

Does this mean that the conclusion from the empty model is that I should not include any regional characteristic? Or does this mean that there is not regional random effect, and thus I should use xtlogit (my dependent is a dummy variable) and include regional predictors?

It means none of those things. The random effects in these models are just a higher level residual term in the model. So just as adding more predictors (covariates, fixed effects) to an ordinary linear regression will change the value of the residuals, adding more predictors (covariates, fixed effects) to a multi-level regression will change the values of the random intercepts and residuals. There's nothing different about it. When you end up with very small variance components at a higher level in one of these multi-level models, it means that you can, optionally, omit that level from that model, but it does not imply that they would be omittable in a different model that had a different set of predictor variables. And it does not mean that region-level variables are unimportant: in fact sometimes the only model in which the region-level random intercepts can be omitted is one that is chock full of region-level variables. Those region-level variables are doing such a good job of accounting for region level effects that there is nothing left for the random-effect at that level to do!

If you plan to work with several models, unless the region level comes out omittable in all of them, it is best to leave that level in for all of them. Otherwise you have to explain to your readers/audience why some of your models include region and some do not.

With regard to the latest question: I imagine that standard errors should be corrected using cluster(region), but xtlogit does not allow this option (only xtreg), what to do then?

Literally this is true, but it is false in the meaningful sense. The -cluster(varname)- option in -xtreg- is just an abbreviation for -vce(cluster varname)-. It is true that -xtlogit- does not allow the -cluster(varname)- abbreviation, but it does support -vce cluster varname)-.

I am very confused with respect to include or not regional predictors since they are very significant, even though the empty model says that the region (as a level) is not significant.

As indicated near the end of my first paragraph here, it is almost surely the case that the region level intercepts are ignorable precisely because the region level variables are strong enough to explain nearly all variation at the region level. Leaving everything you have there in the model is probably your best bet.

I don't know how to produce the kind of graph you show. I hope somebody else following the thread does and will respond.

Digression: You are focusing on what is "significant." The American Statistical Association has recommended that the concept of statistical significance be abandoned. See https://www.tandfonline.com/doi/full...5.2019.1583913 for the "executive summary" and
https://www.tandfonline.com/toc/utas20/73/sup1 for all 43 supporting articles. Or https://www.nature.com/articles/d41586-019-00857-9 for the tl;dr. I urge you to forget about "significance" and express your results in terms of effect sizes (coefficients, or odds ratios or marginal effects), their practical, real-world importance, and the degree of uncertainty in their estimation (standard errors or confidence intervals) instead.

[Damian Tojeiro]

Dear Clyde, thanks for your answer. Let me kindly ask you a few more things to properly understand. In my research topic (Economics of Innovation) people usually do first an empty model for evaluating if random effects are important (using LR tests) and then they consider if including or not fixed effects for firms (if firm variance is significant or "relevant") and the same for including regional fixed effects (depending on the significance of regional variance). However, when you say:

adding more predictors (covariates, fixed effects) to a multi-level regression will change the values of the random intercepts and residuals

Does this mean that the residuals and/or regional variance could increase (or gain significance) when controlling by regional fixed effects? How could you justify that, if what you expect is that including fixed effects of regions should end in a lower regional variance?

Lets say that a model just including firm-level fixed effects with random intercepts for firms and regions conclude that the regional variability is meaningless. But still I decide to include regional fixed effects and they are significant/important. I understand that regional level is not relevant/important, but regional fixed effects are significant/important. Here is when I get lost, should I assume no regional variability and thus not include regional fixed effects? Or should I assume not regional unobserved effects, and thus include those significant/important regional fixed effects in a xtlogit with vce(cluster region)? Sorry for asking the same again.

Thanks for your suggestion about the "significance" of variables, I will have a look at it.

[Clyde Schechter]

I read your question in #3 as very different from the one in #1, and I realize that we are perhaps using language too loosely, hence causing confusion. In #1 you refer to adding regional "predictors" to the model--which I took to mean variables that are attributes of the region, such as region population density, or region geographic area, etc. In #3 you are talking about region "fixed effects" which I take to mean 0/1 indicators ("dummy variables") identifying the regions.

With predictors and fixed effects understood in this way, I would tell you only that a model that includes both a random intercept at the region level and region level "fixed effects" is ill-formed and I would not try to interpret such a model, nor draw any conclusions about anything from it. It does not make sense to have both fixed effects and random effects for the same level in one model. Either/or.

Starting from a null model with just bare multi-level structure: no predictors in the "fixed" portion of the model, just random intercepts at all of the available levels, if you now add region-level predictors to the model, the the region-random-intercepts will change. Normally they will decrease, and possibly go to indistinguishable from zero if the region-level predictors account for essentially all of the variation at the region level. But if the region level predictors are truly unrelated to the outcome variable in the model and are adding noise rather than signal, it is conceivable that the random intercept variance might increase to "compensate" for that noise. Moreover, if predictors defined at higher or lower levels of the model are introduced, to the extent that some of the information they contain is actually at the region level, they, too, can cause the variance of the region level random intercepts to change, again, in either direction.

For this reason, I would not recommend deciding whether to include the region level in your multi-level model based on the variance of the region-level intercepts in a null model. If you say it is standard practice to do that in your field, then perhaps you will be coerced into doing it, but it makes no statistical sense that I can see. That would just be one more thing to add to the long list of entrenched but wrong statistical practices.

[Damian Tojeiro]

Dear Clyde, you are totally right, in #3 I mean to say regional-level predictors (not region dummy variables that are known in econometrics as region fixed effects; even though some textbooks in multilevel refer to "fixed effects" when talking about predictors in the fixed part of a multilevel model.
I know see why you advice not to choose the regional level just seeing the regional random variance from the empty model, if I understood well, it could be the case that firm-level predictors may cause that regional variance decrease (but more important in my case, to increase).

Lets say that a model just including firm-level predictors with random intercepts for firms and regions conclude that the regional variability is meaningless

The latter sentence from #3 might be a better strategy for deciding to include regional random intercept and/or regional

predictors?

My point is that I do not understand the purpose of conducting xtlogit with vce(cluster region) standard errors including firm and regional predictors (without regional random intercepts) as I have seen in some papers.

Another problem I have encountered when doing multilevel model with firm and regional levels is the convergence, ending in something like this:

------------------------+----------------------------------------------------------------
region                  |
              var(_cons)|   1.41e-34   3.42e-19                             .           .
------------------------+----------------------------------------------------------------
region>firm             |
              var(_cons)|   3.367517   .2211876                      2.960742    3.830178
-----------------------------------------------------------------------------------------

As you can see, the regional variance does not show confidence intervals and therefore it is impossible to do LR test for that parameter. I some times see that using vce(robust) do report the intervals, but you cannot do LR test with this and I remember you advice me once not to use the "force" option. Is there any particular solution for solving this problem? Maybe using another command like gllamm? I would like to know your opinion.

[Clyde Schechter]

I do not understand the purpose of conducting xtlogit with vce(cluster region) standard errors including firm and regional predictors (without regional random intercepts) as I have seen in some papers.

The purpose of clustered standard errors is to account for the non-independence of observations within the clusters. Now, one source of non-independence is when each cluster is also characterized by its own random intercept. But even when all the random intercepts are 0, you can still have non-independence of observations within regions. So in that case it would make sense to use vce(cluster region), even though you have determined that there is no need for region-level intercepts in the model.

As you can see, the regional variance does not show confidence intervals and therefore it is impossible to do LR test for that parameter.

In the example you show, the standard error for the region-level variance is 13 orders of magnitude bigger than the estimated variance itself! So the confidence interval, computed in the usual normal-theory way (which is what Stata does here) would give a negative lower bound--which is not very helpful. That's why it's not being shown. But in a situation like this you don't need a likelihood-ratio test to tell you that the variance component at the regional level is small enough to ignore: just look at how utterly tiny it is, and its confidence interval, calculated as mean +/- 1.96 standard errors, extends, relative to its magnitude, a huge distance into both positive and negative territory.

[Damian Tojeiro]

Dear Clyde, thanks a lot for all your answers and patience. It is very helpful to talk to you.
Just if you allow me, a small last question: could you suggest me a reference for start studying multilevel multiple membership in Stata? I have read once that for doing this, dataset should be putted in “wide” form (opposite to “long” form which is what I use due to the panel structure of my dataset).

[Clyde Schechter]

I don't know of any references for multiple membership in multilevel models in Stata. There is the Stata reference manual, of course. I actually first learned it in an online course about multilevel modeling from Bristol University. That course used both HLM and Stata and taught coding for both of them. It was also an excellent course, and I believe it is still offered.

I don't think that wide layout is needed, nor in any way helpful, for this.

[Damian Tojeiro]

Dear Clyde, thanks for your help.