Testing for the significance of categorical by continuous interactions

[Artur Obminski]

Hello,

We are studying the peer effects of the share of girls in classrooms on mathematics test scores. In our model, we use school fixed effects and compare classrooms within schools, to control for e.g. socioeconomic variation between schools. Furthermore, we allow for heterogeneous effects by including interactions between the outcome variable (the test scores) and demographic groups (specifically: girl * girlshare; immigrant * girlshare; girl * immigrant * girlshare).

In our main model, we include school fixed effects, but do not include fixed effects/dummies for the interactions between schools and the heterogeneous effect interactions mentioned. Instead, we aim to include them in a robustness check and test for the significance of these added school-by-demography-by-score variables. How do we do this, and which interactions should we include? (As you see in the code, some interactions are component parts of other interactions.) Preferably, we would also then like to test the significance of individual interaction terms rather than only the whole set of added interactions.

clear
input float mathscore double girlshare float(intGirlShare intNewShare intGirlNewShare) double SchoolID float newlyArr double girl
 -.16130325 .5714285714285714 .5714286         0 0 1 0 1
 -1.8561664 .6190476190476191        0  .6190476 0 1 1 0
  -.3401746 .6190476190476191        0         0 0 1 0 0
 -1.4119294 .5714285714285714 .5714286         0 0 1 0 1
 -.06797387 .5714285714285714 .5714286         0 0 1 0 1
  1.1607436 .5714285714285714 .5714286         0 0 1 0 1
  .01306791 .6190476190476191        0         0 0 1 0 0
   .3450717 .5714285714285714 .5714286         0 0 1 0 1
  -3.188828 .5714285714285714 .5714286         0 0 1 0 1
 -1.4759908 .5714285714285714 .5714286         0 0 1 0 1
   .6042861 .6190476190476191        0         0 0 1 0 0
   .6891549 .5714285714285714 .5714286         0 0 1 0 1
   .1693752 .6190476190476191        0         0 0 1 0 0
 -.07945836 .5714285714285714 .5714286         0 0 1 0 1
 -.17147087 .6190476190476191        0  .6190476 0 1 1 0
   .9958695 .5714285714285714 .5714286         0 0 1 0 1
   .9149415 .6190476190476191        0         0 0 1 0 0
  -.6679537 .5714285714285714 .5714286         0 0 1 0 1
   .5013217 .6190476190476191        0         0 0 1 0 0
  .11758123 .6190476190476191        0         0 0 1 0 0
   .8935187 .5714285714285714 .5714286         0 0 1 0 1
 -.06591168 .5714285714285714 .5714286         0 0 1 0 1
  -1.259523 .3684210526315789        0         0 0 1 0 0
 -1.2864072 .3157894736842105 .3157895         0 0 1 0 1
   .5957668 .3684210526315789        0         0 0 1 0 0
   .2100161 .3684210526315789        0         0 0 1 0 0
  -.5648643 .3157894736842105 .3157895         0 0 1 0 1
  -.8529068 .3684210526315789        0         0 0 1 0 0
   .3019897 .3684210526315789        0         0 0 1 0 0
 -1.7281017 .3157894736842105 .3157895         0 0 1 0 1
  -.8380347 .3157894736842105 .3157895         0 0 1 0 1
  -.2162913 .5813953488372093 .5813953         0 0 2 0 1
  -.2162913 .5813953488372093 .5813953         0 0 2 0 1
 -1.0756866 .6046511627906976        0         0 0 2 0 0
 -1.0756866 .6046511627906976        0         0 0 2 0 0
    .727842 .6046511627906976        0         0 0 2 0 0
    .727842 .6046511627906976        0         0 0 2 0 0
  -.6058232 .5813953488372093 .5813953         0 0 2 0 1
  -.6058232 .5813953488372093 .5813953         0 0 2 0 1
     1.0331 .5813953488372093 .5813953         0 0 2 0 1
     1.0331 .5813953488372093 .5813953         0 0 2 0 1
  .51550436 .5813953488372093 .5813953         0 0 2 0 1
  .51550436 .5813953488372093 .5813953         0 0 2 0 1
  .14169058 .6046511627906976        0         0 0 2 0 0
  .14169058 .6046511627906976        0         0 0 2 0 0
  .22233284 .6046511627906976        0         0 0 2 0 0
  .22233284 .6046511627906976        0         0 0 2 0 0
   .8776796 .5813953488372093 .5813953         0 0 2 0 1
   .8776796 .5813953488372093 .5813953         0 0 2 0 1
   .7596178 .6046511627906976        0         0 0 2 0 0
   .7596178 .6046511627906976        0         0 0 2 0 0
   1.329355 .6046511627906976        0         0 0 2 0 0
   1.329355 .6046511627906976        0         0 0 2 0 0
  .24859844 .5813953488372093 .5813953         0 0 2 0 1
  .24859844 .5813953488372093 .5813953         0 0 2 0 1
  .28471455 .5813953488372093 .5813953         0 0 2 0 1
  .28471455 .5813953488372093 .5813953         0 0 2 0 1
   .6089274 .6046511627906976        0         0 0 2 0 0
   .6089274 .6046511627906976        0         0 0 2 0 0
   .8943321 .5813953488372093 .5813953         0 0 2 0 1
   .8943321 .5813953488372093 .5813953         0 0 2 0 1
  -.1875581                .6       .6         0 0 2 0 1
 -1.0971143                .6       .6         0 0 2 0 1
   1.954816               .65        0         0 0 2 0 0
  -.4486425                .6       .6         0 0 2 0 1
  .25757462                .6       .6         0 0 2 0 1
   .3058077                .6       .6         0 0 2 0 1
   .4133211                .6       .6         0 0 2 0 1
-.006400176               .65        0         0 0 2 0 0
    .899845                .6       .6         0 0 2 0 1
  1.1249338                .6       .6         0 0 2 0 1
  1.2374424               .65        0         0 0 2 0 0
   .7273457                .6       .6         0 0 2 0 1
   .5862164               .65        0         0 0 2 0 0
end
label values SchoolID IDSCHOOL
label values newlyArr newlyArr
label def newlyArr 0 "No", modify
label def newlyArr 1 "Yes", modify
label values girl girl
label def girl 0 "Male", modify
label def girl 1 "Female", modify


quietly reg mathscore girlshare intGirlShare intNewShare intGirlNewShare SchoolID#girl#newlyArr SchoolID#girl#c.girlshare SchoolID#newlyArr#c.girlshare SchoolID#girl#newlyArr#c.girlshare, cluster(SchoolID)

testparm SchoolID#girl#newlyArr SchoolID#girl#c.girlshare SchoolID#newlyArr#c.girlshare SchoolID#girl#newlyArr#c.girlshare

[Clyde Schechter]

Your example data and code do not match up well with your description, which I find confusing in its own right. So I'm going to offer some generic advice.

1. If you include a high level interaction such as SchoolID#girl#newlyArr, you should automatically include all of its sub-interactions and main effects. That is, if you keep that interaction you also need to have SchoolID#girl, SchoolID#newlyArr and girl#newlyArr with it, as well as SchoolID, girl, and newlyArr themselves.

2. But the converse is not true. You could, for example, decide to include SchoolID, girl, newlyArr, girl#newlyArr, SchoolID#girl, and SchoolID#newlyArr without including the three-way term SchoolID#girl#newlyArr.

3. Consequently, I would start from the top and work down. First consider each three-way interaction and decide whether it is to be included. If so, eilminate it, re-run the model, and then decide whether to retain each two way interaction that was contained within it, etc.

4. I do not endorse using any kind of significance testing to decide what to include in a model. I believe that should be done, ideally, before you even analyze the data in the first place, based on what is known or follows from accepted theory. If there is no such prior basis for picking the model terms, then I would do it by looking at the models predictions (see -margins-), not significance tests.

5. Modeling schools with fixed effects may well prove quite unwieldy if you have a large number of schools.